Maximum Inner-Product Search using Tree Data-structures

Maximum Inner-Product Search using Tree Data-structures

Parikshit Ram    Alexander G. Gray
Abstract

The problem of efficiently finding the best match for a query in a given set with respect to the Euclidean distance or the cosine similarity has been extensively studied in literature. However, a closely related problem of efficiently finding the best match with respect to the inner product has never been explored in the general setting to the best of our knowledge. In this paper we consider this general problem and contrast it with the existing best-match algorithms. First, we propose a general branch-and-bound algorithm using a tree data structure. Subsequently, we present a dual-tree algorithm for the case where there are multiple queries. Finally we present a new data structure for increasing the efficiency of the dual-tree algorithm. These branch-and-bound algorithms involve novel bounds suited for the purpose of best-matching with inner products. We evaluate our proposed algorithms on a variety of data sets from various applications, and exhibit up to five orders of magnitude improvement in query time over the naive search technique.

1 Introduction

In this paper, we consider the problem of efficiently finding the best-match for a query from a given set of points with respect to the inner-product similarity. Formally, we consider the following problem:

Problem. For a given set of points and a query , efficiently find a point such that:

(1)

We call this the problem of maximum inner-product search. The focus of this paper is to improve the efficiency of this search. An alternate formulation of the above problem in terms of a vector and matrix multiplication is as following:

Problem. For a given vector and a matrix , efficiently compute the following:

(2)

This problem appears to be very similar much existing work in literature. Efficiently finding the best match with respect to the Euclidean (or more generally ) distance is the widely studied problem of fast nearest-neighbor search in metric spaces [9]. Efficient retrieval of the best match with respect to the cosine similarity is the extensively researched in the field of text mining and information retrieval [1]. But as we will explain in the next section, the maximum inner-product search is not only different from these aforementioned tasks, but also arguably harder.

1.1 Applications

An obvious application of maximum inner-product search stems out of the widely successful matrix-factorization framework in recommender system challenges like the “Netflix prize” [22, 21, 2]. The matrix-factorization task obtains accurate representation of the available data in terms of user vectors and items vectors (examples for items would be movies or music). In this setting, the preference of a user for an item is the inner-product between the corresponding user’s vector and the item’s vector. The efficient retrieval of recommendations for a user is equivalent to the problem in equation 1 with the user as the query and the items as the reference set. For the challenges, linear scan of the items are usually employed to find the best recommendations. An efficient search algorithm would make the retrieval of recommendations in the matrix-factorization framework scalable to real world systems.

The usual document retrieval tasks use the cosine-similarity to match documents. However, in certain setting [11], the documents are represented as (not necessarily normalized) vectors and the inner-product between these vectors represent the similarity between the documents. In this case, unless the vectors are normalized to have the same length, document matching using the cosine-similarity [1] might make the algorithm scalable at the cost of returning inaccurate solutions since the inner-product is not the same as the cosine-similarity (we will explain this more elaborately in the section 2).

There is a similar problem known as the the max-kernel operation: for a given set of points and a query and a kernel-function , the task is to find the point with the maximum value of over the set . This problem is widely used in maximum-a-posteriori inference [20] in machine learning, and for the task of image matching [23] in computer vision. If the kernel function can be explicitly represented in the form a function such that , then this problem reduces to the problem in equation 1 after all the points in the set and the query is transformed into the -space.

1.2 This Paper

In this paper, we consider the general problem of efficient maximum inner-product search and propose two tree-based branch-and-bound algorithms along with a new data structure to solve this problem more efficiently that the naive linear scan. In the following section, we contrast this problem to the usual problems of nearest-neighbor search in metric spaces and best-matches with respect to cosine-similarity. This presents the need for explicit attention to this problem (equation 1). However, we do motivate the use of the existing tree data structures for solving this task efficiently. In section 3, we propose a simple branch-and-bound algorithm using the existing ball-tree data structure [28] and a novel bound. In the following section (section 4), we address the situation where there are multiple queries on the same set of points and propose a dual-tree branch-and-bound algorithm along with a new tree data structure, cone trees, to index the queries. The proposed algorithms are evaluated for their efficiency over a variety of data sets in section 5. Section 6 demonstrates how the proposed algorithms can be applied to the max-kernel operation with a general kernel function where it is not required to have an explicit representation of the points in the -space. In the final section (section 7), we provide our conclusions along with possible future directions for this work.

2 Maximum Inner-product Search

The inner-product between two vectors is very closely related to the Euclidean distance between the points represented by these vectors as well as to the cosine-similarity between these to vectors. Numerous techniques exists for nearest-neighbor search in Euclidean metric space (see surveys like [9]). Large scale best matching algorithms have also been developed for the cosine-similarity measure [1], with a lot of focus on text data. The problem of nearest-neighbor search (in metric space) has also been solved approximately with the widely popular Locality-sensitive hashing (LSH) method [14, 18]. The LSH technique has also been extended to other forms of similarity functions (as opposed to the distance as a dissimilarity function) like the cosine similarity [7]111An important thing to note here is that the similarity function used in Charikar et.al.[7] is not exactly the cosine-similarity. The distance between two points and was measured by , where is the angle made the two points at the origin, making the similarity function . This similarity function has a direct correspondence to the cosine similarity.. The approximate max-kernel operations can also be solved efficiently with LSH under certain conditions on the kernel function. Some other techniques like dimension reduction [30] and dual-tree algorithms [20] have also been used to solve the approximate max-kernel operation efficiently.

Figure 1: Best matches: For a given query , , and denote the best match with respect to the Cosine-similarity, the Euclidean distance and the Inner-product respectively. It is apparent from this figure that even on a plane, the best match with respect to these similarity functions can be very different.

2.1 How is maximum inner-product search different from existing problems?

In what follows, we will try to show that the problem stated in equation 1 is different from these existing problems. Hence techniques applied to these problems (like LSH) cannot be directly applied to this problem.

Nearest-neighbor Search in Euclidean Space.

The problem of finding the nearest-neighbor in this setting can be posed as finding a point for a query such that:

Hence, if the norms of all the points in are normalized to have the same length, then the problem of finding the best match with respect to the inner-product is equivalent to the problem of finding the nearest-neighbor in Euclidean metric space. However, without this restriction, the two problems can have potentially very different answers (figure 1).

Best-matching with Cosine-similarity.

The problem of finding the best match with respect to the cosine-similarity can be posed as finding a point for a query such that

As in the previous case, the best match with cosine similarity is the best match with inner-products if all the points in the set are normalized to have the same length. Under general conditions, the best matches with these two similarity functions can be very different (see figure 1).

Locality-sensitive Hashing.

LSH has been applied to a wide variety of similarity functions. LSH involves constructing hashing functions such that each hash function must satisfy the following for any pair of points :

(3)

where is the similarity function of interest. For our situation, we can normalize our data set such that 222This normalization is different than the normalization mentioned before where all the points were normalized to have the same length. Here the lengths are normalized to be less than equal to one, but not equal to each other., and assume that the all the data is in the first quadrant (so that none of the inner-products go below zero). In that case, is a valid similarity function of interest.

It is known that for any similarity function to admit a locality sensitive hash function family (as defined in equation 3), the distance function must satisfy the triangle inequality (Lemma 1 in [7]). However, the distance function does not satisfy the triangle inequality (even when all the points are restricted to the first quadrant)333Consider the following counter example: Let be points such that , and angles made between , and at the origin are , and respectively. In this case the inequality, does not hold for . .. So LSH cannot be applied to the inner product similarity function even when we assume that all the data lies in the first quadrant (which is quite a restrictive assumption).

Efficient Max-kernel Operation.

There are different existing techniques of solving this problem efficiently. For kernel functions with very high (possibly infinite) dimensional explicit representations, Rahimi, et.al., 2007 [30], propose a technique to transform these high-dimensional representations into lower-dimensions while still approximately preserving the inner-product to improve scalability. However, the final search still involves a linear scan over the set of points for the maximum inner-product or a fast nearest-neighbor search under the assumption that finding the nearest-neighbor is equivalent to maximizing the inner-product. For translation invariant kernels444Kernel functions which are dependent only the (Euclidean) distance between the points and are considered translation invariant kernels. The Gaussian RBF kernel is such a translation invariant kernel function., a tree-based recursive algorithm has been shown to scale to large sets [20]. However, it is not clear how this algorithm can be extended to the general class of kernels. LSH is widely used for image matching in computer vision [23], but only for kernel functions that admit a locality sensitive hashing function [7].

Hence, none of the existing techniques can be directly applied to our problem (equation 1) without introducing inaccurate results or limiting assumptions.

2.2 Why is maximum inner-product search possibly harder?

Unlike the distance functions in metric space, inner products do not induce any form of triangle inequality (even under some assumptions as mentioned in the previous section). Moreover, this lack of any induced triangle inequality causes the similarity function induced by the inner products to have no admissible family of locality sensitive hashing functions. And any modification to the similarity function to conform to widely used similarity functions (like Euclidean distance or Cosine-similarity) will create inaccurate results.

Moreover, inner-products lack a very basic property of generally used similarity functions – the self similarity is high (generally the highest). For example, the Euclidean distance of a point to itself is 0; the cosine-similarity of a point to itself is 1. The inner-product of a point to itself is , which may be high or low depending on the value of the . Moreover, there can possibly be many other points like in the set such that .

Hence, without any assumptions, this problem of obtaining the best match with respect to the maximum inner product is inherently harder than the previously dealt similar problems. This is possibly the reason why there is no existing work for this problem without any restrictions on the domain (at least to the best of our knowledge).

2.3 Are trees the answer?

In this paper, we explore the tree data structure for indexing the points and a branch-and-bound algorithm specifically for the task of maximum inner-product search. Tree data structures have been widely used for the task of nearest-neighbor search [13, 4, 29]. And even though the task of nearest-neighbor search is slightly different from the task of maximum inner-product search, we believe that trees can still be useful for this task.

Trees are known to be good indexing schemes in low to medium dimensions, while some new tree data structures have been developed for data in high dimensions with some low dimensional structure [10, 4]. A hierarchical representation of the data is useful. In this paper, we try to solve the problem of exact maximum inner-product search. The hierarchical tree data structure provides a very intuitive extension to solve the problem approximately to gain efficiency [8, 32].

Moreover, if the search is to be performed with strict constraints – error constraints or time constraints, the tree-based branch-and-bound algorithms can be easily adapted for that purpose. This is because these branch-and-bound algorithms are incremental algorithms. This is not possible with something like LSH – LSH provides theoretical error bounds, but there is no way of ensuring the error constraint during the search. Moreover, LSH is inherently not an incremental algorithm, and hence cannot be used in a limited time setting.

An important advantage of trees is that the trees require a single construction – the branch-and-bound algorithm adapts for the different levels of approximate and/or time limitations. Hashing techniques require multiple hashes for different levels of approximation. The usual norm is to pre-hash for multiple values of approximation. Trees can also be constructed by learning from the data using techniques from machine learning [6, 25] to provide better accuracy and efficiency.

This is why we use trees to solve the problem. Trees might not be the best possible way to solve this problem, but trees do bring a lot of advantages with them.

3 Tree-based Search

Ball trees [29, 28] are binary space-partitioning trees that have been widely used for the task of indexing data sets. Every node in the tree represents a set of points and each node is subsequently indexed with a center and a ball enclosing all the points in the node. The set of point at a node is divided into two disjoint sets which form the child nodes. This partitions the space into (possibly overlapping) hyper-spheres. The tree is built hierarchically and a node is declared to be a leaf node if it contains a set of points of size below a threshold value .

Figure 2: Ball-trees: All the points are limited within the root ball (the bold-face circle). However, the subsequent balls does not necessarily lie within the parent ball – the points still lie within the root ball, but the ball enclosing the points in the child node are not necessarily compact enough to lie within the parent ball. However, the child node would be confined within the parent node if we used hyper-rectangles instead of balls to index the data.

3.1 Tree Construction

We use a simple ball tree construction heuristic that approximately picks a pair of pivot points which are farthest apart from each other [28], and splits the data by assigning the points to their closest pivot. The intuition behind this heuristic is that these two points might lie in the principal direction. The splitting and the recursive tree construction algorithm is presented in Algorithms 1 & 2 for completeness.

The tree is very space efficient since every node only stores the indices of the item vectors instead of the item vectors themselves. Hence the matrix for the items is never duplicated. Another implementation optimization is that the vectors in the items’ matrix are sorted in place (during the tree construction) such that all the items in the same leaf node are arranged serially in the matrix. This is to avoid any random access to the memory when accessing items in the same leaf node.

  Pick a random point
  
  
  return
Algorithm 1 MakeBallTreeSplit(Data )
  Input – Set
  Output – Tree
  
  
  
  if   then
     // Leaf node
     return
  else
     // else split the set
     
     
     
     
     
     return
  end if
Algorithm 2 MakeBallTree(Set of items )
Figure 3: Ball-tree Construction: The object denotes the set of points in the node . denotes the Euclidean mean of the items in the node and denotes the minimum radius of the ball centered around enclosing all the points in the node . and denotes the left and right child of the tree node .

3.2 Branch-and-bound algorithm

Ball trees are widely used for the task of nearest neighbor search and are known to be fairly scalable to moderately high dimensions [28, 26]. The search usually employs the depth-first branch-and-bound algorithm. A nearest neighbor query is answered by traversing the tree in a depth-first manner by first going down the node closer to the query and bounding the minimum possible distance to the other branch with the triangle-inequality. If this bound is greater than the current neighbor candidate for the query, the branch is removed from computation.

An analogous greedy depth-first algorithm can be used for maximum inner-product search. But instead of traversing down the node closer to the query, the choice is made on the basis of the maximum possible inner-product between the query and any potential point from the node. The recursive depth-first branch and bound algorithm is presented in Algorithm 4. The search algorithm for a query () begins at the root of the tree (Alg. 5). At each step, the algorithm is at a tree node (). It checks if the maximum possible inner-product between the query and any point in the node, , is any better than the current best-match for the query (). If the check fails, this branch of the tree is not explored any more. Otherwise, the algorithm recursively traverses the tree, exploring the branch with the better potential candidates in a depth-first manner. If the node is a leaf, the algorithm just finds the best-match within the leaf with a linear search (Alg. 3). This algorithm ensures that the exact solution (i.e., the best-match) is returned by the end of the algorithm.

  for  each  do
     if   then
         
         
     end if
  end for
Algorithm 3 LinearSearch(Query , Reference Set )
  if  then
     // This node has potential
     if isLeaf(T) then
         LinearSearch(, )
     else
         // best depth first traversal
         ; ;
         if  then
            TreeSearch; TreeSearch;
         else
            TreeSearch; TreeSearch;
         end if
     end if
  end if
  // Else the node is pruned from computation
  return;
Algorithm 4 TreeSearch(Query , Tree Node )
   MakeBallTree()
  for  each  do
     ;
     ;
     TreeSearch(, );
     return ;
  end for
Algorithm 5 FindExactMaxIP(Query set , Reference Set )
Figure 4: Single-tree Search: The object contains the current best-match candidate for the query and denotes the inner-product between the query and its current best-match . The function ‘’ denotes the upper bound on the maximum possible inner-product between the query and any point lying in the tree node .

3.2.1 Bounding maximum inner-product with a ball

Since the triangle inequality does not hold for the inner product, we present an novel analytical upper bound for the maximum possible inner product of a given point (in this case, the query ) with points in a ball. It is important to note that the information about the ball is limited to its center and its radius. For the rest of this section, we use the notation to denote the .

Figure 5: Bounding with a ball
Theorem 3.1.

Given a ball of points centered at with radius and (query) point , the maximum possible inner product between the point and the ball is bounded from above by:

(4)
Proof.

Suppose that is the best possible match in the ball for the query and be the Euclidean distance between the ball center and (by definition, ). Let be the angle between the vector and the vector , and be the angles made at the origin between the vector and vectors and respectively (see figure 5). The length of in terms of and is:

(5)

The angle can be expressed in terms of and as:

(6)

Let be the angle between the vectors and . With the triangle inequality of angles, we have:

Assuming that the angles lie in the range (instead of the usual ), and the fact that , we get:

(7)

since is monotonically decreasing in the range . Using this inequality we obtain the following bound for the highest possible inner-product between and any point in the ball:

where the last inequality follows from equation 7. Substituting equations 5 & 6 in the above inequality, we have

The second inequality comes from the definition of maximum. The following equality comes from maximizing over . This gives us the optimal value of . Simplifying the final inequality gives us equation 4. ∎

For the tree-search algorithm (Alg. 4), we set the maximum possible inner-product between and a tree node as

This upper bound can be computed in almost the same time required for a single inner-product (since the norms of the queries can be pre-computed before searching the tree). This algorithm is evaluated against the naive linear search algorithm in section 5.

4 Dual-tree based Search

For a set of queries, the tree can be traversed separately for each query. However, if the set of queries is very large, a common technique to improve efficiency of querying is to index the queries in the form of a tree as well. The search is then subsequently done by traversing both trees simultaneously using the dual-tree algorithm [15]. The basic idea is to amortize the cost of tree-traversal for a set of queries which are very similar to each other and would follow (approximately) the same path down the tree. The dual-tree algorithms have been applied to different tree-based algorithms like nearest-neighbor search [15] and kernel density estimation [16] with provable theoretical runtime bounds [31].

4.1 Dual-tree Branch-and-bound Algorithm

The generic dual-tree algorithm is presented in Algorithm 6. Similar to the Algorithm 4, the algorithm traverses down the tree on the reference set (referred to as the RTree). However, the algorithm also traverses down the tree on the set of queries (QTree), resulting in a four-way recursion. At each step, the algorithm is at a QTree node and a RTree node T. For every , the value denotes the minimum inner-product between any query in and its current best-match candidate. If this value is greater than the maximum possible inner product, , between any query in and any reference point in , this part of the recursion is no longer explored. When the algorithm is at the leaf level of both the trees, it obtains the best-matches for each query in the QTree leaf by doing a linear scan over the RTree leaf.

  if  then
     // This node has potential
     if isLeaf() & isLeaf(then
         for each  do
            LinearSearch()
         end for
         
     else if isLeaf(then
         DualSearch; DualSearch;
         
     else if isLeaf(then
         ; ;
         if  then
            DualSearch; DualSearch;
         else
            DualSearch; DualSearch;
         end if
     else
         // best depth first traversal
         ; ;
         if  then
            DualSearch; DualSearch;
         else
            DualSearch; DualSearch;
         end if
         ; ;
         if  then
            DualSearch; DualSearch;
         else
            DualSearch; DualSearch;
         end if
         
     end if
  end if
  // Else the node is pruned from computation
Algorithm 6 DualSearch(QTree Node , RTree Node )
   MakeBallTree()
   MakeQueryTree()
   trees nodes in the tree , ;
   queries , , ;
  DualSearch(, );
   queries , return ;
Algorithm 7 FindExactMaxIPDualTree(Query Set , Reference Set )
Figure 6: Dual-tree Search: The tree-building subroutine for the set of queries “MakeQueryTree” can be the “MakeBallTree” subroutine (Alg. 2) or the “MakeConeTree” subroutine (Alg. 9). The object contains the current best-match for the query . denotes the lowest affinity between any query in the node and its current best-match. The function denotes the upper bound on the maximum possible inner-product between any query in the node and any point in the node .

In this section, we explore two ways of indexing the queries – (1) indexing the queries using the ball-tree (2) indexing the queries using a novel data structure, the cone-tree. In the following subsections, we derive expressions for each of these kinds of QTree.

4.2 Ball Tree for Queries

Figure 7: Bounding between two balls.
Theorem 4.1.

Given two balls and centered at and with radius and respectively, the maximum possible inner-product with any pair of points and is bounded from above by:

(8)
Proof.

Consider the pair of point be such that

(9)

Let be the angle makes with the vector , and be the corresponding angle in the query ball. Let be the angle between the vectors and and be the angle between the vectors and . Let be the distance between and , be the distance between and . Finally, let be the angle made between and at the origin.

Some facts for the ball (the facts are analogous for the ball ):

Using the triangle inequality of the angles, we know that:

giving us the following:

(10)

Replacing and with and by using the aforementioned equalities (similar to the techniques in proof for theorem 3.1), we have:

(11)
(12)

where the first inequality comes from the definition of and the final inequality comes from the fact that , . ∎

For the dual-tree search algorithm (Alg. 6), the maximum-possible inner-product between two tree nodes and is set as

It is interesting to note that this upper bound bound reduces to the bound in theorem 3.1 when the ball containing the queries is reduced to a single point, implying .

Figure 8: Cone-tree: These cones are open cones and only the angle made at the origin with the axis of the cone is bounded for every point in the cone. The norms of the queries are not bounded at all.

4.3 Cone-trees for Queries

An interesting fact is that in equation 1, the point , where the maximum is achieved, is independent of the norm of the query . Let be the angle between the and at the origin, then the task of searching for the maximum inner-product is equivalent to search for a point such that:

(13)

This implies that we only care about the direction of the queries irrespective of their norms. For this reason, we propose the indexing of the queries on the basis of their direction (from the origin) to form a cone-tree (figure 8). The queries are hierarchically indexed as (possibly overlapping) open cones. Each cone is represented by a vector, which corresponds to its axis, and an angle, which corresponds to the maximum angle made by any point within the cone with the axis at the origin.

4.3.1 Cone-tree Construction

The cone-tree construction is very similar to the ball-tree construction. The only difference is the use of cosine similarity instead of the Euclidean distances for the task of splitting. The cone-tree construction pseudo-code is presented in Figure 8.

  Pick a random point
  
  
  return .
Algorithm 8 MakeConeTreeSplit(Data )
  Input – Set
  Output – Tree
  
  
  
  if   then
     return
  else
     
     
     
     
     
     return
  end if
Algorithm 9 MakeConeTree(Set of items )
Figure 9: Cone-tree Construction: The object denotes the set of points in the node , denotes the Euclidean mean of the items in the node and denotes the cosine of the maximum angle made by any point in the node with at the origin. The angle made between any two points and at the origin is denoted by .

4.3.2 Cone-Ball Bound

Since the norms of the queries do not affect the solution in equation 13, we assume that the norms of the queries are all equal to 1 for convenience.

Figure 10: Bounding between a ball and a cone
Theorem 4.2.

Given a ball of points centered at with radius and a cone with the axis of the cone and aperture555The aperture of the cone is twice the angle made between the axis and the perimeter of the cone. of , the maximum possible inner-product between any pair of points , is bounded from above by:

(14)

where is the angle made between and at the origin and the function .

Proof.

There are two cases to consider here:

For case (i), the center of the ball lies within the cone , implying that

(15)

since there could be some query which is in the same direction as , giving the maximum possible inner-product.

For case (ii), let us assume that without loss of generality. Then . Continuing with the similar notation as in theorem 3.1 & 4.1 for the best pair of points as well as the notation from figure 10, we can say that

(16)

Since is fixed, we can say that

(17)

Expressing and in terms of and , and then subsequently maximizing over and using the fact that , we get that

(18)

Combining case (i) and (ii), we obtain equation 14. ∎

5 Experiments and Results

In this section, we evaluate the efficiency of the two proposed algorithms 5 & 7. For the dual-tree algorithm, we use the two variations – (i) the set of queries indexed as a ball-tree (referred to as Alg. 7(B)), (ii) the set of queries indexed as a cone-tree (referred to as Alg. 7(C)). Since we are not aware of any efficient exact method for maximum inner-product search, we compare our proposed algorithms to the linear search algorithm (Alg. 3). We report the speedup of the proposed algorithms over linear search. Speedup is defined as the ratio of the time taken by the linear search and the time taken by the evaluated algorithm. For the trees, the leaf size can be selected by cross-validation (choosing the leaf size giving the highest speedup). However, for our experiments, we choose a ad hoc value of for all datasets to demonstrate the gain in efficiency without any expensive cross-validation.

Datasets.

We use a variety of datasets from different fields of data mining. We use the following collaborative filtering datasets: MovieLens [17], Netflix [3] and the Yahoo! Music [12] datasets. After the matrix factorization stage, the matrix of item-vectors is used as the reference set and the matrix of user-vectors is used as the set of queries. For text data, we use the LiveJournal blog moods data set [19]. We also use the MNIST digits dataset [24] for evaluation. We also use three astronomy datasets – LCDM [27], PSF and SJ2. A synthetic data set (U-Rand) of uniformly random points in 20 dimensions is used to evaluate the performance of the tree-based algorithms on data sets without any underlying structure. The rest of the datasets are widely used machine learning data sets from the UCI machine learning repository [5]. The details of the datasets are presented in Table 1 and the size of the datasets (in bytes) is presented in figure 11. For the collaborative filtering datasets, there is a clear definition of the reference set (the items) and the set of queries (the users). For the rest of the data sets, we randomly split the datasets into query and reference sets.

Dataset
Bio 74 210,409 75,000
Corel 32 27,749 10,000
Covertype 55 431,012 150,000
LCDM 3 10,777,216 6,000,000
LiveJournal 25,327 121,625 100,000
MNIST 786 60,000 10,000
MovieLens 51 3,706 6,040
Netflix 51 17,770 480,189
OptDigits 64 1,347 450
Pall7 7 100,841 100,841
Physics 78 112,500 37,500
PSF 2 3,056,092 3,056,092
SJ2 2 50,000 50,000
U-Random 20 700,000 300,000
Y!-Music 51 624,961 1,000,990
Table 1: Datasets used for evaluation: The dimensionality and number of points in the reference set and the set of queries .
Figure 11: Total dataset sizes (in bytes): The combined sizes of the reference set and the query set for each data set are presented in this figure.
Tree Construction Times.

The tree-building procedure is extremely efficient. We present the tree construction times in table 2 and contrast them with the runtime of the linear search algorithm (Alg. 3). For some of the larger data sets, the extrapolated runtime of Alg. 3 is reported. In the last column, we present the ratio of the tree construction times with the runtimes of Alg. 3. For algorithm 5 & 7(B), the tree construction involves building one and two ball-trees respectively. For algorithm 7(C), the queries are normalized to have unit length for convenience since the norms of the queries do not affect the answers (equation 13). Following the query normalization, two trees are built. We include the query normalization in the tree construction time for completeness. This is the reason for the significant difference between construction times for algorithm 7(B) and 7(C).

The numbers in the last column of table 2 (R) show how small the construction times are with respect to the actual linear search. The highest ratio is 0.15 for the OptDigits dataset. This implies that any speedup over 1.18 at search time is enough to compensate for the tree construction time. For most of the datasets, this ratio is much lower. Moreover, this tree building cost is a one time cost. Once the tree is built, it can be used for searching the dataset multiple times.

Dataset Alg.5 Alg.7(B) Alg.7(C) Alg.3 R(%)
Bio 4.3 5.7 10.6 4,028 0.25
Corel 0.2 0.27 0.66 43 1.5
Covertype 5.5 7.2 14.8 14,885 0.1
LCDM 36.7 56.46 99.3 1,984,200 0.005
LiveJournal 2223 4073 4745 517,194 0.92
MNIST 8.06 9.1 11.38 817 1.5
MovieLens 0.03 0.08 0.27 4.62 6
Netflix 0.2 8.27 33.5 1,878 1.7
OptDigits 0.01 0.012 0.022 0.135 15
Pall7 0.26 0.52 1.4 364 0.4
Physics 2.33 3.0 5.8 1,114 0.5
PSF 9.06 18.1 34.95 282,514 0.01
SJ2 0.1 0.2 0.46 75 0.6
U-Rand 4.94 6.9 15.64 26,586 0.6
Y! Music 9.72 28.85 112.5 137,306 0.08
Table 2: Tree construction time (in seconds) contrasted with the linear search time (in seconds).
Search Efficiency.

The speedups over linear search are presented in Table 3. We have reported every dataset we evaluated our algorithm on. Overall, the speedup numbers vary from as low as for the OptDigits dataset to over ( orders of magnitude) for the LCDM and the PSF dataset. An important thing to note here is that for datasets with low speedup (below an order of magnitude) with Alg. 5, the speedup numbers for all three algorithms were pretty low and fairly comparable for all three algorithms. However, even a speedup of 2 is pretty significant in terms of absolute times. For example, for the Yahoo! music dataset, a search speedup of mere with a tree construction time of seconds gives a saving of 19 hours of computation time. For most datasets with a high value of speedup for Alg. 5, the speedups for the dual-tree algorithms are also very high.

Dataset Alg.5 Alg.7(B) Alg.7(C)
Bio 7,059.62 6.55 273.52
Corel 14.27 17.38 7.68
Covertype 927.51 10.05 773.34
LCDM 29,526 1,327 101,950
LiveJournal 28.04 10.42 15.45
MNIST 2.61 2.22 2.5
MovieLens 2.23 1.36 1.67
Netflix 1.98 1.92 1.84
OptDigits 1.13 1.10 1.10
Pall7 1,020 23.14 2,285
Physics 4.93 4.0 4.08
PSF 61,502 96,570 125,800
SJ2 544 190 767
U-Rand 3.76 3.18 3.28
Y!-Music 2.11 2.09 2.16
Table 3: Speedups over linear search for .
Figure 12: Speedups over linear search for .

There are three important things to note here. Firstly, the dual-tree algorithms (Alg. 7) do not perform very well if the single-tree algorithms (Alg. 5) does not have a high speedup. This is mostly because the tree is unable to find tight bounds and hence has to travel every branch. The dual-tree scheme loosens the bound to amortize the traversal cost over multiple queries. But if the bounds are bad for algorithm 5, the bounds for the dual-tree are much worse. Hence, the dual-tree algorithm does not show any significant speedup. Secondly, the dual-tree algorithm (especially Alg. 7(C)) starts outperforming the single-tree algorithm significantly when the set of queries is really large. This is a usual behavior for dual-tree algorithms. The query set has to be large enough for the gains from the amortization of query traversal of the reference tree (RTree) to outweigh the computational cost of traversing the query-tree (QTree) itself. Finally, the dual-tree algorithm with ball-trees for the query set is generally significantly slower than the dual-tree with a cone-tree for the queries. There are possibly two possible reasons for that – (i) The cones provide a tighter indexing of the queries than balls. A single cone can be used to index points in multiple balls which lie in the same direction but have varying norms. (ii) The upper bound for in equation 11 is fairly loose. We do provide two ways of obtaining tighter bounds in the Appendix, but we have not yet evaluated the algorithm with the new bounding techniques.

We also consider the general problem of obtaining the points in the set with the highest inner-product with the query . This is analogous to the -nearest neighbor search problem. We present the speedups of our algorithms over linear search for in figure 12.

6 Max-kernel Operation with General Kernel Functions

In this section, we show a method to apply the proposed algorithms in a inner-product space where the inner-products are defined by a kernel function, but it is not possible to explicitly represent the points in the -space.

Without an explicit representation, the tree construction has to be modified since there would be no explicit representation of the mean of a set. For a tree node with the set of point , the mean in -space is defined as

might not have an explicit representation, but it is possible to compute inner products with as follows:

However, this computation is possibly very expensive (as opposed to the operation in equation 4 which is equivalent to a single inner-product). Instead of picking the mean of the set in the -space as the center of the ball, we propose picking the point in the -space which is closest to the mean as the new center. So the new center is given by:

(19)

This operation is quadratic in computation time, but is done at the preprocessing phase to provide efficiency during the search phase. Given this new center , we can compute the radius of the ball enclosing the set as follows:

(20)

Now given this method of choosing the center and evaluating the radius, a ball-tree can be built in the -space using Algorithm 2 without ever requiring the explicit representation of the points. Given this ball-tree, the equation 4 in theorem 3.1 can be modified to this situation as follows:

(21)

where is defined in equation 19 and is defined in equation 20. Computing this upper bound is equivalent to a single kernel function evaluation ( be pre-computed before searching the tree). Using this upper bound, the tree-search algorithm (Alg. 5) can be performed in -space without any explicit representation of the points. We will present the evaluation of this method in the longer version of the paper.

Using the same principles, the dual-tree algorithm (Alg. 7) can also be applied to the -space without any explicit representation of the points. For the dual-tree with ball-tree for the queries, the upper bound on the maximum inner-product between queries in node and points in node in theorem 4.1 becomes:

(22)

where and are the chosen ball centers in the -space with radius and respectively.

For queries indexed in a cone-tree, the central axis of the cone can be the point in the -space making the smallest angle with the mean of the set in the -space. Since the queries are supposed to be normalized in the -space, for a query tree node , the mean of the set is supposed to be:

So the new central axis of the cone is given by:

(23)

Again, this computation is quadratic in the size of the dataset, but provides efficiency during search time. The cosine of half the aperture of the cone is now given by:

(24)

Given and , the upper bound in theorem 4.2 for a cone-tree node of queries and a ball-tree node of reference points is given by:

(25)

where is defined as:

This bound is very efficient to compute as it only requires a single kernel function evaluation (the terms and can be pre-computed and stored in the trees).

7 Conclusion

We consider the general problem of maximum inner-product search and present three novel methods to solve this problem efficiently. We use the tree data structure and present a branch-and-bound algorithm for maximum inner-product search. We further extend it to the case where the set of queries is very large. We evaluate the proposed algorithms with a variety of datasets and exhibit their computational efficiency.

A theoretical analyses of these proposed algorithms would give us a better understanding of the computational efficiency of these algorithms. We do not have any rigorous runtime bounds for our algorithm and it would be part of our future work.

References

  • [1] R. Bayardo, Y. Ma, and R. Srikant. Scaling up all pairs similarity search. In Proceedings of the 16th international conference on World Wide Web. ACM, 2007.
  • [2] R. M. Bell and Y. Koren. Lessons from the netflix prize challenge. SIGKDD Explor. Newsl., 2007.
  • [3] J. Bennett and S. Lanning. The netflix prize. In Proc. KDD Cup and Workshop, 2007.
  • [4] A. Beygelzimer, S. Kakade, and J. Langford. Cover Trees for Nearest Neighbor. Proceedings of the 23rd international conference on Machine learning, 2006.
  • [5] C. L. Blake and C. J. Merz. UCI Machine Learning Repository. http://archive.ics.uci.edu/ml/, 1998.
  • [6] L. Cayton and S. Dasgupta. A learning framework for nearest neighbor search. Advances in Neural Information Processing Systems, 20, 2007.
  • [7] M. S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing. ACM, 2002.
  • [8] P. Ciaccia and M. Patella. PAC Nearest Neighbor Queries: Approximate and Controlled Search in High-dimensional and Metric spaces. Data Engineering, 2000. Proceedings. 16th International Conference on, 2000.
  • [9] K. Clarkson. Nearest-neighbor searching and metric space dimensions. Nearest-Neighbor Methods for Learning and Vision: Theory and Practice, 2006.
  • [10] S. Dasgupta and Y. Freund. Random projection trees and low dimensional manifolds. In Proceedings of the 40th annual ACM symposium on Theory of computing. ACM, 2008.
  • [11] S. C. Deerwester, S. T. Dumais, T. K. Landauer, G. W. Furnas, and R. A. Harshman. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 1990.
  • [12] G. Dror, N. Koenigstein, Y. Koren, and M. Weimer. The yahoo! music dataset and kdd-cup’11. Journal Of Machine Learning Research, 2011.
  • [13] J. H. Freidman, J. L. Bentley, and R. A. Finkel. An Algorithm for Finding Best Matches in Logarithmic Expected Time. ACM Trans. Math. Softw., 1977.
  • [14] A. Gionis, P. Indyk, and R. Motwani. Similarity Search in High Dimensions via Hashing. 1999.
  • [15] A. G. Gray and A. W. Moore. -Body’ Problems in Statistical Learning. In NIPS, 2000.
  • [16] A. G. Gray and A. W. Moore. Nonparametric density estimation: Toward computational tractability. In SIAM Data Mining, 2003.
  • [17] GroupLens. MovieLens dataset.
  • [18] P. Indyk and R. Motwani. Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality. In STOC, 1998.
  • [19] S. Kim, F. Li, G. Lebanon, and I. Essa. Beyond Sentiment: The Manifold of Human Emotions. Arxiv preprint arXiv:1202.1568, 2011.
  • [20] M. Klaas, D. Lang, and N. de Freitas. Fast maximum a posteriori inference in monte carlo state spaces. In Artificial Intelligence and Statistics, 2005.
  • [21] Y. Koren. The bellkor solution to the netflix grand prize. 2009.
  • [22] Y. Koren, R. M. Bell, and C. Volinsky. Matrix factorization techniques for recommender systems. IEEE Computer, 2009.
  • [23] B. Kulis and K. Grauman. Kernelized locality-sensitive hashing for scalable image search. In Computer Vision, 2009 IEEE 12th International Conference on. Ieee, 2009.
  • [24] Y. LeCun. MNist dataset, 2000. http://yann.lecun.com/exdb/mnist/.
  • [25] Z. Li, H. Ning, L. Cao, T. Zhang, Y. Gong, and T. S. Huang. Learning to search efficiently in high dimensions. In Advances in Neural Information Processing Systems 24. 2011.
  • [26] T. Liu, A. W. Moore, A. G. Gray, and K. Yang. An Investigation of Practical Approximate Nearest Neighbor Algorithms. In Advances in Neural Information Processing Systems 17, 2005.
  • [27] R. Lupton, J. Gunn, Z. Ivezic, G. Knapp, S. Kent, and N. Yasuda. The SDSS Imaging Pipelines. Arxiv preprint astro-ph/0101420, 2001.
  • [28] S. M. Omohundro. Five Balltree Construction Algorithms. Technical Report TR-89-063, International Computer Science Institute, December 1989.
  • [29] F. P. Preparata and M. I. Shamos. Computational Geometry: An Introduction. Springer, 1985.
  • [30] A. Rahimi and B. Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 2007.
  • [31] P. Ram, D. Lee, W. March, and A. Gray. Linear-time algorithms for pairwise statistical problems. In Advances in NIPS. 2009.
  • [32] P. Ram, D. Lee, H. Ouyang, and A. G. Gray. Rank-approximate nearest neighbor search: Retaining meaning and speed in high dimensions. In Advances in Neural Information Processing Systems 22. 2009.

Appendix A Tighter Bounds with Optimization

In this section, we present two ways to get a tighter bound on equation 11 with respect to and . The maximum inner product bound between two balls is given as:

(26)

a.1 Two-variable Optimization

Assuming that

we can say that:

(27)
(28)

Now and gives us the following optimality conditions:

(29)
(30)

The second order conditions are the following:

(31)