Higher-order Count Sketch: Dimensionality Reduction That Retains Efficient Tensor Operations
Sketching is a randomized dimensionality-reduction method that aims to preserve relevant information in large-scale datasets. Count sketch is a simple popular sketch which uses a randomized hash function to achieve compression. In this paper, we propose a novel extension known as Higher-order Count Sketch (HCS). While count sketch uses a single hash function, HCS uses multiple (smaller) hash functions for sketching. HCS reshapes the input (vector) data into a higher-order tensor and employs a tensor product of the random hash functions to compute the sketch. This results in an exponential saving (with respect to the order of the tensor) in the memory requirements of the hash functions, under certain conditions on the input data. Furthermore, when the input data itself has an underlying structure in the form of various tensor representations such as the Tucker decomposition, we obtain significant advantages. We derive efficient (approximate) computation of various tensor operations such as tensor products and tensor contractions directly on the sketched data. Thus, HCS is the first sketch to fully exploit the multi-dimensional nature of higher-order tensors. We apply HCS to tensorized neural networks where we replace fully connected layers with sketched tensor operations. We achieve nearly state of the art accuracy with significant compression on the image classification benchmark.
Higher-order Count Sketch: Dimensionality Reduction That Retains Efficient Tensor Operations
Yang Shi University of California, Irvine Irvine, CA 92617 email@example.com Animashree Anandkumar California Institute of Technology Pasadena, CA, 91101 firstname.lastname@example.org
noticebox[b]Preprint. Under review.\end@float
Modern machine learning involves processing of large-scale datasets. Dimensionality-reduction methods attempt to compress the input data while preserving relevant information. Sketching is a popular class of such techniques which aims to reduce memory and computational requirements by using simple randomized hash functions that map the input data to a reduced-sized output space. Count Sketch (CS) (Charikar et al., 2002) is a simple sketch that has been applied in many settings such as estimation of internet packet streams (Demaine et al., 2002) and tracking most frequent items in a database (Cormode and Muthukrishnan, 2005). It uses a simple data-independent random hash function and random signs to combine the input data elements. Despite its simplicity, it enjoys many desirable properties such as unbiased estimation and the ability to approximately perform certain operations directly on the low-dimensional sketched space, e.g., vector inner products and outer products. However, CS is memory inefficient when the data is large. The bottleneck is that it needs to generate a hash table as large as the data size.
Another drawback of CS is that it assumes vector-valued data and does not exploit further structure if data is multi-dimensional. But many modern machine learning and data mining applications involve manipulating large-scale multi-dimensional data. For instance, data can be multi-modal or multi-relational (e.g., a combination of image and text), and intermediate computations can involve higher-order tensor operations (e.g., layers in a tensorized neural network). Memory, bandwidth, and computational requirements are usually bottlenecks when these operations are done at scale. Efficient dimensionality reduction schemes that exploit tensor structures can significantly alleviate this issue if they can find a compact representation while preserving accuracy.
Main contributions: We extend count sketch to Higher-order Count Sketch (HCS), which is the first sketch to fully exploit the multi-dimensional nature of higher-order tensors. It reshapes the input (vector) data to a higher-order tensor of a fixed order. It utilizes multiple randomized hash functions: one for each mode of the tensor. The mapping in HCS is obtained by the tensor product of these hash functions. Figure 1 demonstrates the process. We show a memory saving in storing the hash map: if the input data size is and HCS uses -th order tensor for sketching, we reduce the hash memory requirements from to , compared to the count sketch, under certain conditions.
The conditions for obtaining the best-case memory savings from HCS are related to the concentration of input entries with large magnitudes and require these large entries to be sufficiently spread out. Intuitively, this is because the hash indices in HCS are correlated and we cannot have all the input to be clustered together. If we are allowed multiple passes over the input data , a simple (in-place) reshuffle to spread out the large entries will fix this issue, and thus allows us to obtain maximum memory savings in storing hash functions.
When the input data has further structure as a higher-order tensor, HCS is able to exploit it. HCS allows for efficient (approximate) computations of tensor operations such as tensor products and tensor contractions by directly applying these operations on the sketched components. We obtain exponential saving with respect to the order of the tensor in the memory requirements for tensor product and contraction when compared to sketching using count sketch. We also show times improvement in computation and memory efficiency for computing a -order rank- Tucker tensor when compared to applying CS to each rank-1 component. The computation and memory improvement over CS of these operations are shown in Table 1.
We compare HCS and CS for tensor product and tensor contraction compression using synthetic data. HCS outperforms CS in terms of computation efficiency and memory usage: it uses less compression time and less memory while keeping the same recovery error, compared to CS. Besides, we apply HCS for approximating tensor operations in tensorized neural networks. These networks replace fully connected layers with multi-linear tensor algebraic operations. Applying HCS to tensor operations results in further compression while preserving accuracy. We obtain test accuracy on CIFAR10 dataset with memory savings on the last fully connected layer, compared to the baseline ResNet18.
Related work: Singular value decomposition (SVD) is perhaps the most popular dimensionality reduction technique (Eckart and Young, 1936). However, when data is not inherently low rank or has other constraints such as sparsity and non-negativity, SVD is not suitable. Other matrix decomposition techniques try to impose more structure on matrices (Mahoney and Drineas, 2009; Caiafa and Cichocki, 2010; Praneeth Netrapalli, 2014).
In contrast to matrix techniques which make stringent assumptions on underlying structure, sketching is designed for compressing vector-valued data with almost no assumptions (Bringmann and Panagiotou, 2017; Alon et al., 1999; Weinberger and Saul, 2009). Count Sketch (CS) (Charikar et al., 2002) was proposed to estimate the frequency of each element in a stream. Pagh (2012) propose a fast algorithm to compute CS of an outer product of two vectors using FFT properties. They prove that the CS of the outer product is equal to the convolutions between the CS of each input. This allows for vector operations such as inner product and outer product to be directly computed in the sketch space. Since then, many variants of count sketch have been proposed that preserve different properties of underlying data. Min-hash (Broder, 1997) is a technique for estimating how similar two sets of data are. An extension of that is one-bit CP-hash (Christiani et al., 2018) which generates concurrent hash table for multi-core processors. To make use of parallel computing resources, 2-of-3 cuckoo hashing (Amossen and Pagh, 2011) is proposed based on cuckoo hashing (Pagh and Rodler, 2001).
Sketching can also be applied to multi-dimensional data. Tensor sketch (Pham and Pagh, 2013) is proposed to approximate non-linear kernels. It has been applied to approximately compute tensor CP decomposition (Wang et al., 2015; Yang et al., 2018) and Tucker decomposition (Malik and Becker, 2018). Gao et al. (2016) introduce compact bilinear pooling to estimate joint features from different sources. In Visual Question Answering task, people use compact bilinear pooling to compute joint features from language and image (Fukui et al., 2016). However, all these sketching techniques are sketching tensors into a vector, which destroys their multi-dimensional structure. This does not make it possible to do tensor operations efficiently in the sketched space.
In addition to sketching, efficient multi-dimensional data operation primitives can boost the computation performance. A Low-overhead interface is proposed for multiple small matrix multiplications on NVIDIA GPUs (Jhurani and Mullowney, 2015). Ma et al. (2019); Shi et al. (2016) optimize tensor matrix contraction on GPUs by avoiding data transformation. High-Performance Tensor Transposition (Springer et al., 2017) is one of the open-source library that performs efficient tensor contractions. In future, we can leverage these advances to further speed up tensor sketching operations.
Important tensor applications: We focus on tensor sketching because data is inherently multi-dimensional in many settings. In probabilistic model analysis, tensor decomposition is the crux of model estimation via the method of moments. A variety of models such as topic models, hidden Markov models, Gaussian mixtures etc., can be efficiently solved via the tensor decomposition techniques under certain mild assumptions (Anandkumar et al., 2014). Papalexakis and Pelechrinis (2018) analyze spatio-temporal basketball data via tensor decomposition. Tensor methods are also relevant in deep learning. Yu et al. (2017) learn the nonlinear dynamics in recurrent neural networks directly using higher-order state transition functions through tensor train decomposition. Kossaifi et al. (2017) propose tensor contraction and regression layers in deep convolutional neural networks.
Count Sketch Count Sketch(CS) (Charikar et al., 2002) was first proposed to estimate most frequent data value in a streaming data.
Definition 2.1 (Count Sketch).
Given two 2-wise independent random hash functions h: and s:. Count Sketch of a vector is denoted by where .
In matrix format, we can write it as , where , , if , for , otherwise , is the sign for element-wise product. The estimation can be made more robust by taking independent sketches of the input and calculating the median of the estimators. Charikar et al. (2002) prove that the CS is an unbiased estimator with variance bounded by the 2-norm of the input. See Appendix B.1 for detailed proof. Pagh (2012) use CS and propose a fast algorithm to compute count sketch of an outer product of two vectors.
The convolution operation (represented using ) can be transferred to element-wise product using FFT properties. Thus, the computation complexity reduces from to , if the vectors are of size and the sketching size is .
Some notations we use in the following paper are: : decompression of , : set of .
3 Higher-order count sketch on vector-valued data
We denote vectors by lowercase letters, matrices by uppercase letters, and higher-order tensors by calligraphic uppercase letters. The order of a tensor is the number of modes it admits. For example, is an -order tensor because it has N modes. A fiber is the higher-order analogue of a matrix row or column in tensors. We show different ways to slice a third-order tensor in Figure 2. The p-mode matrix product of a tensor with a matrix is denoted by and is of size . Element-wise it calculates: .
Higher-order count sketch(HCS) Given a vector , random hash functions : , , random sign functions : , , and , we propose HCS as:
where . This is the index mapping between the vector with its reshaping result—a -order tensor with dimensions on each mode, for .
Using tensor operations, we can denote HCS as:
Here, , , , if , otherwise , for . The can be done in-place. We assume is a vectorized layout of a -order tensor.
To recover the original tensor, we have
Assume is a -order tensor by fixing modes of a -order tensor as shown in Figure 2:
Theorem 3.1 (HCS recovery analysis).
Given a vector , assume is the maximum frobenium norm of all , Equation 4 computes an unbiased estimator for with variance bounded by:
Remarks Compared to CS, HCS requires less space for storing the hash functions. Each mode only requires a sized hash matrix with nonzero entries (). Thus, HCS required for hash memory while CS requires . If we choose , then and we save memory from using HCS.
The dominant term in Equation 5 will be as long as all large entries are not clustered close to one another. Notice that CS has variance bounded by . We require to guarantee the same recovery, and that will lead to a total output memory for HCS. However, due to the correlations between hash indices across different modes, HCS needs larger sketching space compared to CS if large entries are clustered. In the worst case, when large magnitude data all locate in one fiber of , HCS has variance bounded by . We require for the same recovery error. HCS output is of size while CS output’s size is .
We present a simple way to reshuffle the data in-place. Step1: Sort in descending order. Step2: Rearrange sorted array in designed space such that it goes diagonally from top to bottom and then consecutive anti-diagonally from bottom to top. Step3: Rearrange the data according to Step2 (column-wise fiber by fiber). We assume all data is well distributed in the rest analysis.
Another concern in HCS is how to choose the order of the reshaping tensor (parameter ). If the data values are fairly evenly distributed, we should select as large as possible (but sublinear in ).
4 Higher-order count sketch on tensors
In the previous section, we discuss how to sketch a vector-valued data using HCS. In this section, we focus on tensor-valued data. In order to use CS on higher-order tensors, we either view the tensor as a set of vectors and sketch along each fiber of the tensor or we flatten the tensor as a vector and apply CS on it. Hence, CS do not exploit tensors. Moreover, operations between tensors have to be performed on sketched vectors. This process is inefficient. But, with the help of HCS, we can compute various operations such as tensor products and tensor contractions by directly applying operations on the sketched components.
It is straightforward to apply HCS on tensors. Given a tensor , random hash functions : , , and random sign functions : , , HCS computes: . To recover the original tensor, we have: . and are defined as same as in Section 3.
4.1 Tensor product
Tensor product is known as outer product in vectors case. It computes every bilinear composition from inputs. We denote the operation with . The tensor product result has dimension equal to the product of the dimensions of the inputs. It has been used in a wide range of applications such as bilinear models (Tenenbaum and Freeman, 2000). Pagh (2012) shows that the count sketch of an outer product equals the convolution between the count sketch of each input vector as shown in Equation 1. Furthermore, the convolution in the time domain can be transferred to the element-wise product in the frequency domain. We extend the outer product between vectors to tensor product.
Given a -order tensor , a -order tensor , assume :
and are p-dimensional Fourier transform and inverse Fourier transform if the input is a -order tensor. The proof is given in Appendix B.2.
We use the Kronecker product, which is a generalization of the outer product from vectors to matrices to compare tensor product approximation using HCS and CS.
Assume inputs are : Given Lemma 4.1, this approximation requires to complete , and to complete 2D Fourier Transform if the HCS sketching parameter is for each mode. It requires memory for final representation and for hashing parameters along each mode.
Baseline CS operation We flatten and as vectors and apply CS on the vector outer product. The computation complexity is and the memory complexity is . It needs for hashing memory because we have elements in the vectorized matrix.
HCS requires approximately times less memory comparing to CS for two matrix Kronecker product. See Table 1 for detailed comparisons.
4.2 Tensor contraction
Matrix product between and is defined as , . Tensor contraction (used more often as Einstein summation in physics community) can be seen as an extension of matrix product in higher-dimensions. It is frequently used in massive network computing. We define a general tensor contraction between and as
where are ordered sequences of indices such that , , , , , . The indices in are called contracted indices. The indices in are called free indices.
Given tensors , , contraction indices , if hash matrices , :
We require the hash matrices for the contraction modes be identity matrices. In other words, we are not compressing along the modes that are being multiplied. The proof is in Appendix B.3.
4.3 Tucker-form tensor
Tensor decomposition is an extension of matrix decomposition to higher-orders. The Tucker decomposition (Tucker, 1966) is analogous to principal component analysis. It decomposes a tensor as a core tensor contracted with a matrix along each mode. For instance, a third-order tensor has the Tucker decomposition: , where , , , . CANDECOMP/PARAFAC(CP) (Harshman, 1970) is a special case of a Tucker-form tensor, where the core tensor is a sparse tensor that only has non-zero values on the superdiagnoal. It can be represented as a sum of rank-1 tensors: . Tensor decomposition has been applied in many field such as data mining (Kolda and Sun, 2008) and latent variable models (Anandkumar et al., 2014).
Given a Tucker tensor , where : The higher-order CS of a Tucker-form tensor can be accomplished by performing HCS on each factor:
Baseline CS operation To apply CS to a Tucker-form tensor, we rewrite the decomposition as a sum of rank-1 tensors:
where are column of respectively.
We show computation and memory analysis in Table 1. In addition, a CP-form tensor can be sketched in the same way as we described above when using HCS. For using CS: instead of summing over all values, we sum over only number of values. The analysis can also be easily extended to higher-order tensors.
We summarize the general tensor product and tensor contraction estimation process in Table 4.3.
|Tensor Product: ,|
|Tensor Contraction: , with contraction indices|
The goals of this section are: evaluate HCS for data compression; demonstrate the advantages of HCS in various tensor operation estimations, compared to CS; present potential application of HCS in deep learning tasks. All synthetic data experiments are run on a MAC with Intel Core i5 processor. Section 5.3 is run on a NVIDIA DGX station with Tesla V100 Tensor Core GPU.
5.1 HCS for unevenly-distributed data
In Section 3, we point out that unevenly-distributed data value may affect the performance of HCS. We generate a matrix , where every entry is sampled from a uniform distribution between and , except the elements in the second column, which are filled with value . We compress this data using HCS and CS. The compression ratio is calculated as where is the sketching dimension along each two mode for HCS. CS sketching dimension . We rearrange the data so that values in the matrix are evenly distributed, and we run the HCS and CS again. We compare the relative error() in Figure 3. HCS performs poorly before rearranging the data. But after the rearrangment, HCS performs as good as CS, which corresponds to our analysis.
5.2 Tensor operations
Kronecker product: We compress Kronecker products using HCS and CS. We compute , where . All inputs are randomly generated from the uniform distribution[-5,5]. The result is obtained by independently running the sketching times and choosing the median. Keeping the same compression ratio, HCS has slightly higher recovery error than CS. But HCS is systematically better in computation speed and memory usage compared to CS in Figure 4.
Tensor contraction: Given , , we compute : the third mode of contract with the first mode of . We compress and decompress the contraction as demonstrated in Section 4.2. All entries are sampled independently and uniformly from [0,10]. We repeat the sketching 20 times and use the median as the final estimation. Overall, HCS outperforms CS in time, memory and recovery error aspects as shown in Figure 5. When the compression ratio is , HCS is x faster than CS and uses x less memory while keeping almost the same recovery error. HCS is more efficient in real computation because it performs compact contraction in matrix/tensor format, while CS requires computation on each slide of the input tensor.
5.3 Tensor regression network
To demonstrate the versatility of our method, we combine it by integrating it into a tensor regression network for object classification. Tensor regression layer (TRL) (Kossaifi et al., 2017) is proposed to learn a Tucker-form tensor weight for the high-order activation tensor. We sketch the Tucker tensor weight using Equation 10. We use a ResNet18, from which the fully-connected layers are removed, and are replaced by our proposed sketched tensor regression layer. The network structure is illustrated in Figure 5.2. The space saving is calculated as where and are the number of parameters in the last layer in the tensorized network and the baseline. In Figure 7, tensorized network outperforms the baseline(original Resnet18) while using less memory in the last layer. With HCS, we can further reduce more memory requirement while keeping the prediction as good as the baseline.
6 Conclusion and future work
In this paper, we extend count sketch to a new sketching technique, called higer-order count sketching (HCS). HCS gains an exponential saving (with respect to the order of the tensor) in the memory requirements of the hash functions and allows efficient approximation of various tensor operations such as tensor products and tensor contractions. Some interesting future works are how to choose the optimal tensor order for input (vector) data when we have limited information about the data and how to extend other hash algorithms such as simhash (Charikar, 2002), minhash (Broder, 1997) and cuckoo hashing (Pagh and Rodler, 2001) to tensors. We are also interested in analyzing the performance differences on sparse and dense tensors using various sketching techniques. Providing HCS implementations within computation platforms such as MKL and CUDA is also part of the future work.
- Alon et al. (1999) Alon, N., Matias, Y., and Szegedy, M. (1999). The space complexity of approximating the frequency moments. Journal of Computer and System Sciences, 58(1):137–147.
- Amossen and Pagh (2011) Amossen, R. R. and Pagh, R. (2011). A new data layout for set intersection on gpus. In Proceedings of the 2011 IEEE International Parallel & Distributed Processing Symposium, IPDPS ’11, pages 698–708, Washington, DC, USA. IEEE Computer Society.
- Anandkumar et al. (2014) Anandkumar, A., Ge, R., Hsu, D., Kakade, S. M., and Telgarsky, M. (2014). Tensor decompositions for learning latent variable models. The Journal of Machine Learning Research, 15(1):2773–2832.
- Bringmann and Panagiotou (2017) Bringmann, K. and Panagiotou, K. (2017). Efficient sampling methods for discrete distributions. Algorithmica, 79(2):484–508.
- Broder (1997) Broder, A. Z. (1997). On the resemblance and containment of documents. IEEE:Compression and Complexity of Sequences: Proceedings, Positano, Amalfitan Coast, Salerno, Italy,, 10:21–29.
- Caiafa and Cichocki (2010) Caiafa, C. F. and Cichocki, A. (2010). Generalizing the column–row matrix decomposition to multi-way arrays. Linear Algebra and its Applications, 433:557–573.
- Charikar et al. (2002) Charikar, M., Chen, K., and Farach-Colton, M. (2002). Finding frequent items in data streams. In Proceedings of ICALP’02, pages 693–703.
- Charikar (2002) Charikar, M. S. (2002). Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry-fourth Annual ACM Symposium on Theory of Computing, STOC ’02, pages 380–388, New York, NY, USA. ACM.
- Christiani et al. (2018) Christiani, T., Pagh, R., and Sivertsen, J. (2018). Scalable and robust set similarity join. The annual IEEE International Conference on Data Engineering.
- Cormode and Muthukrishnan (2005) Cormode, G. and Muthukrishnan, S. (2005). What’s hot and what’s not: Tracking most frequent items dynamically. ACM Trans. Database Syst., 30(1):249–278.
- Demaine et al. (2002) Demaine, E. D., López-Ortiz, A., and Munro, J. I. (2002). Frequency estimation of internet packet streams with limited space. In Proceedings of the 10th Annual European Symposium on Algorithms, ESA ’02, pages 348–360, London, UK, UK. Springer-Verlag.
- Eckart and Young (1936) Eckart, C. and Young, G. (1936). The approximation of one matrix by another of lower rank. In Psychometrika. Springer-Verlag.
- Fukui et al. (2016) Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., and Rohrbach, M. (2016). Multimodal compact bilinear pooling for visual question answering and visual grounding. EMNLP 2016.
- Gao et al. (2016) Gao, Y., Beijbom, O., Zhang, N., and Darrell, T. (2016). Compact bilinear pooling. Computer Vision and Pattern Recognition (CVPR), 2016.
- Harshman (1970) Harshman, R. (1970). Foundations of the parafac procedure: Models and conditions for an explanatory multi-model factor analysis. UCLA Working Papers in Phonetics, 16:1–84.
- Jhurani and Mullowney (2015) Jhurani, C. and Mullowney, P. (2015). A gemm interface and implementation on nvidia gpus for multiple small matrices. J. of Parallel and Distributed Computing, pages 133–140.
- Kolda and Sun (2008) Kolda, T. G. and Sun, J. (2008). Scalable tensor decompositions for multi-aspect data mining. ICDM.
- Kossaifi et al. (2017) Kossaifi, J., Lipton, Z. C., Khanna, A., Furlanello, T., and Anandkumar, A. (2017). Tensor regression networks.
- Ma et al. (2019) Ma, Y., Li, J., Wu, X., Yan, C., Sun, J., and Vuduc, R. (2019). Optimizing sparse tensor times matrix on gpus. Journal of Parallel and Distributed Computingg, pages 99–109.
- Mahoney and Drineas (2009) Mahoney, M. W. and Drineas, P. (2009). Cur matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences, 106(3):697–702.
- Malik and Becker (2018) Malik, O. A. and Becker, S. (2018). Low-rank tucker decomposition of large tensors using tensorsketch. Neural Information Processing Systems.
- Pagh (2012) Pagh, R. (2012). Compressed matrix multiplication. ITCS.
- Pagh and Rodler (2001) Pagh, R. and Rodler, F. F. (2001). Cuckoo hashing. Lecture Notes in Computer Science.
- Papalexakis and Pelechrinis (2018) Papalexakis, E. and Pelechrinis, K. (2018). thoops: A multi-aspect analytical framework for spatio-temporal basketball data. ACM CIKM.
- Pham and Pagh (2013) Pham, N. and Pagh, R. (2013). Fast and scalable polynomial kernels via explicit feature maps. KDD.
- Praneeth Netrapalli (2014) Praneeth Netrapalli, U N Niranjan, S. S. A. A. P. J. (2014). Non-convex robust pca. Conference on Neural Information Processing Systems.
- Shi et al. (2016) Shi, Y., Niranjan, U., Anandkumar, A., and Cecka, C. (2016). Tensor contractions with extended blas kernels on cpu and gpu. HiPC.
- Springer et al. (2017) Springer, P., Su, T., and Bientinesi, P. (2017). Hptt: A high-performance tensor transposition c++ library. Proceedings of the 4th ACM SIGPLAN International Workshop on Libraries, Languages, and Compilers for Array Programming.
- Tenenbaum and Freeman (2000) Tenenbaum, J. B. and Freeman, W. T. (2000). Separating style and content with bilinear models. Neural computation, 12(6):1247–1283.
- Tucker (1966) Tucker, L. R. (1966). Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279–311.
- Wang et al. (2015) Wang, Y., Tung, H.-Y., Smola, A., and Anandkumar, A. (2015). Fast and guaranteed tensor decomposition via sketching. Proceedings of Advances in Neural Information Processing Systems (NIPS).
- Weinberger and Saul (2009) Weinberger, K. Q. and Saul, L. K. (2009). Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10:207–244.
- Yang et al. (2018) Yang, B., Zamzam, A., and Sidiropoulos, N. D. (2018). Parasketch: Parallel tensor factorization via sketching. SIAM International Conference on Data Mining.
- Yu et al. (2017) Yu, R., Zheng, S., Anandkumar, A., and Yue, Y. (2017). Long-term forecasting using tensor-train rnns.
Appendix A List of some algorithms mentioned in the main paper
a.1 Count sketch
a.2 Higher-order count sketch
a.3 Approximate Kronecker product
a.4 Approximate Matrix product
Appendix B Proofs of some technical theorems/lemmas
b.1 Analysis of CS and HCS approximation error
Theorem B.1 (Charikar et al. (2002)).
Given a vector , CS hashing functions s and h with sketching dimension c, for any , the recovery function computes an unbiased estimator for with variance bounded by .
Proof of Theorem b.1.
For let be the indicator variable for the event . We can write as
Observe that , if , , for all , and , we have
To bound the variance, we rewrite the recovery function as
To simplify notation, we assign X as the first term, Y as the second term. , and since for are 2-wise independent. Thus,
for . Consequently,
The last equality uses that , for all . Summing over all terms, we have . ∎
Proof of Theorem 3.1.
For simplicity, we assume is reshaped into a second-order tensor in the following proof. But the analysis can be extended to reshaping into any order tensor.
For , , let be the indicator variable for the event and . We can write as
Notice that , we know the index mapping: , where . Observe that , if , . , , for all , , and , , we have
To bound the variance, we rewrite the recovery function as
To simplify notation, we assign X as the first term, Y as the second term. , and since and for , are both 2-wise independent. Thus,
for or . Therefore, Equation 20 becomes:
This is because , for all , . , for all , . , for all , .
If any fiber of has extreme large data value, or , where is any row or column of , we can omit the first term, . Otherwise, if , we can omit the second and third terms and . ∎
b.2 HCS of the Kronecker product
For simplicity, we show proof for Kronecker product here. But this can be extended to general tensor product.
Given two matrices , ,
Proof of Lemma b.2.
The Kronecker product defines . Thus:
where , , , .
Assign , , , , and , we have
Consequently, we have . The recovery map is for , , ,