Private Machine Learning in TensorFlow using Secure Computation
We present a framework for experimenting with secure multi-party computation directly in TensorFlow. By doing so we benefit from several properties valuable to both researchers and practitioners, including tight integration with ordinary machine learning processes, existing optimizations for distributed computation in TensorFlow, high-level abstractions for expressing complex algorithms and protocols, and an expanded set of familiar tooling. We give an open source implementation of a state-of-the-art protocol and report on concrete benchmarks using typical models from private machine learning.
Several fields come together in private machine learning: cryptography, machine learning, distributed systems, and high-performance computing. As a result, researchers and practitioners developing scalable solutions may find themselves faced with a task requiring many diverse skill sets and expertise.
Adapting machine learning models in a way that allows for privacy-preserving prediction and training is complex and non-trivial on its own, and often requires deeper insight into cryptography or machine learning than experts from either field have about the other. For instance, one may want to mix secure computation protocols with complementary properties and experiment with various cryptographic optimizations such as vectorization and other specializations. Simultaneously, one may want to find variants of machine learning components that give sufficient accuracy yet utilize the fact that some secure operations are significantly faster than others. Thus, modularity and extensibility are crucial for effective experimentation.
One may additionally be faced with a large implementation challenge, potentially having to stitch together several otherwise independent systems or re-implement complex methods from both fields. One must not only optimize local computations by taking advantage of diverse multi-core architectures, but also the distributed processes inherent in secure computation, including how best to orchestrate the joint execution on multiple machines and minimize the overhead of sending large amounts of data across the network. Building all of this at the right level of abstraction can be overwhelming, and often comes at the cost of extensibility and code readability. This makes experimentation harder, potentially harming accessibility and correctness.
Finally, for even small-scale solutions it is highly valuable to have access to tools for visual inspection, debugging, and profiling such as TensorBoard in order to identify issues and bottlenecks in both the protocol and the machine learning model. Lack of such tools represents an additional barrier to entry.
In this paper we illustrate the benefits of embedding protocols directly into TensorFlow in order to show that it can serve as a platform for easily experimenting with secure computation for private machine learning.
We present tf-encrypted, an open source library built on top of TensorFlow with the aim of making private machine learning more accessible to researchers and practitioners coming from either cryptography or machine learning, and without the need to be an expert in distributed systems or high-performance computing. A small example is given in Figure 1, where a logistic regression model is used for making private predictions. Note that the methods processing input and output specify on which party they should run, and that usual TensorFlow operations may be used within to perform local plaintext operations.
To this end we adapt and implement a state-of-the-art secure computation protocol for tensor oriented applications (Section 2). We report on benchmarks using common models from the literature (Section 3) and highlight additional properties of this approach that we find of value: usability: by leveraging TensorFlow we obtain a familiar and comprehensive platform for building scalable solutions; integration: by reducing all secure computations to TensorFlow graphs, it becomes straight-forward to mix these with ordinary computations for hybrid approaches; extensibility: using TensorFlow’s high-level abstractions makes it easier to experiment with and develop new secure protocols on top of optimized primitives while maintaining code readability; performance: we reach high runtime efficiency without sacrificing other properties via TensorFlow’s distributed execution engine heavily optimized for networking, parallel execution, and scalability; benchmarking: combining all of the above we obtain a common framework for comparable private machine learning.
1.2 Related Work
Several freely available implementations of secure computation protocols exist, including those of Smart et al. (); Nordholt et al. (); Demmler et al. (); Zahur and Evans (), yet all of these are standalone frameworks that do not provide integration with existing machine learning platforms, and arguably aim more at general purpose secure computation than our focus on private machine learning. As a result, users are faced with a lower level interface and may have to implement basic machine learning components. We are furthermore not aware of any development tools for these frameworks outside of general purpose debuggers. To the best of our knowledge no machine learning support tools exist for these platforms.
The works of Mohassel and Zhang (2017); Juvekar et al. (2018); Wagh et al. (2018); Mohassel and Rindal (2018) focus on adapting secure computation protocols to private machine learning, in some cases using similar optimizations as the protocol presented in Section 2. However, to the best of our knowledge none of these have openly available implementations and hence require a significant investment from anyone wanting to apply them. To a large extent our aim is to provide a common platform based on TensorFlow for implementing and experimenting with protocols such as these.
2 Secure Computations in TensorFlow
TensorFlow as described by Abadi et al. (2016b) is among the leading frameworks for constructing and deploying machine learning models, offering an optimized engine for executing local and distributed computations as well as a high-level interface for expressing these. Importantly, the latter abstracts away lower-level operations such as networking while remaining powerful enough to succinctly express computations with operations and tensors pinned to specified machines (see for example Figure 2). This combines for a powerful platform for giving efficient implementations of distributed computations, including complex secure computations where it is crucial for privacy that some data remain known only on select machines.
Moreover, the link between TensorFlow’s engine and high-level interface takes the form of stateful dataflow graphs, with nodes for performing tensor operations in the distributed setting. This means that the engine can not only take advantage of optimizations such as lazy evaluation and multi-core processing tailored for the specific runtime environment, but also optimize the graph itself and chose a node execution order based on both static and runtime information (see e.g. Abadi et al. (2016a)). For instance, we observe that large network transfers from Beaver triple generation are often automatically batched and moved to the beginning of the execution.
As an example of a secure computation protocol implemented in TensorFlow, we here outline our variant of the well-known SPDZ protocol by Damgård et al. (2012) with two servers and that is used for benchmarking in Section 3. The protocol is vectorized to improve performance of applications relying heavily on tensor operations and we use generalized triples produced by an independent third server
Following typical practice we use a fixed point encoding for the floating point numbers commonly used in machine learning, i.e. we scale by a fixed factor and treat the result as an integer. To represent these we support both a fixed int64 and a CRT-based int100 tensor type, the former offering higher performance and the latter higher precision. To maintain precision after multiplications we implement both the conservative truncation protocol of Catrina and Saxena (2010) requiring one round of communication, as well as the optimistic non-interactive protocol of Mohassel and Zhang (2017) that may fail with a small probability.
As in other SPDZ variants, private tensors are secret shared into two tensors and , held by server and respectively, such that yet either share on its own reveals nothing about . However, unlike other variants, we also rely on a masked tensor which in addition to and also includes a random tensor held by , shares and of it held by and , and an held by both and such that . While this is simply an explicit representation of the intermediate state of a SPDZ multiplication, having it in this form allows us to easily extend and generalize triples in order to reduce computation and networking. Converting a tensor from private to masked takes one round of offline communication where is sampled by and shares and of it are sent to and , and one round of online communication where is sent by to for and is computed by both.
Secure computation on tensors proceed as in other SPDZ variants, albeit with some operations first converting private tensors into masked tensors. For instance, multiplication is only implemented for masked tensors as with and where is computed and shared by (see Figure 2). Note that the result of a multiplication is a private tensor, meaning will have to be masked before it can be used as input to another multiplication; however, both and can readily be used again as is, meaning every tensor only needs to be masked once. While special triples for squaring are no longer needed with explicit masking, we can still optimize e.g. matrix multiplications and convolutions via specialized triples and avoid reducing everything to (scalar) multiplications, in turn reducing networking further. Finally, operations such as transposing and stacking are done locally without interaction by letting all three servers operate on the values associated with private and masked tensors.
We benchmark the protocol of Section 2 for private inference on the typical MNIST handwritten digit classification task. Using TensorFlow we train each of the neural networks
|Network A||Network B||Network C|
The left part of Table 1 summarizes the combined offline and online runtime averaged over 100 inferences. As seen we get reasonable performance that may already be adequate for concrete applications. The fact that int100 also achieves good performance suggests that we are not limited by the lower precision of int64 when looking at larger models. Additional experiments furthermore indicate sub-linearly scaling with respect to typical batch sizes, leading to an interesting trade-off between latency and through-put; for instance, Network C with batch sizes 1, 10, and 100 take respectively 124, 182, and 541ms.
We also compare our accuracy over the entire testing set against plaintext TensorFlow. As seen in the right part of Table 1 we obtain almost identical accuracy using both int64 and int100, with indications that the latter may give a slightly better output distribution according to the mean KL divergence, in turn potentially justifying its higher runtime cost. We note that these models achieve good performance despite using approximations, further underlining the importance of being able to adapt models to the encrypted setting.
Based on the above we conjecture that our approach scales to larger models, and is at least within an order of magnitude of related work in terms of runtime performance (at what we believe to be a lower implementation cost). We defer proper exploration of both topics to the full version of this paper.
|Runtime average||Runtime deviation||Accuracy||KL divergence|
We have proposed an open source framework for experimenting with secure computation in TensorFlow, and illustrated how implementation of such protocols can be easily expressed using high level abstractions (Figure 2). This additionally allows private machine learning to be expressed in an interface similar to ordinary TensorFlow (Figure 1) while maintaining good performance.
In the full version of this paper we elaborate on these results, and present a modular extension of the concrete protocol presented here that adds features from Wagh et al. (2018) in order to compute exact ReLU and MaxPooling.
- This server is essentially the crypto producer as used in e.g. Mohassel and Zhang (2017) that can run entirely offline as long as the function to compute is known, which can be fully determined at compile time due to TensorFlow’s use of static dataflow graphs.
- These are variants of models studied in Mohassel and Zhang (2017); Juvekar et al. (2018); Wagh et al. (2018). We use polynomials interpolated to fit ReLU on the interval for activation functions and perform the final argmax and softmax on logits in plaintext to avoid computing these securely.
- Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR, abs/1603.04467, 2016a. URL http://arxiv.org/abs/1603.04467.
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A System for Large-scale Machine Learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI), 2016b.
- Octavian Catrina and Amitabh Saxena. Secure computation with fixed-point numbers. In Financial Cryptography and Data Security. Springer Berlin Heidelberg, 2010.
- Ivan Damgård, Valerio Pastro, Nigel P. Smart, and Sarah Zakarias. Multiparty Computation from Somewhat Homomorphic Encryption. In Advances in Cryptology - CRYPTO 2012 - 32nd Annual Cryptology Conference, Santa Barbara, CA, USA, 2012.
- Daniel Demmler, Thomas Schneider, and Michael Zohner. ABY - A Framework for Efficient Mixed-Protocol Secure Two-Party Computation.
- Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. Gazelle: A Low Latency Framework for Secure Neural Network Inference. 2018.
- Payman Mohassel and Peter Rindal. ABY3: A Mixed Protocol Framework for Machine Learning. 2018.
- Payman Mohassel and Yupeng Zhang. SecureML: A System for Scalable Privacy-Preserving Machine Learning. In 2017 IEEE Symposium on Security and Privacy (SP), 2017.
- Peter Sebastian Nordholt, Kasper Damgård, Peter F. Frandsen, and Jonas Lindstrøm. FRESCO: Framework for Efficient and Secure Computation. https://github.com/aicis/fresco.
- Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755, 2017.
- Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable Private Learning with PATE. arXiv preprint arXiv:1802.08908, 2018.
- Nigel P. Smart, Marcel Keller, Dragos Rotaru, and Peter Scholl. SCALE-MAMBA. https://homes.esat.kuleuven.be/~nsmart/SCALE/.
- Sameer Wagh, Divya Gupta, and Nishanth Chandran. SecureNN: Efficient and Private Neural Network Training. IACR ePrint Archive, Report 2018/442, 2018.
- Samee Zahur and David Evans. Obliv-C: A Language for Extensible Data-Oblivious Computation. https://oblivc.org/.