Generating Differentially Private Datasets Using GANs
Abstract
In this paper, we present a technique for generating artificial datasets that retain statistical properties of the real data while providing differential privacy guarantees with respect to this data. We include a Gaussian noise layer in the discriminator of a generative adversarial network to make the output and the gradients differentially private with respect to the training data, and then use the generator component to synthesise privacypreserving artificial dataset. Our experiments show that under a reasonably small privacy budget we are able to generate data of high quality and successfully train machine learning models on this artificial data.
Generating Differentially Private Datasets Using GANs
Aleksei Triastcyn 

Artificial Intelligence Laboratory 
Ecole Polytechnique Fédérale de Lausanne 
Lausanne, Switzerland 
aleksei.triastcyn@epfl.ch 
Boi Faltings 

Artificial Intelligence Laboratory 
Ecole Polytechnique Fédérale de Lausanne 
Lausanne, Switzerland 
boi.faltings@epfl.ch 
1 Introduction^{†}^{†}This paper is the corrected version of the ICLR 2018 submission (https://openreview.net/forum?id=rJv4XWZA)
Following recent advancements in deep learning (Silver et al., 2016; He et al., 2015; Wu et al., 2016), more and more people and companies are interested in putting their data in use as they see that machine learning is able to generate a wide range of benefits, including financial, social, medical, security, and so on. At the same time, however, such models are often able to capture a fine level of detail in training data potentially compromising privacy of individuals who’s features sharply differ from others. This problem is partially mitigated by the use of regularisation techniques that “smooth out” outstanding details and avoid overfitting, but it does not give any theoretical privacy guarantees. Recent research by Fredrikson et al. (2015) suggests that even without access to internal model parameters, by using hill climbing on output probabilities of a neural network, it is possible to recover (up to a certain degree) individual faces from a training set.
The latter result is especially disturbing knowing that deep learning models are becoming an integral part of our lives, making its way to phones, smart watches, cars, and appliances. And since these models are often trained on customers data, such training set recovery techniques will endanger privacy even without access to the manufacturer’s servers where these models are being trained.
In order to protect privacy while still benefiting from the use of statistics and machine learning, a number of techniques for data anonymisation has been developed over the years, including anonymity (Sweeney, 2002), diversity (Machanavajjhala et al., 2007), closeness (Li et al., 2007), and differential privacy (Dwork, 2006; Dwork et al., 2006; Dwork, 2008; Dwork et al., 2014). The latter has been recognised as a strong standard and is widely accepted by the research community.
We study the task of publishing datasets in a differentially private manner. In particular, we are interested in solving two problems. First, we want to be able to benefit from the use of machine learning by third parties while protecting sensitive information of individuals in our dataset. Second, we want to be sure that even if adversaries get access to the thirdparty model trained on our data, they would not be able to recover private information. An additional challenge is to be able to publish an entire dataset, as opposed to being required to use a query interface like in a typical differentially private framework.
In this paper, we propose a simple solution to this problem. The main idea of our approach is to use generative adversarial networks (GANs) introduced in Goodfellow et al. (2014), trained with addition of Gaussian noise in the embedding space, to create artificial datasets that follow the same distribution as the real data while providing differential privacy guarantees. This method has a number of advantages over the methods proposed earlier. First of all, this solution is simple to implement, e.g. it does not require training ensembles of models on disjoint data. Second, it can be done on a user side, and not on the side of the machine learning service provider, which eliminates the necessity of trusting this service provider or implementing privacypreserving models locally. Third, similarly to Abadi et al. (2016), privacy cannot be compromised even if the entire trained model is accessible to an adversary.
Our contributions in this paper are the following:

we propose a novel mechanism for noninteractive differentially private labelled data release, and to the best of our knowledge, this is the first practical solution for complex realworld data;

we introduce a new technique of preserving privacy in neural networks via adding noise in the forward pass during training;

we show that our technique guarantees differential privacy for both the outputs and the learned weights of the network;

we demonstrate that we are able to achieve high accuracy in learning tasks while maintaining a reasonable (singledigit) privacy budget.
The remainder of the paper is structured as follows. In Section 2, we give an overview of related work. Section 3 contains necessary background on differential privacy and generative adversarial networks. In Section 4, we describe our approach and provide its theoretical analysis and some practical aspects. Experimental results and implementation details are presented in Section 5, and Section 6 concludes the paper. Proofs and additional details can be found in the Appendix.
2 Related work
Given the level of attention to deep learning and the rising importance of privacy, it is unsurprising that there has been a significant increase in the number of publications on the topic of privacypreserving deep learning (and machine learning in general) in recent years.
One take on the problem is to distribute training and use disjoint sets of training data. An example of such approach is the paper of Shokri & Shmatikov (2015), where they propose to train in a distributed manner by communicating sanitised updates from participants to a central authority. Such a method, however, yields high privacy losses as pointed out by Abadi et al. (2016) and Papernot et al. (2016). An alternative technique, also using disjoint training sets, suggested by Papernot et al. (2016), applies an ensemble of independently trained teacher models and semisupervised knowledge transfer to a student model to achieve almost stateoftheart (nonprivate) accuracy on MNIST (LeCun et al., 1998) and SVHN (Netzer et al., 2011) with singledigit differential privacy bounds. This work was based on a paper by Hamm et al. (2016) and extends their method to generic learning models with any type of loss functions or optimisation algorithms. To the best of our knowledge, this is the most accurate privacypreserving learning result to date, although one has to make sure that all the teaching ensemble and the aggregator are inaccessible to an adversary and the model is queried for teachers’ votes only a small number of times.
A somewhat different approach is taken in Abadi et al. (2016). They suggest using differentially private stochastic gradient descent (for brevity, we will refer to it as DPSGD in the remainder of the paper) to train deep learning models in a private manner. This approach allows to achieve high accuracy while maintaining low differential privacy bounds, and does not require distributed training.
As stated above, our goal is to enable data usage by third party machine learning service providers to benefit from their expertise. All of the aforementioned methods, however, require every provider of such service to comply with the chosen privacypreserving procedure which is not realistic. An alternative solution to this problem is to focus on sanitising data and making sure that training machine learning models on it would not compromise privacy. This direction is taken, for example, by Bindschaedler et al. (2017). The authors use a graphical probabilistic model to learn an underlying data distribution and transform real data points (seeds) into synthetic data points. Synthetic data is then filtered by a privacy test based on a plausible deniability criterion, which can be equivalent to differential privacy under certain conditions.
Our approach, on the other hand, is to generate private data without requiring any real seeds. Thus, there is no need for privacy tests at the release stage, and the only requirement is that the generative model is privacypreserving. By using GANs (Goodfellow et al., 2014) we ensure that our method is scalable and applicable to complex realworld data.
3 Background
This section gives a short introduction to GANs and differential privacy. Another important notion is the moments accountant method (Abadi et al., 2016) used to compute actual privacy bounds during training. However, since it is not essential for understanding the paper, we defer its description to the Appendix.
3.1 Generative adversarial networks
In recent years, generative adversarial networks (Goodfellow et al., 2014; Salimans et al., 2016) and its extensions, such as DCGAN (Radford et al., 2015), EBGAN (Zhao et al., 2016), and Wasserstein GAN (Arjovsky et al., 2017; Gulrajani et al., 2017), have received great attention and pushed the boundaries for deep generative models along with variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014; Gregor et al., 2015) and recursive neural networks (e.g. PixelRNN by Oord et al. (2016)). The most successful application for such generative models so far has been realistic image generation, perhaps due to abundance of training data and inherent geometric structure.
In our work, we decided to choose GANs for several reasons. Firstly, GANs have shown very good results in practice, generating sharper images compared to other generative models. Secondly, the forward pass for generating data is much faster than that of, for instance, RNNs. Thirdly, the generator part of the model, the one we eventually interested in, does not interact with real training data at any point in the learning process, only getting gradients from the discriminator.
In short, GANs can be described as follows. The model consists of two separate components: the generator and the discriminator . The generator’s goal is to produce realistic samples of data based on a random variable , while the discriminator is tasked with distinguishing real data samples from generated samples . These two models are trained in an adversarial fashion, essentially playing a twoplayer game, with the goal to converge to the Nash equilibrium. Since training GANs in practice can be challenging, there is a number of commonly used tricks to improve convergence, such as using the Adam optimisation method (Kingma & Ba, 2015), feature matching, batch normalisation, and onesided label smoothing (Salimans et al., 2016). In this paper, we use an improved version of Wasserstein GAN by Gulrajani et al. (2017) as it has been shown to perform consistently well on a number of datasets and different network architectures.
3.2 Differential privacy
The notion of differential privacy has been introduced and extended in a series of papers by Dwork et al. (Dwork, 2006; Dwork et al., 2006; Dwork, 2008; Dwork et al., 2014), and is regarded as a strong privacy standard. It is defined for two adjacent datasets that differ by a single element:
Definition 1.
A randomized mechanism with domain and range satisfies differential privacy if for any two adjacent inputs and for any subset of outputs it holds that:
(1) 
Among the mechanisms to achieve differential privacy, two of the most widely used are Laplace and Gaussian noise mechanisms. We are primarily interested in the latter, because of the improved privacy bounds analysis provided by the moments accountant method described in the Appendix. The Gaussian noise mechanism is defined as follows:
(2) 
where is the sensitivity of (i.e. for ), and is the Gaussian distribution with the mean and the standard deviation .
4 Our approach
In this section, we describe our solution and provide a theoretical proof of privacy guarantees, as well as discuss limitations of the method. Let us begin with the formal problem statement.
Problem Statement.
Given the dataset , generate an artificial dataset using the privacy mechanism , such that

it follows the same data distribution: ;

it provides differential privacy guarantees: for any adjacent datasets , and for any .
Here is the space of all datasets formed by points drawn from the same distribution .
In most realworld problems, the true data distribution is unknown and needs to be estimated empirically. Since we are primarily interested in data synthesis, we will turn to generative models, and in particular we are going to use GANs as the mechanism to estimate and draw samples from it. If trained properly, GAN will provide a solution to the subproblem (1).
Despite the fact that the generator does not have access to the real data in the training process, one cannot guarantee differential privacy because of the information passed through with the gradients from the discriminator. A simple high level example will illustrate such breach of privacy. Let the datasets contain small real numbers. The only difference between these two datasets is the number , which happens to be extremely large. Since the gradients of the model depend on , one of the updates of the discriminator trained on may be very different from the rest, and this difference will the be propagated to the generator breaking privacy in general case.
In order to maintain differential privacy guarantees, we propose the following solution.
Proposition.
Introduce a Gaussian noise layer in the discriminator network of GAN, so that its output, and therefore the weights of the trained generator, are differentially private with respect to the input data . Use this generator to create a publishable differentially private dataset.
The components of our solution are depicted in Figure 1.
4.1 Theoretical analysis of the approach
To validate the proposed solution, we first analyse it theoretically and show that the addition of a Gaussian noise layer in the discriminator network yields differential privacy in the generator. We will take the following steps to do that:

analyse privacy of the output of the noise layer w.r.t. the inputs and ;

determine privacy bounds on the output of the whole network;

show that the same bounds hold for gradient updates.
Let us start by describing the setting and notation used in the remainder of the section. We are given two adjacent datasets and and a feedforward neural network with a Gaussian noise layer . We also require not to have any randomised or batch normalisation layers before .
We denote the dimensional inputs of the layer as and , and the outputs of the final layer of the network and correspondingly. To ensure bounded sensitivity, the norm of and is clipped to a predefined constant . Then, Gaussian noise with standard deviation is added. Assume for the moment that this noise is sufficient to ensure differential privacy.
In practice, one could either choose the values of before training and derive corresponding , or compute posttraining given the value of used in training.
Lemma 1.
If the output of the noise layer is differentially private w.r.t. and the network layers before preserve adjacency of and , then is also differentially private w.r.t. .
The proof of this lemma can be found in Appendix. Note that allowing randomised layers or batch normalisation in the network before would break the preservation of adjacency and therefore privacy guarantees.
Using Lemma 1, we are able demonstrate that outputs, as well as gradients, of a feedforward neural network with a Gaussian noise layer are differentially private with respect to the input data, which is expressed in the following theorems.
Theorem 1.
(Forward pass) The output of a feedforward neural network with differentially private layer , is also differentially private with respect to .
Proof.
Theorem 2.
(Backward pass) Given a feedforward neural network with differentially private outputs , weight updates are also differentially private with respect to in each iteration of gradient descent.
Proof.
Since we are interested in generating data using GANs, we will also need the following corollary to finalise the theoretical foundation for our framework.
Corollary 1.
(GANs) Given a generative adversarial network consisting of the generator and the privacypreserving discriminator , gradient updates of will have the same privacy bounds as gradient updates of .
Proof.
The above analysis is applicable for each individual iteration of the gradient descent, and privacy bounds on the final parameters can be obtained using composition theorems or a more efficient moments accountant method (Abadi et al., 2016).
Note that Theorems 1 and 2 define differential privacy of the neural network with respect to the inputs only, not taking into account the labels . In certain cases, when labels of interest are already a public knowledge and do not reveal any information about data, it may be sufficient. However, if labels privacy is required, it is possible to incorporate it in the proposed approach in two ways.
A first solution, the one we adopt in this paper, is to modify the learning problem so that labels become a part of data. For example, if one wants to train a face recognition model with privacybreaking labels (e.g. specific names—John, Bob, Julia, etc.), it is possible to add these labels to , and instead use True and False labels in , indicating whether the input image and the input name correspond to each other. This way, label privacy will be handled by the same framework.
Alternatively, one can use a separate privacypreserving mechanism to retrieve labels during training. In this case, the eventual privacy w.r.t. the pair may be derived from a composition of two mechanisms, which is shown in the theorem below. One possible candidate for such mechanism is the noisy voting scheme as used in Papernot et al. (2016).
Theorem 3.
(Private labels) Given a feedforward neural network with –differentially private outputs , and the training labels satisfying –differential privacy w.r.t. the true labels , the gradient updates are –differentially private with respect to on each iteration of gradient descent.
Proof.
There are two privacy mechanisms and applied to and correspondingly. Observe that does not have access to , and thus, cannot influence the output probabilities of . The same is true for and . Consequently, we can assume that both mechanisms are applied to a pair . This allows us to employ a basic sequential composition theorem for differential privacy (Dwork & Lei, 2009) to obtain the privacy bounds. ∎
While it may appeal to use parallel composition instead of sequential composition to obtain a tighter bound, since and appear to be disjoint, it would be incorrect. The reason is that and are strongly correlated and breaking privacy of one can reveal the other. Alternatively, one could use advanced composition theorems (see e.g. Dwork et al. (2010); Kairouz et al. (2015); Dwork & Rothblum (2016)) to prove tighter privacy bounds, but it is not the goal of our paper.
4.2 Practical aspects
Based on the analysis above, we can do a number of important observations regarding applicability of this technique.
First of all, the analysis is performed for feedforward networks. Other architectures, such as RNNs or memory networks, require additional investigation. Second, conditions of Lemma 1 dictate that the network layers prior to must preserve adjacency of the input. This condition is not fulfilled, for example, by batch normalisation, because it introduces interdependencies between examples inside a batch and one different instance can change outputs for all instances of a batch. Randomised layers, such as dropout, may also break adjacency.
Summarising these limitations, the neural network under question must

have feedforward network architecture;

have no adjacency breaking layers before the privacy layer, e.g. batch normalisation or dropout.
In the following section, we will touch upon some implications of it that affect practical performance. Note that these restrictions only apply to the network, in which we insert a privacypreserving layer, i.e. only the discriminator in our case.
5 Evaluation
In this section, we provide implementation details and discuss evaluation results obtained on MNIST (LeCun et al., 1998) and SVHN (Netzer et al., 2011) datasets.
5.1 Experimental setup
We evaluate our solution as follows. First, we train a generative model on original datasets (using only training parts of each) with differential privacy by adding a Gaussian noise layer to the discriminator. We will call this model a teacher, analogously to Papernot et al. (2016). Then, we generate an artificial dataset of comparable size using the obtained model. Finally, we train a separate (nonprivate) classifier, which we call a student, on generated data and test it using heldout test sets. The last step is important from two perspectives: we can quantify the quality of generated samples as opposed to visual inspection typically done with GANs, and we can compare test errors to previously reported values. Note that there is no dependencies between the teacher and the student models. Moreover, student models are not constrained to neural networks and can be implemented as any type of machine learning algorithm.
We choose two commonly used image classification datasets for our experiments: MNIST and SVHN. MNIST is a handwritten digit recognition dataset consisting of 60’000 training examples and 10’000 test examples, each example is a 28x28 size greyscale image. SVHN is also a digit recognition task, with 73’257 images for training and 26’032 for testing. The examples are coloured 32x32 pixel images of house numbers from Google Street View.
Dataset  NonPrivate Baseline  Papernot et al. (2016)  Our approach 

MNIST  ()  ()  
SVHN  ()  () 
5.2 Implementation details
Implementation was done in Python using Pytorch framework^{2}^{2}2http://pytorch.org. For a generative model, we implemented (with some modifications) an improved Wasserstein GAN by Gulrajani et al. (2017). More specifically, the discriminator (also called critic) consists of three convolutional layers with leaky ReLU activations followed by a fully connected linear layer which outputs a dimensional feature vector (). We clip the norm of this vector to (to ensure bounded sensitivity) and add Gaussian noise with before passing it through the final linear classification layer. The generator starts with a fully connected linear layer that transforms noise and labels into a dimensional feature vector which is then passed to a batch normalisation layer with ReLU activation and three deconvolution layers with batch normalisation and ReLU activations. The output of the third deconvolution layer is processed by fractional max pooling and tanh activation function.
Similarly to the original paper, we use a classical Wasserstein GAN value function with the gradient penalty that enforces Lipschitz constraint on the critic (Gulrajani et al., 2017). We also set the penalty parameter and the number of critic iterations . Furthermore, we have modified the architecture to allow for conditioning the generative model on class labels. Binarised labels are appended to the input of the generator and to the linear layer of the critic after convolutions. Therefore, the generator can be used to create labelled datasets for supervised learning.
Both networks were trained using Adam optimiser (Kingma & Ba, 2015) with parameters similar to (Gulrajani et al., 2017): learning rate set to , , , and a batch size of .
Privacy bounds were evaluated using the moments accountant method and the privacy amplification theorem (Abadi et al., 2016), and therefore, are datadependent and are tighter than using normal composition theorems. Application of the method is almost identical to the original paper as we use the same norm clipping and Gaussian noise.
The student network is constructed of two convolutional layers with ReLU activations, batch normalisation and max pooling, followed by two fully connected layers with ReLU, and a softmax output layer. It is worth mentioning that this network does not achieve stateoftheart performance on the used datasets, but we are primarily interested in evaluating the performance drop compared to a nonprivate model rather than getting the best test score.
5.3 Discussion
Using the experimental setup and implementation described above, we were able to get results close to Papernot et al. (2016), even outperforming their model in terms of accuracy for similar or better privacy guarantees. Overall, we managed to achieve accuracy on MNIST and accuracy on SVHN while maintaining approximately and differential privacy. These numbers, along with the corresponding results of Papernot et al. (2016), can be found in Table 1. It is also worth mentioning that we did not perform rigorous hyperparameter tuning due to limited computational resources; better accuracy could be achieved have we had done that. Additionally, we trained a simple logistic regression model on MNIST, and obtained accuracy on privately generated data compared to on the original data, which confirms that any model can be used as a student.
A natural question to ask is how dimensionality of data influences performance of our technique, knowing that the norm of Gaussian noise scales as a square root of the number of dimensions. As a matter of fact, contrary to DPSGD, the scale of noise in our approach only depends on the embedding dimensionality which is generally much lower than the data dimensionality or the number of network parameters. For example, the data dimensionality of SVHN is while the embedding dimensionality is just .
Examples of real and generated privacypreserving images for MNIST and SVHN data are depicted on Figure 2. It can be seen that generated images don’t have the same contrast and dynamic range as real examples, which is not a problem in nonprivate GANs.
In addition to quantitative analysis of test errors and privacy bounds, we perform visual inspection of generated examples and corresponding nearest neighbours in real data. Figure 3 depicts a set of generated private examples and their nearest real counterparts. We observe that while some generated images are very close to real examples they don’t match exactly, differing either in shape, colour or surrounding digits. Moreover, a lot of pairs come from entirely different classes.
6 Conclusions
We investigate the problem of noninteractive private data release with differential privacy guarantees. We employ generative adversarial networks to produce artificial privacypreserving datasets. Contrary to existing privacy protection work in deep learning, this method allows to publish sanitised data and train any nonprivate models on it. The choice of GANs as a generative model ensures scalability and makes the technique suitable for realworld data with complex structure. Moreover, this method does not require running privacy tests on generated data before releasing it.
Additionally, we introduce a novel method for preserving privacy of training data specific to deep neural networks based on adding noise in the embedding space during forward pass. It provides differential privacy guarantees and allows to construct privacypreserving models in a simple and straightforward fashion, without modifying optimisation algorithms.
In our experiments, we show that student models trained on artificial data can achieve high utility on MNIST dataset, while maintaining performance costs of added privacy and flexibility at acceptable levels on a more complicated SVHN data. One of the possible directions for future work is to improve the quality of generated data for given privacy bounds. Extending presented technique and analysis to other types of deep neural networks provides another exciting opportunity for further research.
References
 Abadi et al. (2016) Martín Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318. ACM, 2016.
 Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.
 Bindschaedler et al. (2017) Vincent Bindschaedler, Reza Shokri, and Carl A Gunter. Plausible deniability for privacypreserving data synthesis. Proceedings of the VLDB Endowment, 10(5), 2017.
 Dwork (2006) Cynthia Dwork. Differential privacy. In 33rd International Colloquium on Automata, Languages and Programming, part II (ICALP 2006), volume 4052, pp. 1–12, Venice, Italy, July 2006. Springer Verlag. ISBN 3540359079. URL https://www.microsoft.com/enus/research/publication/differentialprivacy/.
 Dwork (2008) Cynthia Dwork. Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of Computation, pp. 1–19. Springer, 2008.
 Dwork & Lei (2009) Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In Proceedings of the fortyfirst annual ACM symposium on Theory of computing, pp. 371–380. ACM, 2009.
 Dwork & Rothblum (2016) Cynthia Dwork and Guy N Rothblum. Concentrated differential privacy. arXiv preprint arXiv:1603.01887, 2016.
 Dwork et al. (2006) Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology (EUROCRYPT 2006), volume 4004, pp. 486â503, Saint Petersburg, Russia, May 2006. Springer Verlag. URL https://www.microsoft.com/enus/research/publication/ourdataourselvesprivacyviadistributednoisegeneration/.
 Dwork et al. (2010) Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. Boosting and differential privacy. In Foundations of Computer Science (FOCS), 2010 51st Annual IEEE Symposium on, pp. 51–60. IEEE, 2010.
 Dwork et al. (2014) Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
 Fredrikson et al. (2015) Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322–1333. ACM, 2015.
 Goodfellow et al. (2014) Ian Goodfellow, Jean PougetAbadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680, 2014.
 Gregor et al. (2015) Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In Proceedings of the 32nd International Conference on Machine Learning, pp. 1462–1471, 2015.
 Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5769–5779, 2017.
 Hamm et al. (2016) Jihun Hamm, Yingjun Cao, and Mikhail Belkin. Learning privately from multiparty data. In Proceedings of The 33rd International Conference on Machine Learning, pp. 555–563, 2016.
 He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026–1034, 2015.
 Kairouz et al. (2015) Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International Conference on Machine Learning (ICML), 2015.
 Kingma & Ba (2015) Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations, 2015.
 Kingma & Welling (2014) Diederik P Kingma and Max Welling. Autoencoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
 LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
 Li et al. (2007) Ninghui Li, Tiancheng Li, and Suresh Venkatasubramanian. tcloseness: Privacy beyond kanonymity and ldiversity. In Data Engineering, 2007. ICDE 2007. IEEE 23rd International Conference on, pp. 106–115. IEEE, 2007.
 Machanavajjhala et al. (2007) Ashwin Machanavajjhala, Daniel Kifer, Johannes Gehrke, and Muthuramakrishnan Venkitasubramaniam. ldiversity: Privacy beyond kanonymity. ACM Transactions on Knowledge Discovery from Data (TKDD), 1(1):3, 2007.
 Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 5, 2011.
 Oord et al. (2016) Aaron Van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on Machine Learning, pp. 1747–1756, 2016.
 Papernot et al. (2016) Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semisupervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755, 2016.
 Radford et al. (2015) Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
 Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. pp. 1278–1286, 2014.
 Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2226–2234, 2016.
 Shokri & Shmatikov (2015) Reza Shokri and Vitaly Shmatikov. Privacypreserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1310–1321. ACM, 2015.
 Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016.
 Sweeney (2002) Latanya Sweeney. Kanonymity: A model for protecting privacy. Int. J. Uncertain. Fuzziness Knowl.Based Syst., 10(5):557–570, October 2002. ISSN 02184885. doi: 10.1142/S0218488502001648. URL http://dx.doi.org/10.1142/S0218488502001648.
 Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
 Zhao et al. (2016) Junbo Zhao, Michael Mathieu, and Yann LeCun. Energybased generative adversarial network. arXiv preprint arXiv:1609.03126, 2016.
Appendix A Appendix
In this appendix, we state again and prove the Lemma 1 from Section 4.1. We also provide details on privacy analysis method—moments accountant (Abadi et al., 2016).
a.1 Proof of Lemma 1
Lemma 2.
If the output of the noise layer is differentially private w.r.t. and the network layers before preserve adjacency of and , then is also differentially private w.r.t. .
Proof.
By definition of differential privacy:
(3) 
for all adjacent and .
We need to show that the same holds for all adjacent inputs , i.e. . Observe that we defined our network as deterministic (i.e. not having any randomness apart from initial data shuffling) before the privacypreserving layer . Therefore, , where is a Dirac delta function. Conceptually, it means that the entire mass of the distribution of is concentrated on the point .
Using the above observation,
(4)  
(5)  
(6)  
(7)  
(8)  
(9)  
(10) 
∎
a.2 Moments Accountant
The privacy bound produced by the strong composition theorem is often too loose, and therefore, we exploit the moments accountant technique developed by Abadi et al. (2016) for analysing their DPSGD algorithm.
To give the main idea of the method, let us start with defining the privacy loss.
Definition 2.
Let be a randomized mechanism and a pair of adjacent databases. Let aux denote an auxiliary input. For an outcome , the privacy loss at is defined as:
(11) 
And the privacy loss random variable is defined as .
The moments accountant is then defined as follows:
Definition 3.
Again, let be a randomized mechanism, a pair of adjacent databases, and aux denote an auxiliary input. The moments accountant is
(12) 
where is a momentgenerating function.
In short, the moments accountant method tracks the bounds on the moments of the privacy loss random variable and then uses Markov inequality to obtain the tail bound on this random variable corresponding to the values of and .