Supervised machine learning assumes that training and testing data are sampled i.i.d from the same distribution, while in practice, the training and testing data are typically collected from related domains but under different distributions, a phenomenon known as domain shift Quionero-Candela et al. (2009). To avoid the cost of annotating each new test domain, Unsupervised Domain Adaptation (UDA) tackles domain shift by aligning the feature distribution of the source domain with that of the target domain, resulting in domain-invariant features. However, current methods assume that target samples have domain labels and therefore can be isolated into separate homogeneous domains. For many practical applications, this is an overly strong assumption. For example, a hand-written character recognition system could encounter characters written by different people, on different materials, and under different lighting conditions; an image recognition system applied to images scraped from the web must handle mixed-domain data (e.g. paintings, sketches, clipart) without their domain labels.

In this paper, we consider Domain-Agnostic Learning (DAL), a more difficult but practical problem of knowledge transfer from one labeled source domain to multiple unlabeled target domains. The main challenges of domain-agnostic learning are that: (1) the target data has mixed domains, which hampers the effectiveness of mainstream feature alignment methods Long et al. (2015); Sun & Saenko (2016); Saito et al. (2018), and (2) class-irrelevant information leads to negative transfer Pan & Yang (2010), especially when the target domain is highly heterogeneous.

Mainstream UDA methods align the source domain to the target domain by minimizing the Maximum Mean Discrepancy Long et al. (2015); Tzeng et al. (2014), aligning high-order moments Sun & Saenko (2016); Zellinger et al. (2017), or adversarial training Ganin & Lempitsky (2015); Tzeng et al. (2017). However, these methods are designed for one-to-one domain alignment and do not account for multiple latent domains in the target. Multi-source domain adaptation Peng et al. (2018); Xu et al. (2018); Mansour et al. (2009) considers adaptation between multiple sources and a single target domain and assumes domain labels on the source data. Continuous domain adaptation Hoffman et al. (2014) aims to transfer knowledge to a continuously changing domain (e.g. cars in different decades), but in their scenario the target data are temporally related. Recently, domain generalization approaches Li et al. (2018a); Carlucci et al. (2018b); Li et al. (2018b) have been introduced to adapt from multiple labeled source domains to an unseen target domain. All of the above models make a strong assumption that the target data are homogeneously sampled from the same distribution, unlike the scenario we consider here.

We postulate that a solution to domain-agnostic learning should not only learn invariance between source and target, but should also actively disentangle the class-specific features from the remaining information in the image. Deep neural networks are known to extract features in which multiple hidden factors are highly entangled Bengio et al. (2013). Recent work attempts to disentangle features in the latent space of autoencoders with adversarial training Cao et al. (2018); Liu et al. (2018b); Odena et al. (2017); Lee et al. (2018). However, the above models have limited capacity in transferring features learned from one domain to heterogeneous target domains.  Liu et al. (2018a) proposes a framework that takes samples from multiple domains as input, and derives a domain-invariant latent feature space via adversarial training. This model is limited by two factors when applied to the DAL task. First, it only disentangles the embeddings into domain-invariant features and domain-specific features such as weather conditions, and discards the latter, but does not explicitly try to separate class-relevant features from class-irrelevant features like background. Second, there is no guarantee that the domain-invariant features are fully disentangled from the domain-specific features.

To address the issues mentioned above, we propose a novel Deep Adversarial Disentangled Autoencoder (DADA), aiming to tackle domain-agnostic learning by disentangling the domain-invariant features from both domain-specific and class-irrelevant features simultaneously. First, in addition to domain disentanglement Liu et al. (2018a); Cao et al. (2018); Lee et al. (2018), we employ class disentanglement to remove class-irrelevant features, as shown in Figure 1. The class disentanglement is trained in an adversarial fashion: a class identifier is trained on the labeled source domain and the disentangler generates features to fool the class identifier. To the best of our knowledge, we are the first to show that class disentanglement boosts domain adaptation performance. Second, to enhance the disentanglement, we propose to minimize the mutual information between the disentangled features. We implement a neural network to estimate the mutual information between the disentangled feature distributions, inspired by a recently published theoretical work Belghazi et al. (2018). Comprehensive experiments on standard image recognition datasets demonstrate that our derived disentangled representation achieves significant improvements over the state-of-the-art methods on the task of domain-agnostic learning.

The main contributions of this paper are highlighted as follows: (1) we propose a novel learning paradigm of domain-agnostic learning; 2) we develop an end-to-end Deep Adversarial Disentangled Autoencoder (DADA) which learns a better disentangled feature representation to tackle the task; and (3) We propose class disentanglement to remove class-irrelevant features, and minimize the mutual information to enhance the disentanglement.

Domain Adaptation Unsupervised domain adaptation (UDA) aims to transfer the knowledge learned from one or more labeled source domains to an unlabeled target domain. Various methods have been proposed, including discrepancy-based UDA approaches Long et al. (2017); Tzeng et al. (2014); Ghifary et al. (2014); Peng & Saenko (2018), adversary-based approaches Liu & Tuzel (2016); Tzeng et al. (2017); Liu et al. (2018a), and reconstruction-based approaches Yi et al. (2017); Zhu et al. (2017); Hoffman et al. (2018); Kim et al. (2017). These models are typically designed to tackle single source to single target adaptation. Compared with single source adaptation, multi-source domain adaptation (MSDA) assumes that training data are collected from multiple sources. Originating from the theoretical analysis in Ben-David et al. (2010); Mansour et al. (2009); Crammer et al. (2008), MSDA has been applied to many practical applications Xu et al. (2018); Duan et al. (2012); Peng et al. (2018). Specifically,  Ben-David et al. (2010) introduce an $H\Delta H$-divergence between the weighted combination of source domains and a target domain. We propose a new and more practical learning paradigm, not yet considered in the UDA literature, where labeled data come from a single source domain but the testing data contain multiple unknown domains.

Representation Disentanglement The goal of learning disentangled representations is to model the factors of data variation. Recent works Mathieu et al. (2016); Makhzani et al. (2016); Liu et al. (2018a); Odena et al. (2017) aim at learning an interpretable representation using generative adversarial networks (GANs) Goodfellow et al. (2014); Kingma et al. (2014) and variational autoencoders (VAEs) Rezende et al. (2014); Kingma & Welling (2013). In a fully supervised setting, Lee et al. (2018) proposes to disentangle the feature representation into a domain-invariant content space and a domain-specific attribute space, producing diverse outputs without paired training images. Another work Odena et al. (2017) proposes an auxiliary classifier GAN (AC-GAN) to achieve representation disentanglement. Despite promising performance, these methods focus on disentangling representation in a single domain.  Liu et al. (2018a) introduces a unified feature disentangler to learn a domain-invariant representation from data across multiple domains. However, their model assumes that multiple source domains are available during training, which limits its practical application. In contrast, our model disentangles the representation based on one source domain and multiple unknown target domains, and proposes an improved approach to disentanglement that considers the class label and mutual information between features.

Agnostic Learning There are several prior studies of agnostic learning that are related to our work. Model-Agnostic Meta-Learning (MAML) Finn et al. (2017) aims to train a model on a variety of learning tasks and solve a new task using only a few training examples. Different from MAML, our method mainly focuses on transferring knowledge to heterogeneous domains. Carlucci et al. (2018a) proposes a learning framework to seamlessly extend the knowledge from multiple source domain to an unseen target domain by pixel-adaptation in an incremental architecture. Romijnders et al. (2018) introduces a domain agnostic normalization layer for adversarial UDA and improves the performance of deep models on an unseen domain. Though the results are promising, we argue that only normalizing the feature representation is not enough for domain-agnostic learning, and that extracting disentangled domain-invariant and domain-specific features is also important.

We define the domain-agnostic learning task as follows: Given a source domain ${\hat{D}}_s=\left\{\right(x_i^s,y_i^s){\}}_{i=1}^{n_s}$ with $n_s$ labeled examples, the goal is to minimize risk on $N$ target domains ${\hat{D}}_t$ = {${\hat{D}}_1,{\hat{D}}_2,...,{\hat{D}}_N\}$ without domain labels. We denote the target domains as ${\hat{D}}_t=\{x_j^t{\}}_{j=1}^{n_t}$ with $n_t$ unlabeled examples. Empirically, we want to minimize the target risk ${ϵ}_t\left(\theta \right)={Pr}_{\left(x,y\right)\sim {\hat{D}}_t}\left[\theta \left(x\right)\ne y\right]$, where $\theta \left(x\right)$ is the classifier.

We propose to solve the task by learning domain-invariant features that are discriminative of the class. Figure 1 shows the proposed model. The feature generator $G$ maps the input image to a feature vector $f_G$, which has many highly entangled factors. The disentangler $D$ is responsible for disentangling the features ($f_G$) into domain-invariant features ($f_{di}$), domain-specific features ($f_{ds}$), and class-irrelevant features ($f_{ci}$). The feature reconstructor $R$ aims to recover $f_G$ from either ($f_{di}$, $f_{ds}$) or ($f_{di}$, $f_{ci}$). D and R are implemented as the encoder and decoder in a Variational Autoencoder. A mutual information minimizer is applied between $f_{di}$ and $f_{ci}$, as well as between $f_{di}$ and $f_{ds}$, to enhance the disentanglement. Adversarial training via a domain identifier aligns the source domain and the heterogeneous target domain in the $f_{di}$ space. A class identifier $C$ is trained on the labeled source domain to predict the class distribution $f_C$ and to adversarially extract class-irrelevant features $f_{ci}$. We next describe each component in detail.

Variational Autoencoders VAEs Kingma & Welling (2013) are a class of deep generative models that simultaneously train both a probabilistic encoder and decoder. The encoder is trained to generate latent vectors that roughly follow a Gaussian distribution. In our case, we learn each part of our disentangled representations by applying a VAE architecture with the following objective function:

where the first term aims at recovering the original features extracted by $G$, and the second term calculates Kullback-Leibler divergence which penalizes the deviation of latent features from the prior distribution $p\left(z_c\right)$ (as $z\sim N(0,I)$). However, this property cannot guarantee that domain-invariant features are well disentangled from the domain-specific features or from class-irrelevant features, as the loss function in Equation 1 only aligns the latent features to a normal distribution.

Class Disentanglement To address the above problem, we employ class disentanglement to remove class-irrelevant features, such as background, in an adversarial way. First, we train the disentangler $D$ and the $K$-way class identifier $C$ to correctly predict the labels, supervised by the cross-entropy loss:

where $f_D\in \{f_{di},f_{ci}\}$.

In the second step, we fix the class identifier and train the disentangler $D$ to fool the class identifier by generating class-irrelevant features $f_{ci}$. This can be achieved by minimizing the negative entropy of the predicted class distribution:

where the first term and the second term indicate minimizing the entropy on the source domain and on heterogeneous target, respectively. The above adversarial training process forces the corresponding disentangler to extract class-irrelevant features.

Domain Disentanglement To tackle the domain agnostic learning task, disentangling class-irrelevant features is not enough, as it fails to align the source domain with the target. To achieve better alignment, we further propose to disentangle the learned features into domain-specific and domain-invariant and to thus align the source with the target domain in the domain-invariant latent space. This is achieved by exploiting adversarial domain classification in the resulting latent space. Specifically, we leverage a domain identifier $DI$, which takes the disentangled feature ($f_{di}$ or $f_{ds}$ ) as input and outputs the domain label $l_f$ (source or target). The objective function of the domain identifier is as follows:

Then the disentangler is trained to fool the domain identifier $DI$ to extract domain-invariant features.

where $x\in \{f_{ds},f_{ci}\}$, $P_{XZ}$ is the joint probability distribution of ($D_x$, $D_{f_{di}}$), and $P_X={\int }_{Z}d{P}_{XZ}$ and $P_Z={\int }_{X}d{P}_{XZ}$ are the marginals. Despite being a pivotal measure across different domains, the mutual information is only tractable for discrete variables, or for a limited family of problems where the probability distributions are unknown Belghazi et al. (2018). The computation incurs a complexity of $O\left(n^2\right)$, which is undesirable for deep CNNs. Is this paper, we adopt the Mutual Information Neural Estimator (MINE) Belghazi et al. (2018)

which provides unbiased estimation of mutual information on $n$ i.i.d samples by leveraging a neural network $T_{\theta }$.

Practically, MINE (6) can be computed as $I(X;Z)=\int \int P_{XZ}^n(x,z)T(x,z,\theta )$ - $\log (\int \int P_X^n(x\left)P_Z^n\right(z\left)e^{T(x,z,\theta )}\right)$. Additionally, to avoid computing the integrals, we leverage Monte-Carlo integration:

where $(x,z)$ are sampled from the joint distribution and $z^{\prime }$ is sampled from the marginal distribution. We implement a neural network to perform the Monte-Carlo integration defined in Equation 7.

Ring-style Normalization Conventional batch normalization Ioffe & Szegedy (2015) diminishes internal covariate shift by subtracting the batch mean and dividing by the batch standard deviation. Despite promising results on domain adaptation, batch normalization alone is not enough to guarantee that the embedded features are well normalized in the scenario of heterogeneous domains. The target data are sampled from multiple domains and their embedded features are scattered irregularly in the latent space. Zheng et al. (2018) proposes a ring-style norm constraint to maintain a balance between the angular classification margins of multiple classes. Its objective is as follows:

where $R$ is the learned norm value. However, ring loss is not robust and may cause mode collapse if the learned $R$ is small. Instead, we incorporate the ring loss into a Geman-McClure model and minimize the following loss function:

where $\beta$ is the scale factor of the Geman-McClure model.

Optimization Our model is trained in an end-to-end fashion. We train the class and domain disentanglement component, MINE and the reconstruction component iteratively with Stochasitc Gradient Descent Kiefer et al. (1952) or Adam Kingma & Ba (2014) optimizer. We employ the popular neural networks (e.g. LeNet, AlexNet, or ResNet) as our feature generator $G$. The detailed training procedure is presented in Algorithm 1.

We compare the DADA model to state-of-the-art domain adaptation algorithms on the following tasks: digit classification (MNIST, SVHN, USPS, MNIST-M, Synthetic Digits) and image recognition (Office-Caltech10 Gong et al. (2012), DomainNet Peng et al. (2018)). Sample images of these datasets can be seen in Figure 2. Table 6 (suppementary material) shows the detailed number of images we use in our experiments. In the main paper, we only report major results; more implementation details are provided in the supplementary material. All of our experiments are implemented in the PyTorch¹¹1http://pytorch.org platform.

In this paper, we first propose a novel domain agnostic learning (DAL) schema and demonstrate the importance of DAL in practical scenarios. Towards tackling DAL task, we have proposed a novel Deep Adversarial Disentangled Autoencoders (DADA) to disentangle domain-invariant features in the latent space. We have proposed to leveraging class disentanglement and mutual information minimizer to enhance the feature disentanglement. Empirically, we demonstrate that the ring-loss-style normalization boosts the performance of DADA in DAL task. An extensive empirical evaluation on DAL benchmarks demonstrate the efficacy of the proposed model against several state-of-the-art domain adaptation algorithms.

We thank Saito Kuniaki, Ben Usman, Ping Hu for their useful discussions and suggestions. We thank anonymous reviewers and area chairs for their useful insight to improve this work. This work was partially supported by NSF and Honda Research Institute.