A Conditional Generative Model for Predicting Material Microstructures from Processing Methods
Abstract
Microstructures of a material form the bridge linking processing conditions  which can be controlled, to the material property  which is the primary interest in engineering applications. Thus a critical task in material design is establishing the processingstructure relationship, which requires domain expertise and techniques that can model the highdimensional material microstructure. This work proposes a deep learning based approach that models the processingstructure relationship as a conditional image synthesis problem. In particular, we develop an auxiliary classifier Wasserstein GAN with gradient penalty (ACWGANGP) to synthesize microstructures under a given processing condition. This approach is free of feature engineering, requires modest domain knowledge and is applicable to a wide range of material systems. We demonstrate this approach using the ultra high carbon steel (UHCS) database, where each microstructure is annotated with a label describing the cooling method it was subjected to. Our results show that ACWGANGP can synthesize highquality multiphase microstructures for a given cooling method.
1 Introduction
Delineating processing–structure–property relationships constitutes a major focus in design of advanced material systems (olson1997computational). While analytical and statistical methods have been successfully used for design of certain materials (florescu2009designer; fullwood2010microstructure; lee2017concurrent), the underlying assumptions on homogeneity and isotropy limit their generalizability and transferability to other material systems. To address these challenges, machine learning and datadriven techniques have piqued interest in the material science community. Microstructure reconstruction, by allowing an effective method to understand the high dimensional microstructure space, plays a critical role in computational material design. Prior work along this line has used deep learning to predict material property from microstructure (cecen2018material; cang2018improving; yang2018microstructural), reconstruct statistically equivalent microstructures (li2018transfer), and synthesize microstructures with desired properties (yang2018microstructural; cang2018improving).
Generative models, such as variational autoencoders (VAE) (kingma2014adam) and generative adversarial networks (GAN) (goodfellow2014generative), are key enablers of deep learning based microstructure reconstruction. cang2018improving used VAEs to synthesize twophase microstructures and demonstrated that convolutional networks can be used for material property prediction. yang2018microstructural used deep convolutional GAN to synthesize microstructures and transfer learning to improve structureproperty predictions. Both of these works augmented the generative model loss function with style transfer and mode collapse losses. singh2018physics leveraged WGANGP and used generative invariance checker & discriminator concurrently to generate twophase microstructures.
However, this line of work on using generative models for microstructure reconstruction has two limitations. First, previous works have focused on twophase microstructures, while many material systems comprise multiphase microstructures. Characterization and reconstruction of multiphase materials have been studied scarcely, especially due to the fact that evaluating higher order correlation for multiphase materials is challenging. To the best of our knowledge, transfer learning technique by li2018transfer is the only method that has been used to reconstruct multiphase microstructures. This method uses first few convolutional layers of a pretrained VGG16 (simonyan2014very) network to minimize difference in Gram Matrices of the target and reconstructed image. Although accurate, this method can only reconstruct images that match a single target microstructure and cannot model a distribution in the way generative models do. Second, previous works do not account for the influence of processing conditions on microstructure. This is a key aspect in material design since we strive to not only design an optimal microstructure but also identify the processing conditions necessary to manufacture it.
We address these two challenges by developing an auxiliary classifier Wasserstein GAN with gradient penalty (ACWGANGP) to synthesize multiphase alloy microstructures from user defined processing methods and demonstrate this approach using the Ultra High Carbon Steel Database (decost2017uhcsdb). Modelling this dataset is extremely challenging owing to the multiphase, heterogeneous microstructures it contains. The key contributions of this work are:

[noitemsep,topsep=10pt]

We demonstrate that GANs can synthesize multiphase microstructure images.

We demonstrate that ACWGANGP enables us to condition the generator on a critical processing condition, namely the cooling method.

We use VGG16based feature extractor and tSNE to validate the proposed approach.
2 Learning Framework
2.1 Auxiliary Classifier Wasserstein GAN with Gradient Penalty (ACWGANGP)
To learn underlying latent distributions from highdimensional data, such as images, GANs formulate the learning problem as a twoplayer zerosum game between a generator (which tries to generate synthetic images indistinguishable from the real ones) and a discriminator (which tries to distinguish whether an image is real or has been synthesized). Furthermore, by imposing additional structure into the GAN latent space, conditional generative models provide an efficient means to better control features in synthesized samples. Conditional GAN (mirza2014conditional), by providing side information (e.g. class labels) to both generator and discriminator, proposes an implementation of this approach and subsequently improves visual quality and diversity of the synthesized images. A later work by odena2017conditional introduces the auxiliary classifier GAN (ACGAN) architecture which, in addition to using class labels for synthesizing class conditional image samples, also includes a classifier which predicts class labels for images. This current work expands upon this architecture to synthesize microstructures from given values of processing parameters.
After GAN was introduced by goodfellow2014generative, several improvements have been proposed to achieve stable training and fast convergence. Wasserstein GAN (arjovsky2017wasserstein), by using earthmover distance (Wasserstein1 metric) as a geometrically meaningful measure of mismatch between probability distributions, introduced a loss function that correlates with the quality of synthesized images. A later work by gulrajani2017improved has shown that penalizing the norm of discriminator gradient, in stead of constraining the discriminator weights, provides a better alternative to enforce the Lipschitz constraint. This change in the discriminator training results in faster convergence as well as overall improvement in the quality of synthesized images.
In this work, to synthesize alloy microstructures conditioned on processing methods, we develop a hybrid framework that uses label conditioning suggested in ACGAN architecture but leverages Wasserstein1 metric to design loss functions. Additionally, we use gradient penalty instead of weight clipping for discriminator training. In particular, the generator () and the critic () in this ACWGANGP framework tries to minimize the following loss functions

and
where, , , is the true label associated with an image (both real and synthesized) , and and represent the distributions of real and synthesized images, respectively. Moreover, denote the distribution of samples which have been drawn uniformly along straight lines between pairs of images sampled from and .
2.2 Network architecture and training
In this work, we represent the cooling method via a 20dimensional embedding vector and concatenate it with a 100dimensional Gaussian noise vector. This 120dimensional combined vector is then supplied to the generator . The first layer of is a Fully connected network with 1024 neurons, followed by a dropout layer with rate = 0.25. This is followed by three UpsamplingConvolutionLeaky ReLU blocks and a final UpsamplingConvolution block with activation. The critic is an approximate mirror image of the generator with four ConvolutionLeaky ReLU blocks. These operations extract a 1024dimensional feature vector which is then passed through a dropout layer with rate = 0.25 and subsequently used by two separate fully connected layers to determine the image score and the corresponding cooling method.
3 Experiment
3.1 The Ultra High Carbon Steel DataBase (UHCSDB)
UHCSDB is a collection of 961 microstructures obtained from Scanning Electron Microscopy (SEM) of samples with identical composition but subjected to varied heat treatments. The variation in heat treatment influences the microstructure and relevant properties. SEM microstructures curated in UHCSDB correspond to 5 different cooling methods (no heat treatment, quenching, furnace cooling, air cooling and constant heating at C for 1 Hour). Within this database, we focus on microstructures captured at a magnification of 10.3 pix/micron. To train our ACWGANGP model to synthesize microstructures of size 128x128 pixels, we created a dataset containing random crops from the original 172 SEM microstructures in UHCSDB. Also, to mitigate the imbalance in the number of images corresponding to different cooling methods, we adjust the number of random crops in such a way that each of the cooling methods have approx. 1400 images. In addition we augment the dataset by rotating the images by 90, 180 and 270 degrees.
3.2 Results
We train the ACWGANGP for 6000 epochs with a batch size of 64 and use Adam Optimizer (kingma2014adam) with a learning rate of , while keeping & . Also, : training ratio is maintained at 5:1. Figure 2 shows a few representative images synthesized by ACWGANGP along with real microstructures for visual comparison. Each column of synthesized images correspond to the same Gaussian noise vector while each row corresponds to a specific cooling method. Although this figure highlights the visual resemblance between real and synthesized microstructures, we present rigorous evaluation methods in what follows.
Furthermore, to perform a quantitative evaluation of the synthesized microstructures, we randomly select 4000 pairs of real and synthesized images and compare their 2point spatial correlations (yeong1998reconstructing). Figure 3(a) shows a good match between the correlation values of the real and the synthesized microstructures. However, for complex, multiphase microstructures, this is only a necessary condition, not sufficient. We provide an alternative evaluation which first extracts appropriate feature vectors by using a pretrained VGG16 and then uses tSNE based dimensionality reduction to project these vectors onto a 2dimensional space. We applied this procedure to 4000 pairs of real & synthesized images to obtain Fig. 3(b). This figure shows vast regions of overlap between real and synthesized microstructures, indicating similarities between them.
4 Conclusion
To the best of our knowledge, this is the first attempt to model processingstructure relationship as a conditional image synthesis problem. To accomplish this objective we have utilized the ACWGANGP framework, which inherits training stability of WGANGP and conditional image generation of ACGAN. We use this framework for reconstructing multiphase microstructures from UHCSDB and demonstrate its capability in synthesizing high quality microstructures from a given cooling method. In our future work, we will introduce additional processing parameters, such as annealing temperature and time, to achieve tighter control and better insight over the synthesized microstructures.