A Conditional Generative Model for Predicting Material Microstructures from Processing Methods

# A Conditional Generative Model for Predicting Material Microstructures from Processing Methods

Akshay Iyer
Northwestern University
Evanston, IL 60208
akshayiyer2021@u.northwestern.edu
Siemens Corporate Technology
Princeton, NJ 08536
\AND
Arindam Dasgupta
Siemens Corporate Technology
Princeton, NJ 08536
arindam.dasgupta@siemens.com
&Wei Chen
Northwestern University
Evanston, IL 60208
weichen@northwestern.edu
\AND
Amit Chakraborty
Siemens Corporate Technology
Princeton, NJ 08536
amit.chakraborty@siemens.com
The work was done during this author’s internship at Siemens Corporate Technology.
###### Abstract

Microstructures of a material form the bridge linking processing conditions - which can be controlled, to the material property - which is the primary interest in engineering applications. Thus a critical task in material design is establishing the processing-structure relationship, which requires domain expertise and techniques that can model the high-dimensional material microstructure. This work proposes a deep learning based approach that models the processing-structure relationship as a conditional image synthesis problem. In particular, we develop an auxiliary classifier Wasserstein GAN with gradient penalty (ACWGAN-GP) to synthesize microstructures under a given processing condition. This approach is free of feature engineering, requires modest domain knowledge and is applicable to a wide range of material systems. We demonstrate this approach using the ultra high carbon steel (UHCS) database, where each microstructure is annotated with a label describing the cooling method it was subjected to. Our results show that ACWGAN-GP can synthesize high-quality multiphase microstructures for a given cooling method.

## 1 Introduction

Delineating processingstructureproperty relationships constitutes a major focus in design of advanced material systems (olson1997computational). While analytical and statistical methods have been successfully used for design of certain materials (florescu2009designer; fullwood2010microstructure; lee2017concurrent), the underlying assumptions on homogeneity and isotropy limit their generalizability and transferability to other material systems. To address these challenges, machine learning and data-driven techniques have piqued interest in the material science community. Microstructure reconstruction, by allowing an effective method to understand the high dimensional microstructure space, plays a critical role in computational material design. Prior work along this line has used deep learning to predict material property from microstructure (cecen2018material; cang2018improving; yang2018microstructural), reconstruct statistically equivalent microstructures (li2018transfer), and synthesize microstructures with desired properties (yang2018microstructural; cang2018improving).

Generative models, such as variational autoencoders (VAE) (kingma2014adam) and generative adversarial networks (GAN) (goodfellow2014generative), are key enablers of deep learning based microstructure reconstruction. cang2018improving used VAEs to synthesize two-phase microstructures and demonstrated that convolutional networks can be used for material property prediction. yang2018microstructural used deep convolutional GAN to synthesize microstructures and transfer learning to improve structure-property predictions. Both of these works augmented the generative model loss function with style transfer and mode collapse losses. singh2018physics leveraged WGAN-GP and used generative invariance checker & discriminator concurrently to generate two-phase microstructures.

However, this line of work on using generative models for microstructure reconstruction has two limitations. First, previous works have focused on two-phase microstructures, while many material systems comprise multiphase microstructures. Characterization and reconstruction of multiphase materials have been studied scarcely, especially due to the fact that evaluating higher order correlation for multiphase materials is challenging. To the best of our knowledge, transfer learning technique by li2018transfer is the only method that has been used to reconstruct multiphase microstructures. This method uses first few convolutional layers of a pre-trained VGG16 (simonyan2014very) network to minimize difference in Gram Matrices of the target and reconstructed image. Although accurate, this method can only reconstruct images that match a single target microstructure and cannot model a distribution in the way generative models do. Second, previous works do not account for the influence of processing conditions on microstructure. This is a key aspect in material design since we strive to not only design an optimal microstructure but also identify the processing conditions necessary to manufacture it.

We address these two challenges by developing an auxiliary classifier Wasserstein GAN with gradient penalty (ACWGAN-GP) to synthesize multiphase alloy microstructures from user defined processing methods and demonstrate this approach using the Ultra High Carbon Steel Database (decost2017uhcsdb). Modelling this dataset is extremely challenging owing to the multiphase, heterogeneous microstructures it contains. The key contributions of this work are:

• [noitemsep,topsep=-10pt]

• We demonstrate that GANs can synthesize multiphase microstructure images.

• We demonstrate that ACWGAN-GP enables us to condition the generator on a critical processing condition, namely the cooling method.

• We use VGG16-based feature extractor and t-SNE to validate the proposed approach.

## 2 Learning Framework

### 2.1 Auxiliary Classifier Wasserstein GAN with Gradient Penalty (ACWGAN-GP)

To learn underlying latent distributions from high-dimensional data, such as images, GANs formulate the learning problem as a two-player zero-sum game between a generator (which tries to generate synthetic images indistinguishable from the real ones) and a discriminator (which tries to distinguish whether an image is real or has been synthesized). Furthermore, by imposing additional structure into the GAN latent space, conditional generative models provide an efficient means to better control features in synthesized samples. Conditional GAN (mirza2014conditional), by providing side information (e.g. class labels) to both generator and discriminator, proposes an implementation of this approach and subsequently improves visual quality and diversity of the synthesized images. A later work by odena2017conditional introduces the auxiliary classifier GAN (AC-GAN) architecture which, in addition to using class labels for synthesizing class conditional image samples, also includes a classifier which predicts class labels for images. This current work expands upon this architecture to synthesize microstructures from given values of processing parameters.

After GAN was introduced by goodfellow2014generative, several improvements have been proposed to achieve stable training and fast convergence. Wasserstein GAN (arjovsky2017wasserstein), by using earth-mover distance (Wasserstein-1 metric) as a geometrically meaningful measure of mismatch between probability distributions, introduced a loss function that correlates with the quality of synthesized images. A later work by gulrajani2017improved has shown that penalizing the -norm of discriminator gradient, in stead of constraining the discriminator weights, provides a better alternative to enforce the Lipschitz constraint. This change in the discriminator training results in faster convergence as well as overall improvement in the quality of synthesized images.

In this work, to synthesize alloy microstructures conditioned on processing methods, we develop a hybrid framework that uses label conditioning suggested in AC-GAN architecture but leverages Wasserstein-1 metric to design loss functions. Additionally, we use gradient penalty instead of weight clipping for discriminator training. In particular, the generator () and the critic () in this ACWGAN-GP framework tries to minimize the following loss functions

and

 \scalebox0.95$LC=E~x∼PG[D(~x)]−Ex∼Pr[D(x)]% Discriminator loss+λ1E^x∼P^x[(∥∇^xD(^x)∥2−1)2]Gradient-penalty+λ2E[−log(Prob[C(x)=cx])]Sparse Cross-entropy over all images$,

where, , , is the true label associated with an image (both real and synthesized) , and and represent the distributions of real and synthesized images, respectively. Moreover, denote the distribution of samples which have been drawn uniformly along straight lines between pairs of images sampled from and .

### 2.2 Network architecture and training

In this work, we represent the cooling method via a 20-dimensional embedding vector and concatenate it with a 100-dimensional Gaussian noise vector. This 120-dimensional combined vector is then supplied to the generator . The first layer of is a Fully connected network with 1024 neurons, followed by a dropout layer with rate = 0.25. This is followed by three Upsampling-Convolution-Leaky ReLU blocks and a final Upsampling-Convolution block with -activation. The critic is an approximate mirror image of the generator with four Convolution-Leaky ReLU blocks. These operations extract a 1024-dimensional feature vector which is then passed through a dropout layer with rate = 0.25 and subsequently used by two separate fully connected layers to determine the image score and the corresponding cooling method.

## 3 Experiment

### 3.1 The Ultra High Carbon Steel DataBase (UHCSDB)

UHCSDB is a collection of 961 microstructures obtained from Scanning Electron Microscopy (SEM) of samples with identical composition but subjected to varied heat treatments. The variation in heat treatment influences the microstructure and relevant properties. SEM microstructures curated in UHCSDB correspond to 5 different cooling methods (no heat treatment, quenching, furnace cooling, air cooling and constant heating at C for 1 Hour). Within this database, we focus on microstructures captured at a magnification of 10.3 pix/micron. To train our ACWGAN-GP model to synthesize microstructures of size 128x128 pixels, we created a dataset containing random crops from the original 172 SEM microstructures in UHCSDB. Also, to mitigate the imbalance in the number of images corresponding to different cooling methods, we adjust the number of random crops in such a way that each of the cooling methods have approx. 1400 images. In addition we augment the dataset by rotating the images by 90, 180 and 270 degrees.

### 3.2 Results

We train the ACWGAN-GP for 6000 epochs with a batch size of 64 and use Adam Optimizer (kingma2014adam) with a learning rate of , while keeping & . Also, : training ratio is maintained at 5:1. Figure 2 shows a few representative images synthesized by ACWGAN-GP along with real microstructures for visual comparison. Each column of synthesized images correspond to the same Gaussian noise vector while each row corresponds to a specific cooling method. Although this figure highlights the visual resemblance between real and synthesized microstructures, we present rigorous evaluation methods in what follows.

Furthermore, to perform a quantitative evaluation of the synthesized microstructures, we randomly select 4000 pairs of real and synthesized images and compare their 2-point spatial correlations (yeong1998reconstructing). Figure 3(a) shows a good match between the correlation values of the real and the synthesized microstructures. However, for complex, multiphase microstructures, this is only a necessary condition, not sufficient. We provide an alternative evaluation which first extracts appropriate feature vectors by using a pre-trained VGG16 and then uses t-SNE based dimensionality reduction to project these vectors onto a 2-dimensional space. We applied this procedure to 4000 pairs of real & synthesized images to obtain Fig. 3(b). This figure shows vast regions of overlap between real and synthesized microstructures, indicating similarities between them.

## 4 Conclusion

To the best of our knowledge, this is the first attempt to model processing-structure relationship as a conditional image synthesis problem. To accomplish this objective we have utilized the ACWGAN-GP framework, which inherits training stability of WGAN-GP and conditional image generation of ACGAN. We use this framework for reconstructing multiphase microstructures from UHCSDB and demonstrate its capability in synthesizing high quality microstructures from a given cooling method. In our future work, we will introduce additional processing parameters, such as annealing temperature and time, to achieve tighter control and better insight over the synthesized microstructures.

## References

You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters