Autonomous Scanning Probe Microscopy in-situ Tip Conditioning through Machine Learning

Autonomous Scanning Probe Microscopy in-situ Tip Conditioning through Machine Learning


Atomic scale characterization and manipulation with scanning probe microscopy rely upon the use of an atomically sharp probe. Here we present automated methods based on machine learning to automatically detect and recondition the quality of the probe of a scanning tunneling microscope. As a model system, we employ these techniques on the technologically relevant hydrogen-terminated silicon surface, training the network to recognize abnormalities in the appearance of surface dangling bonds. Of the machine learning methods tested a convolutional neural network yielded the greatest accuracy, achieving a positive identification of degraded tips in 97% of the test cases. By using multiple points of comparison and majority voting the accuracy of the method is improved beyond 99%. The methods described here can easily be generalized to other material systems and nanoscale imaging techniques.

The ability to directly visualize and manipulate individual atoms using scanning probe microscopy (SPM) Schweizer and Eigler (1990); Crommie et al. (1993); Stroscio and Celotta (2004); Sugimoto et al. (2008); Kalff et al. (2016); Slot et al. (2017); Drost et al. (2017); Huff et al. (2017a); Pavliček et al. (2017); Fölsch et al. (2014); Kawai et al. (2014) has inspired scientists to develop atomic scale technology for over two decades. Among other things, that technology can be used to create smaller, more efficient, faster and cheaper devices Huff et al. (2017b); Khajetoorians et al. (2011). To be practical, a rapid and commercially feasible way must be found to use SPM techniques to fabricate millions of components with atomic precision. To that end, several studies have developed algorithms that make it feasible to build parallelized atomically precise robots that manipulate and analyze atoms automatically Woolley et al. (2011); Castellano-Hernandez and Sacha (2012); Stirling et al. (2013); Møller et al. (2017); Ziatdinov and Maksov (2017); Ziatdinov et al. (2018).

SPM techniques, and atomic manipulation in particular, rely on atomically sharp metal tips. Preparing such tips is a process with several steps. First, a metal wire, commonly tungsten, is electrochemically etched to yield an apex with approximately 10 nm radius of curvature Ju et al. (2011). This step is usually followed by thermal annealing in ultra-high vacuum to remove an oxide layer formed at the apex during the etching process. Optionally, a single atom tip can be formed in a field ion microscope Rezeq et al. (2006), a technique that virtually ensures a clean working tip.

During imaging and atomic manipulation, interaction of the tip with the surface can unintentionally degrade the tip apex. The signature of this is the loss of atomic resolution, or the appearance of secondary imaging features, indicating that the apex of the tip no-longer has a single predominant atom. Such tips are generally called “double tips” for this reason. Because a single atom tip is required for SPM atomic fabrication and experiments, in-situ tip treatments are necessary to return the tip to its ideal (sharp) condition. This is usually the most time-consuming process for SPM operators. Common methods are applying short voltage pulses between the tip and sample or controllably indenting the tip into the sample. These processes typically must be repeated many times before the tip’s quality is restored.

Here, we use machine vision to automate the in-situ tip conditioning process, with limited human intervention. In our automation routine, we train and implement a convolutional neural network (CNN) model to analyze the quality of the tip and obtain a high precision score of  97%. In our case, we work on hydrogen terminated Si(100) substrate, a promising platform to develop atomic circuitry Huff et al. (2017b); Livadaru et al. (2010); Rashidi et al. (2016); Schofield et al. (2013); Fuechsle et al. (2012). We use isolated surface dangling bond images to train the CNN. To detect the quality of the tip we use several dangling bond images and employ majority voting to increase the operational accuracy to beyond 99%. Upon detection of degraded probe quality, the routine performs tip conditioning on a preselected spot on the surface. This procedure is repeated until the network registers a sharp probe.

Figure 1: Randomly selected labeled data used for training. 5.65.6 nm dangling bond images are extracted from the STM images recorded at -1.8 V and 50 pA.
Figure 2: Tip Quality Analysis with Convolutional Neural Network. (a) STM image (100100 nm) of hydrogen-terminated Si(100) recorded at -1.80 V and 50 pA. Bright features are surface dangling bonds. The dangling bonds are automatically extracted from the image (black squares) and sequentially fed into the CNN. (b) Close up of the dangling bond indicated by the red square in (a). (c) A depiction of the CNN used in this study. It consists of two convolution layers followed by a pooling layer, a densely connected layer and an output layer. The result of the output layer is “0” for sharp tips or “1” for double tips. As an example, the output of each CNN layer is shown for the dangling bond image in (b). (d) The outputs of the CNN for each automatically extracted dangling bond image in (a).

All experiments were performed on an Omicron LT STM operating at 4.5 K and under ultrahigh vacuum. The tips were electrochemically etched from polycrystalline tungsten wire. Tips were heated via electron bombardment in ultrahigh vacuum to remove the surface oxide, and sharpened to single atom by field ion microscopy Rezeq et al. (2006). In-situ tip processing was performed by controlled tip indentation with the surface. Samples are highly arsenic-doped (1.510 atom/cm) Si(100). Samples were degassed at 600C for  12 hours followed by flash annealing at 1250C. For hydrogen passivation, they are exposed to atomic hydrogen gas at 330C. A Nanonis SPM controller was used for imaging and data acquisition. The tip conditioning automation routine was programmed in Python and Labview using Nanonis programming interface. K nearest neighbor, random forest, support vector machine and fully connected neural network were implemented using Scikit-Learn(0.19.1), Python machine learning library. The CNN was implemented using Keras(2.1.3) with TensorFlow backend.

For training, we used approximately 3500 STM images of isolated dangling bonds recorded at a sample bias of  V, where they typically appear as bright protrusions. These images were selected from five years of archived data from two of our microscopes. To enable direct comparison, each 5.65.6 nm image was resized to 2828 pixels. Each of the images was labeled manually (Fig. 1). To greatly expand the training dataset we augmented each image by rotating it by 90 four times and mirroring each rotated image. This resulted in a significant performance increase to each of the models we tested.

We tested several machine learning models on our dataset and selected a CNN model to implement in our automation routine because of its high precision score. Table I summarizes the outcome of each model.

Precision 0.84 0.89 0.88 0.78 0.97
Table 1: Precision score of different machine learning models that have been tested for our dataset. KNN, RFC, SVM, FCNN and CNN denote K nearest neighbor, random forest classifier, support vector machine, fully connected neural network and convolutional neural network, respectively. KNN classifier with 5 neighbors resulted the best accuracy score for our dataset. 5000 trees were used for RFC. The (Gaussian) radial basis function kernel with C and parameters of 500 and 0.5 was used for SVM. The FCC had 18 hidden layers with ReLU activation function and Adam optimizer Kingma and Ba (2015) (learning rate of 10).

Figure 2 displays the workflow of the tip quality analysis using a CNN. Our routine automatically identifies and isolates subsections of the STM image containing dangling bonds, and feeds them sequentially to the CNN to analyse the tip quality. The black squares in Fig. 2a indicate the dangling bond images that were used for analysis in this example. As an example, the output of the CNN for a dangling bond image in Fig. 2b is shown in Fig. 2c. The CNN consists of two back to back 30 and 40 kernels (55, stride 1) convolutional layers with ReLU activation function. These layers were followed by a max-pooling layer (22, stride 2) flattened and fully connected to a 128 node layer with ReLU activation function. A 2-node fully connected layer with Softmax activation function was used for classification at the end. The Adam optimizer Kingma and Ba (2015) with learning rate of 10 and the categorical cross-entropy as loss function was used. Figure 2d displays the output of the CNN for all the dangling bonds in the STM image. The program performs majority voting at the end to determine the outcome.

Figure 3: An Example of Automatic Tip Sharpening. (a) Initial image to judge the quality of the tip. User sets an image frame and a spot for in-situ tip conditioning before starting the program. The majority vote outputs of the CNN is “1” (double tip) for this image and 4 other subsequent images (not shown here), indicating that the tip conditioning was not successful. (b) After a successful tip conditioning the majority vote output of the CNN becomes “0” (sharp tip) and the program stops its operation.

Figure 3a shows an STM image obtained with a “double tip”. Our program successfully identified the tip was not ideal and performed tip conditioning in an attempt to restore the tip’s quality. Four conditioning steps were performed, at which point the program successfully recognized that the tip’s quality had been restored (Fig. 3b) and the sequence was terminated. We note that the image frame to assess the quality of the tip must be carefully selected by the user to achieve accurate results. For instance, the defect close to the lower left dangling bond in Fig. 3b results in the outcome of “double tip” for that dangling bond. Because all the other dangling bonds in the image frame are isolated, the program detect an accurate outcome.

To summarize, as a starting point for developing parallel atomic-precision fabrication tools, we have implemented a machine learning based scheme to automate the SPM in-situ probe sharpening. We have used hydrogen terminated silicon surface as a model system. Our automation routine extracts images of selected features (isolated surface dangling bonds in our case) from the scan frame, analyzes them one by one using a CNN, performs majority voting and detects the quality of the tip. If the detection is “double tip”, the program performs tip conditioning and repeats the same procedure until the probe becomes “sharp” again. The framework that we have developed here is an important step towards creating autonomous atomic-scale fabrication tools and can be also straightforwardly generalized to other materials as well as other nanoscale imaging techniques. Applications other than atom scale fabrication, such as critical dimension analysis as used in modern semiconductor fabrication, will also benefit by variants of the techniques described here.

We would like to thank Mark Salomons and Martin Cloutier for their technical expertise. We also thank Alberta Innovates for their support. M.R. thanks Wyatt Vine and Ken Gordon for their help to edit and proofread the manuscript.


  1. D. K. Schweizer and E. K. Eigler, Nature 344, 524 (1990).
  2. M. F. Crommie, C. P. Lutz,  and D. M. Eigler, Science 262, 218 (1993).
  3. J. A. Stroscio and R. J. Celotta, Science 306, 242 (2004).
  4. Y. Sugimoto, P. Pou, O. Custance, P. Jelinek, M. Abe, R. Perez,  and S. Morita, Science 322, 413 (2008).
  5. F. E. Kalff, M. P. Rebergen, E. Fahrenfort, J. Girovsky, R. Toskovic, J. L. Lado, J. Fernández-Rossier,  and A. F. Otte, Nat. Nanotechnol. 11, 926 (2016).
  6. M. R. Slot, T. S. Gardenier, P. H. Jacobse, G. C. P. van Miert, S. N. Kempkes, S. J. M. Zevenhuizen, C. M. Smith, D. Vanmaekelbergh,  and I. Swart, Nat. Phys. 13, 672 (2017).
  7. R. Drost, T. Ojanen, A. Harju,  and P. Liljeroth, Nat. Phys. 13, 668 (2017).
  8. T. R. Huff, H. Labidi, M. Rashidi, M. Koleini, R. Achal, M. H. Salomons,  and R. A. Wolkow, ACS Nano 11, 8636 (2017a).
  9. N. Pavliček, Z. Majzik, G. Meyer,  and L. Gross, Appl. Phys. Lett. 111, 053104 (2017).
  10. S. Fölsch, J. Martínez-Blanco, J. Yang, K. Kanisawa,  and S. C. Erwin, Nat. Nanotechnol. 9, 505 (2014).
  11. S. Kawai, A. S. Foster, F. F. Canova, H. Onodera, S.-i. Kitamura,  and E. Meyer, Nat. Commun. 5, 4403 (2014).
  12. T. Huff, H. Labidi, M. Rashidi, R. Achal, L. Livadaru, T. Dienel, J. Pitters,  and R. A. Wolkow, arXiv:1706.07427  (2017b).
  13. A. A. Khajetoorians, J. Wiebe, B. Chilian,  and R. Wiesendanger, Science 332, 1062 (2011).
  14. R. A. J. Woolley, J. Stirling, A. Radocea, N. Krasnogor,  and P. Moriarty, Appl. Phys. Lett. 98, 253104 (2011).
  15. E. Castellano-Hernandez and G. M. Sacha, Appl. Phys. Lett. 100, 023101 (2012).
  16. J. Stirling, R. A. J. Woolley,  and P. Moriarty, Rev. Sci. Instrum. 84, 113701 (2013).
  17. M. Møller, S. P. Jarvis, L. Guérinet, P. Sharp, R. Woolley, P. Rahe,  and P. Moriarty, Nanotechnology 28, 075302 (2017).
  18. M. Ziatdinov and A. Maksov, npj Computational Materials 3, 31 (2017).
  19. M. Ziatdinov, O. Dyck, A. Maksov, B. M. Hudak, A. R. Lupini, J. Song, P. C. Snijders, R. K. Vasudevan, S. Jesse,  and S. V. Kalinin, arXiv:1801.05133  (2018).
  20. B.-f. Ju, Y.-l. Chen,  and Y. Ge, Rev. Sci. Instrum. 82, 013707 (2011).
  21. M. Rezeq, J. Pitters,  and R. Wolkow, J. Chem. Phys. 124, 204716 (2006).
  22. L. Livadaru, P. Xue, Z. Shaterzadeh-Yazdi, G. a. DiLabio, J. Mutus, J. L. Pitters, B. C. Sanders,  and R. a. Wolkow, New J. Phys. 12, 083018 (2010).
  23. M. Rashidi, M. Taucer, I. Ozfidan, E. Lloyd, M. Koleini,  and H. Labidi, Phys. Rev. Lett. 117, 276805 (2016).
  24. S. R. Schofield, P. Studer, C. F. Hirjibehedin, N. J. Curson, G. Aeppli,  and D. R. Bowler, Nat. Commun. 4, 1649 (2013).
  25. M. Fuechsle, J. A. Miwa, S. Mahapatra, H. Ryu, S. Lee, O. Warschkow, L. C. L. Hollenberg, G. Klimeck,  and M. Y. Simmons, Nat. Nanotechnol. 7, 242 (2012).
  26. D. P. Kingma and J. L. Ba, arXiv:1412.6980  (2015).
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description