Machine Learning for Assisting Atomic- Resolution Electron Microscopy

Matthew Helmi Leth Larsen

Research output: Book/ReportPh.D. thesis

13 Downloads (Pure)


This thesis investigates how deep learning neural networks can solve two significant problems that currently hinder the field of atomic resolution transmission electron microscopy. The first problem is the large-scale analysis of data. Reliable information for materials design must be quantitative and statistically significant. While modern electron microscopes can produce large data sets of high resolution images, the data is typically analysed manually by an electron microscopist, which is a tedious and time-consuming task.
The second issue is the problem of electron beam influence on the material samples. The high-energy electrons in a microscope will inevitably influence the sample, both by directly damaging it and by inducing diffusion and chemical reactions.
To tackle these issues, the thesis presents a software pipeline that utilises simulated high-resolution transmission electron microscopy (HR-TEM) images to train robust neural networks. The software pipeline generates thousands of varied atomic systems of oxide supported metallic nanoparticles and 2-Dimensional monolayer nanoflakes. From these atomic systems, the user can generate thousands of HR-TEM images with assorted microscope conditions, by varying the contrast transfer function, presence of noise and the modulation transfer function, which enables image simulations for different electron detectors. The generated HR-TEM images are then paired with a ground truth label that defines the task the neural network should solve. The user can select from three labels: Mask labels, Disk labels, and Exitwave labels.
A study was made training neural networks with Exitwave labels paired with images of MoS2 nanoflakes (2H phase) to perform exit wave reconstructions. The study presents that neural networks can perform exit wave reconstructions with smaller focal series and no information of the specific CTF parameters that are comparable to traditional algorithms. The exit waves reconstructed by the neural networks permit structural determination of 1Mo, 2S, and 1S atomic columns via Argand plots, but were limited in differentiating 1S atomic columns that exist in the upper or lower sulphur layer with respect to the optical axis. The study also shows that the neural networks perform best when trained on data sets of a single type of atomic system with minimal complexity.
Neural networks trained with Mask labels can perform binary segmentations of Au nanoparticles on CeO2. The binary segmentation maps can be separated via a watershed algorithm, producing a multi-valued map that separates every instance of a nanoparticle in the image. Tools are implemented in a graphical user interface based software to track these instances across all frames. These tools facilitate large-scale data analysis by allowing for properties of each instance to be contained across frames and extracted by the user with the click of a button. This work shows that the segmentations can be leveraged to provide Fourier transforms of each isolated nanoparticle across all frames with a high enough resolution that crystal planes corresponding to different regions of a twinned nanoparticle can be easily distinguished. These tools allow for a statistically significant quantification of various dynamic properties.
The thesis presents a study that gauges the ability of the U-net and MSD-net neural network architectures included in the software pipeline to perform mask segmentations at low signal-to-noise ratios. The study identifies a lower limit for a reliable neural network segmentation of a CeO2 supported Au nanoparticle at 200 e −2. The MSDnet presents an enhanced ability to differentiate between signal and noise and perform reasonable segmentations below the lower limit of the training data, which shows a strong generalisability. The study also highlights the importance of modelling the modulation transfer function to optimise the segmentations by the neural networks in all electron dose regimes. This provides intuition into the minimal electron beam conditions that still allow for reliable neural network data extraction.
Disk labels are shown to train neural networks to perform multi-class segmentations of individual atomic columns in monolayer MoS2 nanoflakes (2H phase). Multi-class segmentation consists of both identifying the 1Mo, 2S, and 1S atomic columns and classifying them. The U-net, U-net++, and MSD-net neural network architectures in the software pipeline all display a powerful ability to perform the multi-class segmentation when provided a simulated focal series of at least 3 HR-TEM images over a large range of microscope conditions. This presents the ability for neural networks to interpret the intricate intensity variations in HR-TEM images and classify various atomic columns to identify local structure and defects in monolayer MoS2.
This work shows that deep learning neural networks provide a powerful tool for analysing atomic-resolution image sequences captured by transmission electron microscopes. The thesis presents a software pipeline that allows for the generation of large and diverse data sets, the pairing of simulated images with ground truth labels, the training of multiple neural network architectures, and tools to apply the trained neural networks for various tasks. The work also highlights the limitations of neural networks and the importance of modelling noise parameters to optimise results. Overall, this work contributes to the field of electron microscopy and paves the way for future automated analysis of atomic-resolution transmission electron microscopy data.
Original languageEnglish
PublisherDepartment of Physics, Technical University of Denmark
Number of pages160
Publication statusPublished - 2023


Dive into the research topics of 'Machine Learning for Assisting Atomic- Resolution Electron Microscopy'. Together they form a unique fingerprint.

Cite this