Diffusion Models for Image Segmentation

Image segmentation is the problem of classifying different regions in images. The goal of universal image segmentation is to segment many types of data, e.g. detecting all objects in all photographs. In contrast, the narrow version of the problem is usually concerned with a single dataset, e.g. segmentation of cancer in the brain. The narrow problem is well-studied and effectively solved with deep learning, as long as enough data is available.
Data availability is a common issue, which often leads researchers to use universal models to solve their narrow problem, since it avoids needing to annotate a lot of data. Current universal models are rarely generative, and instead return a mean prediction. We argue this limitation is significant for current universal models, as many universal models are only able to do binary segmentation problems as a consequence.

The project will investigate the possibility of using diffusion models as a generative universal image segmenter. Diffusion models are primarily known for text-to-image generation, but we believe they have much untapped potential in producing segmentations instead. Our aim is to enable researchers and scientists to go from capturing raw data to segmentation in as few steps as possible.