Multi-modal data generation with a deep metric variational autoencoder

Josefine Vilsbøll Sundgaard; Morten Rieger Hannemose; Søren Laugesen; James Harte; Yosuke Kamide; Chiemi  Tanaka; Rasmus Reinhold Paulsen; Anders Nymark Christensen

doi:10.7557/18.6803

Multi-modal data generation with a deep metric variational autoencoder

Josefine Vilsbøll Sundgaard, Morten Rieger Hannemose, Søren Laugesen, James Harte, Yosuke Kamide, Chiemi Tanaka, Rasmus Reinhold Paulsen, Anders Nymark Christensen

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

7 Downloads (Pure)

Abstract

We present a deep metric variational autoencoder for multi-modal data generation. The variational autoencoder employs triplet loss in the latent space, which allows for conditional data generation by sampling new embeddings in the latent space within each class cluster. The approach is evaluated on a multi-modal dataset consisting of otoscopy images of the tympanic membrane with corresponding wideband tympanometry measurements. The modalities in this dataset are correlated, as they represent different aspects of the state of the middle ear, but they do not present a direct pixel-to-pixel correlation. The approach shows promising results for the conditional generation of pairs of images and tympanograms, and will allow for efficient data augmentation of data from multi-modal sources.

Original language	English
Title of host publication	Proceedings of the Northern Lights Deep Learning Workshop 2023
Number of pages	9
Volume	4
Publisher	Septentrio Academic Publishing
Publication date	2023
DOIs	https://doi.org/10.7557/18.6803
Publication status	Published - 2023
Event	Northern Lights Deep Learning Workshop 2023 - Tromsø, Norway Duration: 10 Jan 2023 → 12 Jan 2023

Workshop

Workshop	Northern Lights Deep Learning Workshop 2023
Country/Territory	Norway
City	Tromsø
Period	10/01/2023 → 12/01/2023

Access to Document

10.7557/18.6803Licence: CC BY

FulltextFinal published version, 15.5 MBLicence: CC BY

OpenUrl availability

Full text

Cite this

@inproceedings{d5b18b03701f4153bd58a6456e45b596,

title = "Multi-modal data generation with a deep metric variational autoencoder",

abstract = "We present a deep metric variational autoencoder for multi-modal data generation. The variational autoencoder employs triplet loss in the latent space, which allows for conditional data generation by sampling new embeddings in the latent space within each class cluster. The approach is evaluated on a multi-modal dataset consisting of otoscopy images of the tympanic membrane with corresponding wideband tympanometry measurements. The modalities in this dataset are correlated, as they represent different aspects of the state of the middle ear, but they do not present a direct pixel-to-pixel correlation. The approach shows promising results for the conditional generation of pairs of images and tympanograms, and will allow for efficient data augmentation of data from multi-modal sources.",

author = "Sundgaard, {Josefine Vilsb{\o}ll} and Hannemose, {Morten Rieger} and S{\o}ren Laugesen and James Harte and Yosuke Kamide and Chiemi Tanaka and Paulsen, {Rasmus Reinhold} and Christensen, {Anders Nymark}",

year = "2023",

doi = "10.7557/18.6803",

language = "English",

volume = "4",

booktitle = "Proceedings of the Northern Lights Deep Learning Workshop 2023",

publisher = "Septentrio Academic Publishing",

address = "Norway",

note = " Northern Lights Deep Learning Workshop 2023 ; Conference date: 10-01-2023 Through 12-01-2023",

}

Sundgaard, JV , Hannemose, MR, Laugesen, S, Harte, J, Kamide, Y, Tanaka, C, Paulsen, RR & Christensen, AN 2023, Multi-modal data generation with a deep metric variational autoencoder. in Proceedings of the Northern Lights Deep Learning Workshop 2023. vol. 4, Septentrio Academic Publishing, Northern Lights Deep Learning Workshop 2023, Tromsø, Norway, 10/01/2023. https://doi.org/10.7557/18.6803

Multi-modal data generation with a deep metric variational autoencoder. / Sundgaard, Josefine Vilsbøll ; Hannemose, Morten Rieger; Laugesen, Søren et al.
Proceedings of the Northern Lights Deep Learning Workshop 2023. Vol. 4 Septentrio Academic Publishing, 2023.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

TY - GEN

T1 - Multi-modal data generation with a deep metric variational autoencoder

AU - Sundgaard, Josefine Vilsbøll

AU - Hannemose, Morten Rieger

AU - Laugesen, Søren

AU - Harte, James

AU - Kamide, Yosuke

AU - Tanaka, Chiemi

AU - Paulsen, Rasmus Reinhold

AU - Christensen, Anders Nymark

PY - 2023

Y1 - 2023

N2 - We present a deep metric variational autoencoder for multi-modal data generation. The variational autoencoder employs triplet loss in the latent space, which allows for conditional data generation by sampling new embeddings in the latent space within each class cluster. The approach is evaluated on a multi-modal dataset consisting of otoscopy images of the tympanic membrane with corresponding wideband tympanometry measurements. The modalities in this dataset are correlated, as they represent different aspects of the state of the middle ear, but they do not present a direct pixel-to-pixel correlation. The approach shows promising results for the conditional generation of pairs of images and tympanograms, and will allow for efficient data augmentation of data from multi-modal sources.

AB - We present a deep metric variational autoencoder for multi-modal data generation. The variational autoencoder employs triplet loss in the latent space, which allows for conditional data generation by sampling new embeddings in the latent space within each class cluster. The approach is evaluated on a multi-modal dataset consisting of otoscopy images of the tympanic membrane with corresponding wideband tympanometry measurements. The modalities in this dataset are correlated, as they represent different aspects of the state of the middle ear, but they do not present a direct pixel-to-pixel correlation. The approach shows promising results for the conditional generation of pairs of images and tympanograms, and will allow for efficient data augmentation of data from multi-modal sources.

U2 - 10.7557/18.6803

DO - 10.7557/18.6803

M3 - Article in proceedings

VL - 4

BT - Proceedings of the Northern Lights Deep Learning Workshop 2023

PB - Septentrio Academic Publishing

T2 - Northern Lights Deep Learning Workshop 2023

Y2 - 10 January 2023 through 12 January 2023

ER -

Multi-modal data generation with a deep metric variational autoencoder

Abstract

Workshop

Access to Document

OpenUrl availability

Fingerprint

Cite this