How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Casper Kaae Sønderby; Tapani Raiko; Lars Maaløe; Søren Kaae Sønderby; Ole Winther

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, Ole Winther

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

20746 Downloads (Pure)

Abstract

Variational autoencoders are a powerful framework for unsupervised learning. However, previous work has been restricted to shallow models with one or two layers of fully factorized stochastic latent variables, limiting the flexibility of the latent representation. We propose three advances in training algorithms of variational autoencoders, for the first time allowing to train deep models of up to five stochastic layers, (1) using a structure similar to the Ladder network as the inference model, (2) warm-up period to support stochastic units staying active in early training, and (3) use of batch normalization. Using these improvements we show state-of-the-art log-likelihood results for generative modeling on several benchmark datasets.

Original language	English
Title of host publication	Proceedings of the 33rd International Conference on Machine Learning (ICML 2016)
Number of pages	9
Publication date	2016
Publication status	Published - 2016
Event	33rd International Conference on Machine Learning (ICML 2016) - New York, United States Duration: 19 Jun 2016 → 24 Jun 2016 Conference number: 33 http://icml.cc/2016/

Conference

Conference	33rd International Conference on Machine Learning (ICML 2016)
Number	33
Country/Territory	United States
City	New York
Period	19/06/2016 → 24/06/2016
Internet address	http://icml.cc/2016/

Series	JMLR: Workshop and Conference Proceedings
Volume	48
ISSN	1938-7228

Access to Document

1602.02282Submitted manuscript, 2.2 MB

http://arxiv.org/abs/1602.02282

OpenUrl availability

Full text

Cite this

@inproceedings{4ac1c67ae5a1439db82bda1c175dea94,

title = "How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks",

abstract = "Variational autoencoders are a powerful framework for unsupervised learning. However, previous work has been restricted to shallow models with one or two layers of fully factorized stochastic latent variables, limiting the flexibility of the latent representation. We propose three advances in training algorithms of variational autoencoders, for the first time allowing to train deep models of up to five stochastic layers, (1) using a structure similar to the Ladder network as the inference model, (2) warm-up period to support stochastic units staying active in early training, and (3) use of batch normalization. Using these improvements we show state-of-the-art log-likelihood results for generative modeling on several benchmark datasets.",

author = "S{\o}nderby, {Casper Kaae} and Tapani Raiko and Lars Maal{\o}e and S{\o}nderby, {S{\o}ren Kaae} and Ole Winther",

year = "2016",

language = "English",

series = "JMLR: Workshop and Conference Proceedings",

booktitle = "Proceedings of the 33rd International Conference on Machine Learning (ICML 2016)",

note = "33rd International Conference on Machine Learning (ICML 2016), ICML ; Conference date: 19-06-2016 Through 24-06-2016",

url = "http://icml.cc/2016/",

}

Sønderby, CK, Raiko, T, Maaløe, L, Sønderby, SK & Winther, O 2016, How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks. in Proceedings of the 33rd International Conference on Machine Learning (ICML 2016). JMLR: Workshop and Conference Proceedings, vol. 48, 33rd International Conference on Machine Learning (ICML 2016), New York, United States, 19/06/2016. <http://arxiv.org/abs/1602.02282>

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks. / Sønderby, Casper Kaae; Raiko, Tapani; Maaløe, Lars et al.

Proceedings of the 33rd International Conference on Machine Learning (ICML 2016). 2016. (JMLR: Workshop and Conference Proceedings, Vol. 48).

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

TY - GEN

T1 - How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

AU - Sønderby, Casper Kaae

AU - Raiko, Tapani

AU - Maaløe, Lars

AU - Sønderby, Søren Kaae

AU - Winther, Ole

N1 - Conference code: 33

PY - 2016

Y1 - 2016

N2 - Variational autoencoders are a powerful framework for unsupervised learning. However, previous work has been restricted to shallow models with one or two layers of fully factorized stochastic latent variables, limiting the flexibility of the latent representation. We propose three advances in training algorithms of variational autoencoders, for the first time allowing to train deep models of up to five stochastic layers, (1) using a structure similar to the Ladder network as the inference model, (2) warm-up period to support stochastic units staying active in early training, and (3) use of batch normalization. Using these improvements we show state-of-the-art log-likelihood results for generative modeling on several benchmark datasets.

AB - Variational autoencoders are a powerful framework for unsupervised learning. However, previous work has been restricted to shallow models with one or two layers of fully factorized stochastic latent variables, limiting the flexibility of the latent representation. We propose three advances in training algorithms of variational autoencoders, for the first time allowing to train deep models of up to five stochastic layers, (1) using a structure similar to the Ladder network as the inference model, (2) warm-up period to support stochastic units staying active in early training, and (3) use of batch normalization. Using these improvements we show state-of-the-art log-likelihood results for generative modeling on several benchmark datasets.

M3 - Article in proceedings

T3 - JMLR: Workshop and Conference Proceedings

BT - Proceedings of the 33rd International Conference on Machine Learning (ICML 2016)

T2 - 33rd International Conference on Machine Learning (ICML 2016)

Y2 - 19 June 2016 through 24 June 2016

ER -

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Abstract

Conference

Access to Document

OpenUrl availability

Fingerprint

Cite this