Towards Hierarchical Discrete Variational Autoencoders

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

1 Downloads (Pure)


Variational Autoencoders (VAEs) have proven to be powerful latent variable models. However, the form of the approximate posterior can limit the expressiveness of the model. Categorical distributions are flexible and useful building blocks for example in neural memory layers. We introduce the Hierarchical Discrete Variational Autoencoder (HD-VAE): a hierarchy of variational memory layers. The Concrete/Gumbel-Softmax relaxation allows maximizing a surrogate of the Evidence Lower Bound by stochastic gradient ascent. We show that, when using a limited number of latent variables, HD-VAE outperforms the Gaussian baseline on modelling multiple binary image datasets. Training very deep HD-VAE remains a challenge due to the relaxation bias that is induced by the use of a surrogate objective. We introduce a formal definition and conduct a preliminary theoretical and empirical study of the bias.
Original languageEnglish
Title of host publicationProceedings of 2nd Symposium on Advances in Approximate Bayesian Inference, 20
Number of pages16
PublisherInternational Machine Learning Society (IMLS)
Publication date2020
Publication statusPublished - 2020
Event2nd Symposium on Advances in Approximate Bayesian Inference - Pan Pacific Hotel Vancouver, Vancouver, Canada
Duration: 8 Dec 20208 Dec 2020


Conference2nd Symposium on Advances in Approximate Bayesian Inference
LocationPan Pacific Hotel Vancouver
SeriesProceedings of Machine Learning Research


Dive into the research topics of 'Towards Hierarchical Discrete Variational Autoencoders'. Together they form a unique fingerprint.

Cite this