Bayesian dropout

In the past decade, Dropout has emerged as a powerful and simple method for training neural networks preventing co-adaptation by stochastically omitting neurons. Dropout is currently not grounded in explicit modelling assumptions which so far has precluded its adoption in Bayesian modeling. Using Bayesian entropic reasoning we show that dropout can be interpreted as optimal inference under constraints. We demonstrate this on an analytically tractable regression model providing a Bayesian interpretation of its mechanism for regularizing and preventing co-adaptation as well as its connection to other Bayesian techniques, and in our experiments we find that dropout can provide robustness under model misspecification. Our framework roots dropout as a theoretically justified and practical tool for statistical modeling allowing Bayesian practitioners to tap into the benefits of dropout training.