Probabilistic 3D Reconstruction

Deep learning has emerged as a powerful paradigm to create accurate 3D maps for autonomous agents. This 3D awareness enable robots to navigate and interact with the world. However, the deep learning methods to create such maps often come without any notion of uncertainties, which can have catastrophic consequences if wrong predictions are propagated through the system.
This thesis explores the de-facto pipeline for 3D reconstruction and how uncertainties can be incorporated into each step to ensure the safe deployment of autonomous agents.
More specifically, we present a new mental model for understanding uncertainties in deep learning, namely learned versus deduced uncertainties, that originates in pragmatic considerations and offers practical guidelines on how to model uncertainties. The thesis chronologically makes improvements to the four steps of the reconstruction pipeline: 1. Retrieval. We present a Bayesian training procedure to deduce uncertainties for stochastic representations that reduces the risk of silent errors in image retrieval. 2. Structure from Motion. We propose a detector-agnostic method to estimate the uncertainties of deep keypoint detectors and show that the deduced uncertainties improve camera localization accuracy. 3. Multiview Stereo. We present a factorization of dynamic 3D maps that is memory efficient and enable fast training and rendering. Further, we present a probabilistic model to distill a learned 3D prior of local shapes into the reconstruction process. 4. 3D reasoning. Last, we present a novel framework to interact with and manipulate 3D maps in a semantically consistent manner.