Differentiable formulations for inverse rendering

Research output: Book/ReportPh.D. thesis

807 Downloads (Pure)


Computer vision has become ubiquitous in the modern world and is applied in both industrial and consumer applications, where it is used for things such as pose estimation and 3D scanning. However, most of the algorithms used for this have a disadvantage. They work by computing positions of points, such as corners or 3D points, and then use only these points to compute the result. This is an effective way of reducing the amount of data taken into consideration, but it disregards possible uncertainties or ambiguities in the detected points. In this thesis, we will address this and other issues by using inverse rendering. In inverse rendering, we fit a model to the objects present in the image. To fit the model we use optimization, which is faster and significantly more robust when our model is differentiable. This is why we focus on differentiable models in this thesis. Our goal is to develop practical methods for common problems in computer vision that are more accurate or widely applicable than previous methods. Specifically, we contribute with two methods for common problems using differentiable inverse rendering. The first is a method for camera calibration that uses all pixels of the calibration artifact thereby overcoming problems of relying on a corner detector. The
second is a method for 3D scanning, where our method can reconstruct a 3D mesh directly from images taken with a structured light 3D scanner, which enables us to bypass the usual mesh reconstruction step. In addition to this, we also provide a method for video frame interpolation, where we use a neural network-based method to generate interpolated frames. We train the neural network to minimize the differences between real and interpolated images, as a form of inverse rendering. Additionally, we present a practical method for estimating the pose of an object and a light source, again using a differentiable model. The estimated poses can be used to compare a photograph to a rendering, to quantify their differences. Finally, we also present applications within industrial automation and augmented reality, where camera calibration and 3D scanning play a key role.
Original languageEnglish
PublisherTechnical University of Denmark
Number of pages138
Publication statusPublished - 2020


Dive into the research topics of 'Differentiable formulations for inverse rendering'. Together they form a unique fingerprint.

Cite this