Single-Shot Analysis of Refractive Shape Using Convolutional Neural Networks

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedings – Annual report year: 2019Researchpeer-review



View graph of relations

The appearance of a transparent object is determined by a combination of refraction and reflection, as governed by a complex function of its shape as well as the surrounding environment. Prior works on 3D reconstruction have largely ignored transparent objects due to this challenge, yet they occur frequently in real-world scenes. This paper presents an approach to estimate depths and normals for transparent objects using a single image acquired under a distant but otherwise arbitrary environment map. In particular, we use a deep convolutional neural network (CNN) for this task. Unlike opaque objects, it is challenging to acquire ground truth training data for refractive objects, thus, we propose to use a large-scale synthetic dataset. To accurately capture the image formation process, we use a physically-based renderer. We demonstrate that a CNN trained on our dataset learns to reconstruct shape and estimate segmentation boundaries for transparent objects using a single image, while also achieving generalization to real images at test time. In experiments, we extensively study the properties of our dataset and compare to baselines demonstrating its utility.
Original languageEnglish
Title of host publicationProceedings of IEEE Winter Conference on Applications of Computer Vision
Publication date2019
Publication statusPublished - 2019
Event2019 IEEE Winter Conference on Applications of Computer Vision - Hilton Waikoloa Village, Waikoloa Village, United States
Duration: 7 Jan 201911 Jan 2019


Conference2019 IEEE Winter Conference on Applications of Computer Vision
LocationHilton Waikoloa Village
CountryUnited States
CityWaikoloa Village
Series2019 Ieee Winter Conference on Applications of Computer Vision (wacv)
CitationsWeb of Science® Times Cited: No match on DOI

    Research areas

  • Shape, Glass, Estimation, Three-dimensional displays, Rendering (computer graphics), Computer vision, Image reconstruction
Download as:
Download as PDF
Select render style:
Download as HTML
Select render style:
Download as Word
Select render style:

Download statistics

No data available

ID: 170888835