The physical properties of polycrystalline materials depend on their microstructure, which is the nano- to centimeter scale arrangement of phases and defects in their interior. Such microstructure depends on the shape, crystallographic phase and orientation, and interfacing of the grains constituting the material. This article presents a new non-destructive 3D technique to study centimeter-sized bulk samples with a spatial resolution of hundred micrometers: time-of-flight three-dimensional neutron diffraction (ToF 3DND). Compared to existing analogous X-ray diffraction techniques, ToF 3DND enables studies of samples that can be both larger in size and made of heavier elements. Moreover, ToF 3DND facilitates the use of complicated sample environments. The basic ToF 3DND setup, utilizing an imaging detector with high spatial and temporal resolution, can easily be implemented at a time-of-flight neutron beamline. The technique was developed and tested with data collected at the Materials and Life Science Experimental Facility of the Japan Proton Accelerator Complex (J-PARC) for an iron sample. We successfully reconstructed the shape of 108 grains and developed an indexing procedure. The reconstruction algorithms have been validated by reconstructing two stacked Co-Ni-Ga single crystals, and by comparison with a grain map obtained by post-mortem electron backscatter diffraction (EBSD).