Generalized framework for the parallel semantic segmentation of multiple objects and posterior manipulation

The end-to-end approach presented in this paper deals with the recognition, detection, segmentation and grasping of objects, assuming no prior knowledge of the environment nor objects. The proposed pipeline is as follows: 1) Usage of a trained Convolutional Neural Net (CNN) that recognizes up to 80 different classes of objects in real time and generates bounding boxes around them. 2) An algorithm to derive in parallel the pointclouds of said regions of interest (ROI). 3) Eight different segmentation methods to remove background data and noise from the pointclouds and obtain a precise result of the semantically segmented objects. 4) Registration of the object's pointclouds over time to generate the best possible model. 5) Utilization of an algorithm to detect an array of grasping positions and orientations based mainly on the geometry of the object's model. 6) Implementation of the system on the humanoid robot MyBot, developed in the RIT Lab at KAIST. 7) An algorithm to find the bounding box of the object's model in 3D to then create a collision object and add it to the octomap. The collision checking between robot's hand and the object is removed to allow grasping using the MoveIt libraries. 8) Selection of the best grasping pose for a certain object, plus execution of the grasping movement. 9) Retrieval of the object and moving it to a desired final position.