Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots

Adrian Llopart, Ole Ravn, Nils Axel Andersen, Jong-Hwan Kim

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Abstract

Task Intelligence is the capacity of a robot to learn, reason and execute specific behaviours based on its environment. In this paper, the Task Intelligence problem formulated by the Robot Intelligence and Technology Laboratory at KAIST is researched further: specifically the proposed contribution is a brand new perceptual pipeline in which the recognition, detection, segmentation and grasping of objects is achieved assuming no prior knowledge of the environments arrangement nor the objects appearance. A Convolutional Neural Net (CNN) is used to detect, recognize and semantically label those objects that need to be interacted with. 3D point clouds, corresponding to the objects model, are extracted after several segmentation procedures and registered over time. Dimensional and positional information of the object is acquired. Additional grasping pose data is calculated. All of the collected knowledge is parsed so that the Task Intelligence system is able to deal with previously unknown objects in dynamic environments. This system is formed by an Episodic Memory (Deep-ART), an action sequence generator (FF-planner) and a trajectory warping module for pre-learnt behaviours. The proposed approach has been tested using the Webots simulator.
Original languageEnglish
Title of host publicationProceedings of 2018 15th International Conference on Control, Automation, Robotics and Vision
PublisherIEEE
Publication date2018
Pages198-203
ISBN (Print)9781538695821
DOIs
Publication statusPublished - 2018
Event2018 15th International Conference on Control, Automation, Robotics and Vision - Marina Bay Sands Expo and Convention Centre, Singapore, Singapore
Duration: 18 Nov 201821 Nov 2018
https://icarcv.net/

Conference

Conference2018 15th International Conference on Control, Automation, Robotics and Vision
LocationMarina Bay Sands Expo and Convention Centre
CountrySingapore
CitySingapore
Period18/11/201821/11/2018
Internet address
Series2018 15th International Conference on Control, Automation, Robotics and Vision (icarcv)

Keywords

  • Task analysis
  • Three-dimensional displays
  • Encoding
  • Trajectory
  • Pipelines
  • End effectors
  • Task Intelligence
  • Semantic segmentation
  • CNN
  • Episodic Memory
  • Deep-ART
  • Humanoid robot

Cite this

Llopart, A., Ravn, O., Andersen, N. A., & Kim, J-H. (2018). Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots. In Proceedings of 2018 15th International Conference on Control, Automation, Robotics and Vision (pp. 198-203). IEEE. 2018 15th International Conference on Control, Automation, Robotics and Vision (icarcv) https://doi.org/10.1109/ICARCV.2018.8581135
Llopart, Adrian ; Ravn, Ole ; Andersen, Nils Axel ; Kim, Jong-Hwan. / Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots. Proceedings of 2018 15th International Conference on Control, Automation, Robotics and Vision. IEEE, 2018. pp. 198-203 (2018 15th International Conference on Control, Automation, Robotics and Vision (icarcv)).
@inproceedings{98ef4e04926441eeb7968f2c3f66af69,
title = "Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots",
abstract = "Task Intelligence is the capacity of a robot to learn, reason and execute specific behaviours based on its environment. In this paper, the Task Intelligence problem formulated by the Robot Intelligence and Technology Laboratory at KAIST is researched further: specifically the proposed contribution is a brand new perceptual pipeline in which the recognition, detection, segmentation and grasping of objects is achieved assuming no prior knowledge of the environments arrangement nor the objects appearance. A Convolutional Neural Net (CNN) is used to detect, recognize and semantically label those objects that need to be interacted with. 3D point clouds, corresponding to the objects model, are extracted after several segmentation procedures and registered over time. Dimensional and positional information of the object is acquired. Additional grasping pose data is calculated. All of the collected knowledge is parsed so that the Task Intelligence system is able to deal with previously unknown objects in dynamic environments. This system is formed by an Episodic Memory (Deep-ART), an action sequence generator (FF-planner) and a trajectory warping module for pre-learnt behaviours. The proposed approach has been tested using the Webots simulator.",
keywords = "Task analysis, Three-dimensional displays, Encoding, Trajectory, Pipelines, End effectors, Task Intelligence, Semantic segmentation, CNN, Episodic Memory, Deep-ART, Humanoid robot",
author = "Adrian Llopart and Ole Ravn and Andersen, {Nils Axel} and Jong-Hwan Kim",
year = "2018",
doi = "10.1109/ICARCV.2018.8581135",
language = "English",
isbn = "9781538695821",
pages = "198--203",
booktitle = "Proceedings of 2018 15th International Conference on Control, Automation, Robotics and Vision",
publisher = "IEEE",
address = "United States",

}

Llopart, A, Ravn, O, Andersen, NA & Kim, J-H 2018, Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots. in Proceedings of 2018 15th International Conference on Control, Automation, Robotics and Vision. IEEE, 2018 15th International Conference on Control, Automation, Robotics and Vision (icarcv), pp. 198-203, 2018 15th International Conference on Control, Automation, Robotics and Vision, Singapore, Singapore, 18/11/2018. https://doi.org/10.1109/ICARCV.2018.8581135

Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots. / Llopart, Adrian; Ravn, Ole; Andersen, Nils Axel; Kim, Jong-Hwan.

Proceedings of 2018 15th International Conference on Control, Automation, Robotics and Vision. IEEE, 2018. p. 198-203 (2018 15th International Conference on Control, Automation, Robotics and Vision (icarcv)).

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

TY - GEN

T1 - Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots

AU - Llopart, Adrian

AU - Ravn, Ole

AU - Andersen, Nils Axel

AU - Kim, Jong-Hwan

PY - 2018

Y1 - 2018

N2 - Task Intelligence is the capacity of a robot to learn, reason and execute specific behaviours based on its environment. In this paper, the Task Intelligence problem formulated by the Robot Intelligence and Technology Laboratory at KAIST is researched further: specifically the proposed contribution is a brand new perceptual pipeline in which the recognition, detection, segmentation and grasping of objects is achieved assuming no prior knowledge of the environments arrangement nor the objects appearance. A Convolutional Neural Net (CNN) is used to detect, recognize and semantically label those objects that need to be interacted with. 3D point clouds, corresponding to the objects model, are extracted after several segmentation procedures and registered over time. Dimensional and positional information of the object is acquired. Additional grasping pose data is calculated. All of the collected knowledge is parsed so that the Task Intelligence system is able to deal with previously unknown objects in dynamic environments. This system is formed by an Episodic Memory (Deep-ART), an action sequence generator (FF-planner) and a trajectory warping module for pre-learnt behaviours. The proposed approach has been tested using the Webots simulator.

AB - Task Intelligence is the capacity of a robot to learn, reason and execute specific behaviours based on its environment. In this paper, the Task Intelligence problem formulated by the Robot Intelligence and Technology Laboratory at KAIST is researched further: specifically the proposed contribution is a brand new perceptual pipeline in which the recognition, detection, segmentation and grasping of objects is achieved assuming no prior knowledge of the environments arrangement nor the objects appearance. A Convolutional Neural Net (CNN) is used to detect, recognize and semantically label those objects that need to be interacted with. 3D point clouds, corresponding to the objects model, are extracted after several segmentation procedures and registered over time. Dimensional and positional information of the object is acquired. Additional grasping pose data is calculated. All of the collected knowledge is parsed so that the Task Intelligence system is able to deal with previously unknown objects in dynamic environments. This system is formed by an Episodic Memory (Deep-ART), an action sequence generator (FF-planner) and a trajectory warping module for pre-learnt behaviours. The proposed approach has been tested using the Webots simulator.

KW - Task analysis

KW - Three-dimensional displays

KW - Encoding

KW - Trajectory

KW - Pipelines

KW - End effectors

KW - Task Intelligence

KW - Semantic segmentation

KW - CNN

KW - Episodic Memory

KW - Deep-ART

KW - Humanoid robot

U2 - 10.1109/ICARCV.2018.8581135

DO - 10.1109/ICARCV.2018.8581135

M3 - Article in proceedings

SN - 9781538695821

SP - 198

EP - 203

BT - Proceedings of 2018 15th International Conference on Control, Automation, Robotics and Vision

PB - IEEE

ER -

Llopart A, Ravn O, Andersen NA, Kim J-H. Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots. In Proceedings of 2018 15th International Conference on Control, Automation, Robotics and Vision. IEEE. 2018. p. 198-203. (2018 15th International Conference on Control, Automation, Robotics and Vision (icarcv)). https://doi.org/10.1109/ICARCV.2018.8581135