TY - JOUR
T1 - Tomato harvesting robotic system based on Deep-ToMaToS
T2 - Deep learning network using transformation loss for 6D pose estimation of maturity classified tomatoes with side-stem
AU - Kim, Joon Young
AU - Pyo, Hye Ran
AU - Jang, Inhoon
AU - Kang, Jaehyeon
AU - Ju, Byeong Kwon
AU - Ko, Kwang Eun
N1 - Funding Information:
This study has been conducted with the support of the Korea Institute of Industrial Technology as “Development of holonic manufacturing system for future industrial environment (kitech EO-22-010)”
Publisher Copyright:
© 2022
PY - 2022/10
Y1 - 2022/10
N2 - This paper presents the development of an autonomous harvesting robot system for tomato, a representative crop cultivated in the facility horticulture smart farm. The automated harvesting work using a robotic system is very challenging because of the appearance, environmental features, such as the atypical directions of the peduncles or their growing form in a bunch. Also, the robot system should enable the harvesting of the target fruit only without damaging other fruits, stems, and branches. Hence, this paper presents a deep learning network pipeline, Deep-ToMaToS, capable of three-level maturity classification and 6D pose (3D translation + 3D rotation) estimation of the target fruit simultaneously. Due to the difficulties encountered in building a large-scale dataset to train and test the deep learning model for the 6D pose estimation in the real world, we presented an automatic data collection scheme based on a photo-realistic 3D simulator environment. The robotic harvesting system includes a harvesting motion control algorithm based on the result of the 6D pose estimation. The overall process of the motion control phase is described along with the decision way of the appropriate final posture of the harvesting module mounted at the end-effector of the robot manipulator via removal of invalid motions getting out of the valid workspace or redundant motions. We conducted experiments on the 6D pose estimation based on the Deep-ToMaToS and the harvesting motion control in virtual and real smart farm environments. The experimental results showed a 6D pose estimation accuracy of 96 % based on the ADD_S metric, and the proposed harvesting motion control algorithm achieves the harvesting success rate of 84.5 % on average. The experimental results reveal that the harvesting robot system has significant potential to extend to harvesting works for other fruits and crops.
AB - This paper presents the development of an autonomous harvesting robot system for tomato, a representative crop cultivated in the facility horticulture smart farm. The automated harvesting work using a robotic system is very challenging because of the appearance, environmental features, such as the atypical directions of the peduncles or their growing form in a bunch. Also, the robot system should enable the harvesting of the target fruit only without damaging other fruits, stems, and branches. Hence, this paper presents a deep learning network pipeline, Deep-ToMaToS, capable of three-level maturity classification and 6D pose (3D translation + 3D rotation) estimation of the target fruit simultaneously. Due to the difficulties encountered in building a large-scale dataset to train and test the deep learning model for the 6D pose estimation in the real world, we presented an automatic data collection scheme based on a photo-realistic 3D simulator environment. The robotic harvesting system includes a harvesting motion control algorithm based on the result of the 6D pose estimation. The overall process of the motion control phase is described along with the decision way of the appropriate final posture of the harvesting module mounted at the end-effector of the robot manipulator via removal of invalid motions getting out of the valid workspace or redundant motions. We conducted experiments on the 6D pose estimation based on the Deep-ToMaToS and the harvesting motion control in virtual and real smart farm environments. The experimental results showed a 6D pose estimation accuracy of 96 % based on the ADD_S metric, and the proposed harvesting motion control algorithm achieves the harvesting success rate of 84.5 % on average. The experimental results reveal that the harvesting robot system has significant potential to extend to harvesting works for other fruits and crops.
KW - 6D pose of tomato‘s fruit and side-stem
KW - Autonomous harvesting robot
KW - Peck-and-pull harvesting
KW - Smart farm
UR - http://www.scopus.com/inward/record.url?scp=85136462827&partnerID=8YFLogxK
U2 - 10.1016/j.compag.2022.107300
DO - 10.1016/j.compag.2022.107300
M3 - Article
AN - SCOPUS:85136462827
SN - 0168-1699
VL - 201
JO - Computers and Electronics in Agriculture
JF - Computers and Electronics in Agriculture
M1 - 107300
ER -