Shape Based Deep Estimation of Future Plant Images

Joo Yeon Jung, Sang Ho Lee, Tae Hyeon Kim, Myung Min Oh, Jong Ok Kim

Research output: Contribution to journalArticlepeer-review

4 Citations (Scopus)


Plants exhibit dynamic changes as they grow. For example, a new leaf may appear suddenly, and rotate and fold over time. Therefore, it is difficult to predict the growth of plants. For accurate growth predictions, it is important to predict the shapes, colors and textures of the leaves. The conventional methods simply use RGB images to predict the next plant at once. In this paper, we propose a novel deep network which is divided into two subnets of shape estimation and color reconstruction. Four gray time-series images are first aligned to a future target using a spatial transformer network (STN) for shape estimation. They are then fused using U-Net with two LSTMs to generate a future shape image. The color reconstruction subnet fuses the predicted shape with a RGB plant image to restore the color information. In addition, we use gray images with texture information for shape estimation instead of binary images with only simple information and RGB images with too much information. The proposed deep network can robustly generate future plant images for plant growth prediction. It is evaluated using our proprietary dataset as well as two public datasets for different types of plants. The experimental results demonstrate that our proposed network predicts the leaf shape more accurately and restores RGB. As a result, our method can create accurate future plant images.

Original languageEnglish
Pages (from-to)4763-4776
Number of pages14
JournalIEEE Access
Publication statusPublished - 2022

Bibliographical note

Publisher Copyright:
© 2021 IEEE.


  • Plant growth prediction
  • image generation
  • shape estimation
  • spatial transformer network
  • time-series images

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering


Dive into the research topics of 'Shape Based Deep Estimation of Future Plant Images'. Together they form a unique fingerprint.

Cite this