TY - JOUR
T1 - HF-UNet
T2 - Learning Hierarchically Inter-Task Relevance in Multi-Task U-Net for Accurate Prostate Segmentation in CT Images
AU - He, Kelei
AU - Lian, Chunfeng
AU - Zhang, Bing
AU - Zhang, Xin
AU - Cao, Xiaohuan
AU - Nie, Dong
AU - Gao, Yang
AU - Zhang, Junfeng
AU - Shen, Dinggang
N1 - Funding Information:
Manuscript received February 13, 2021; revised April 2, 2021; accepted April 6, 2021. Date of publication April 13, 2021; date of current version July 30, 2021. This work was supported by the Jiangsu Provincial Key Research and Development Program under Grant BE2020620, Grant BE2020723, and Grant BE2018610. (Corresponding authors: Junfeng Zhang; Dinggang Shen.) Kelei He and Junfeng Zhang are with the Medical School, Nanjing University, Nanjing 210023, China, and also with the National Institute of Healthcare Data Science, Nanjing University, Nanjing 210023, China (e-mail: jfzhang@nju.edu.cn).
Publisher Copyright:
© 1982-2012 IEEE.
PY - 2021/8
Y1 - 2021/8
N2 - Accurate segmentation of the prostate is a key step in external beam radiation therapy treatments. In this paper, we tackle the challenging task of prostate segmentation in CT images by a two-stage network with 1) the first stage to fast localize, and 2) the second stage to accurately segment the prostate. To precisely segment the prostate in the second stage, we formulate prostate segmentation into a multi-task learning framework, which includes a main task to segment the prostate, and an auxiliary task to delineate the prostate boundary. Here, the second task is applied to provide additional guidance of unclear prostate boundary in CT images. Besides, the conventional multi-task deep networks typically share most of the parameters (i.e., feature representations) across all tasks, which may limit their data fitting ability, as the specificity of different tasks are inevitably ignored. By contrast, we solve them by a hierarchically-fused U-Net structure, namely HF-UNet. The HF-UNet has two complementary branches for two tasks, with the novel proposed attention-based task consistency learning block to communicate at each level between the two decoding branches. Therefore, HF-UNet endows the ability to learn hierarchically the shared representations for different tasks, and preserve the specificity of learned representations for different tasks simultaneously. We did extensive evaluations of the proposed method on a large planning CT image dataset and a benchmark prostate zonal dataset. The experimental results show HF-UNet outperforms the conventional multi-task network architectures and the state-of-the-art methods.
AB - Accurate segmentation of the prostate is a key step in external beam radiation therapy treatments. In this paper, we tackle the challenging task of prostate segmentation in CT images by a two-stage network with 1) the first stage to fast localize, and 2) the second stage to accurately segment the prostate. To precisely segment the prostate in the second stage, we formulate prostate segmentation into a multi-task learning framework, which includes a main task to segment the prostate, and an auxiliary task to delineate the prostate boundary. Here, the second task is applied to provide additional guidance of unclear prostate boundary in CT images. Besides, the conventional multi-task deep networks typically share most of the parameters (i.e., feature representations) across all tasks, which may limit their data fitting ability, as the specificity of different tasks are inevitably ignored. By contrast, we solve them by a hierarchically-fused U-Net structure, namely HF-UNet. The HF-UNet has two complementary branches for two tasks, with the novel proposed attention-based task consistency learning block to communicate at each level between the two decoding branches. Therefore, HF-UNet endows the ability to learn hierarchically the shared representations for different tasks, and preserve the specificity of learned representations for different tasks simultaneously. We did extensive evaluations of the proposed method on a large planning CT image dataset and a benchmark prostate zonal dataset. The experimental results show HF-UNet outperforms the conventional multi-task network architectures and the state-of-the-art methods.
KW - Multi-task learning
KW - attention
KW - boundary-aware
KW - consistency learning
KW - prostate cancer
KW - segmentation
UR - http://www.scopus.com/inward/record.url?scp=85104259264&partnerID=8YFLogxK
U2 - 10.1109/TMI.2021.3072956
DO - 10.1109/TMI.2021.3072956
M3 - Article
AN - SCOPUS:85104259264
SN - 0278-0062
VL - 40
SP - 2118
EP - 2128
JO - IEEE Transactions on Medical Imaging
JF - IEEE Transactions on Medical Imaging
IS - 8
M1 - 9402788
ER -