Medical image synthesis with context-aware generative adversarial networks

Dong Nie; Roger Trullo; Jun Lian; Caroline Petitjean; Su Ruan; Qian Wang; Dinggang Shen

doi:10.1007/978-3-319-66179-7_48

Medical image synthesis with context-aware generative adversarial networks

Dong Nie, Roger Trullo, Jun Lian, Caroline Petitjean, Su Ruan, Qian Wang, Dinggang Shen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

511 Citations (Scopus)

Abstract

Computed tomography (CT) is critical for various clinical applications, e.g., radiation treatment planning and also PET attenuation correction in MRI/PET scanner. However, CT exposes radiation during acquisition, which may cause side effects to patients. Compared to CT, magnetic resonance imaging (MRI) is much safer and does not involve radiations. Therefore, recently researchers are greatly motivated to estimate CT image from its corresponding MR image of the same subject for the case of radiation planning. In this paper, we propose a data-driven approach to address this challenging problem. Specifically, we train a fully convolutional network (FCN) to generate CT given the MR image. To better model the nonlinear mapping from MRI to CT and produce more realistic images, we propose to use the adversarial training strategy to train the FCN. Moreover, we propose an image-gradient-difference based loss function to alleviate the blurriness of the generated CT. We further apply Auto-Context Model (ACM) to implement a context-aware generative adversarial network. Experimental results show that our method is accurate and robust for predicting CT images from MR images, and also outperforms three state-of-the-art methods under comparison.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings
Editors	Lena Maier-Hein, Alfred Franz, Pierre Jannin, Simon Duchesne, Maxime Descoteaux, D. Louis Collins
Publisher	Springer Verlag
Pages	417-425
Number of pages	9
ISBN (Print)	9783319661780
DOIs	https://doi.org/10.1007/978-3-319-66179-7_48
Publication status	Published - 2017
Event	20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017 - Quebec City, Canada Duration: 2017 Sept 11 → 2017 Sept 13

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10435 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017
Country/Territory	Canada
City	Quebec City
Period	17/9/11 → 17/9/13

Bibliographical note

Publisher Copyright:
© Springer International Publishing AG 2017.

Keywords

Auto-context
Deep learning
GAN
Generative models
Image synthesis

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-319-66179-7_48

Cite this

Nie, D., Trullo, R., Lian, J., Petitjean, C., Ruan, S., Wang, Q., & Shen, D. (2017). Medical image synthesis with context-aware generative adversarial networks. In L. Maier-Hein, A. Franz, P. Jannin, S. Duchesne, M. Descoteaux, & D. L. Collins (Eds.), Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings (pp. 417-425). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10435 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-66179-7_48

Medical image synthesis with context-aware generative adversarial networks. / Nie, Dong; Trullo, Roger; Lian, Jun et al.
Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings. ed. / Lena Maier-Hein; Alfred Franz; Pierre Jannin; Simon Duchesne; Maxime Descoteaux; D. Louis Collins. Springer Verlag, 2017. p. 417-425 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10435 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Nie, D, Trullo, R, Lian, J, Petitjean, C, Ruan, S, Wang, Q & Shen, D 2017, Medical image synthesis with context-aware generative adversarial networks. in L Maier-Hein, A Franz, P Jannin, S Duchesne, M Descoteaux & DL Collins (eds), Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10435 LNCS, Springer Verlag, pp. 417-425, 20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017, Quebec City, Canada, 17/9/11. https://doi.org/10.1007/978-3-319-66179-7_48

Nie D, Trullo R, Lian J, Petitjean C, Ruan S, Wang Q et al. Medical image synthesis with context-aware generative adversarial networks. In Maier-Hein L, Franz A, Jannin P, Duchesne S, Descoteaux M, Collins DL, editors, Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings. Springer Verlag. 2017. p. 417-425. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-66179-7_48

Nie, Dong ; Trullo, Roger ; Lian, Jun et al. / Medical image synthesis with context-aware generative adversarial networks. Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings. editor / Lena Maier-Hein ; Alfred Franz ; Pierre Jannin ; Simon Duchesne ; Maxime Descoteaux ; D. Louis Collins. Springer Verlag, 2017. pp. 417-425 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{833a4c8fa53c46dc82670ac723614bc0,

title = "Medical image synthesis with context-aware generative adversarial networks",

abstract = "Computed tomography (CT) is critical for various clinical applications, e.g., radiation treatment planning and also PET attenuation correction in MRI/PET scanner. However, CT exposes radiation during acquisition, which may cause side effects to patients. Compared to CT, magnetic resonance imaging (MRI) is much safer and does not involve radiations. Therefore, recently researchers are greatly motivated to estimate CT image from its corresponding MR image of the same subject for the case of radiation planning. In this paper, we propose a data-driven approach to address this challenging problem. Specifically, we train a fully convolutional network (FCN) to generate CT given the MR image. To better model the nonlinear mapping from MRI to CT and produce more realistic images, we propose to use the adversarial training strategy to train the FCN. Moreover, we propose an image-gradient-difference based loss function to alleviate the blurriness of the generated CT. We further apply Auto-Context Model (ACM) to implement a context-aware generative adversarial network. Experimental results show that our method is accurate and robust for predicting CT images from MR images, and also outperforms three state-of-the-art methods under comparison.",

keywords = "Auto-context, Deep learning, GAN, Generative models, Image synthesis",

author = "Dong Nie and Roger Trullo and Jun Lian and Caroline Petitjean and Su Ruan and Qian Wang and Dinggang Shen",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG 2017.; 20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017 ; Conference date: 11-09-2017 Through 13-09-2017",

year = "2017",

doi = "10.1007/978-3-319-66179-7_48",

language = "English",

isbn = "9783319661780",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "417--425",

editor = "Lena Maier-Hein and Alfred Franz and Pierre Jannin and Simon Duchesne and Maxime Descoteaux and Collins, {D. Louis}",

booktitle = "Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings",

}

TY - GEN

T1 - Medical image synthesis with context-aware generative adversarial networks

AU - Nie, Dong

AU - Trullo, Roger

AU - Lian, Jun

AU - Petitjean, Caroline

AU - Ruan, Su

AU - Wang, Qian

AU - Shen, Dinggang

N1 - Publisher Copyright: © Springer International Publishing AG 2017.

PY - 2017

Y1 - 2017

N2 - Computed tomography (CT) is critical for various clinical applications, e.g., radiation treatment planning and also PET attenuation correction in MRI/PET scanner. However, CT exposes radiation during acquisition, which may cause side effects to patients. Compared to CT, magnetic resonance imaging (MRI) is much safer and does not involve radiations. Therefore, recently researchers are greatly motivated to estimate CT image from its corresponding MR image of the same subject for the case of radiation planning. In this paper, we propose a data-driven approach to address this challenging problem. Specifically, we train a fully convolutional network (FCN) to generate CT given the MR image. To better model the nonlinear mapping from MRI to CT and produce more realistic images, we propose to use the adversarial training strategy to train the FCN. Moreover, we propose an image-gradient-difference based loss function to alleviate the blurriness of the generated CT. We further apply Auto-Context Model (ACM) to implement a context-aware generative adversarial network. Experimental results show that our method is accurate and robust for predicting CT images from MR images, and also outperforms three state-of-the-art methods under comparison.

AB - Computed tomography (CT) is critical for various clinical applications, e.g., radiation treatment planning and also PET attenuation correction in MRI/PET scanner. However, CT exposes radiation during acquisition, which may cause side effects to patients. Compared to CT, magnetic resonance imaging (MRI) is much safer and does not involve radiations. Therefore, recently researchers are greatly motivated to estimate CT image from its corresponding MR image of the same subject for the case of radiation planning. In this paper, we propose a data-driven approach to address this challenging problem. Specifically, we train a fully convolutional network (FCN) to generate CT given the MR image. To better model the nonlinear mapping from MRI to CT and produce more realistic images, we propose to use the adversarial training strategy to train the FCN. Moreover, we propose an image-gradient-difference based loss function to alleviate the blurriness of the generated CT. We further apply Auto-Context Model (ACM) to implement a context-aware generative adversarial network. Experimental results show that our method is accurate and robust for predicting CT images from MR images, and also outperforms three state-of-the-art methods under comparison.

KW - Auto-context

KW - Deep learning

KW - GAN

KW - Generative models

KW - Image synthesis

UR - http://www.scopus.com/inward/record.url?scp=85029498722&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-66179-7_48

DO - 10.1007/978-3-319-66179-7_48

M3 - Conference contribution

AN - SCOPUS:85029498722

SN - 9783319661780

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 417

EP - 425

BT - Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings

A2 - Maier-Hein, Lena

A2 - Franz, Alfred

A2 - Jannin, Pierre

A2 - Duchesne, Simon

A2 - Descoteaux, Maxime

A2 - Collins, D. Louis

PB - Springer Verlag

T2 - 20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017

Y2 - 11 September 2017 through 13 September 2017

ER -

Medical image synthesis with context-aware generative adversarial networks

Abstract

Publication series

Other

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this