Medical Image Synthesis with Deep Convolutional Adversarial Networks

Dong Nie, Roger Trullo, Jun Lian, Li Wang, Caroline Petitjean, Su Ruan, Qian Wang, Dinggang Shen

Research output: Contribution to journalArticlepeer-review

409 Citations (Scopus)


Medical imaging plays a critical role in various clinical applications. However, due to multiple considerations such as cost and radiation dose, the acquisition of certain image modalities may be limited. Thus, medical image synthesis can be of great benefit by estimating a desired imaging modality without incurring an actual scan. In this paper, we propose a generative adversarial approach to address this challenging problem. Specifically, we train a fully convolutional network (FCN) to generate a target image given a source image. To better model a nonlinear mapping from source to target and to produce more realistic target images, we propose to use the adversarial learning strategy to better model the FCN. Moreover, the FCN is designed to incorporate an image-gradient-difference-based loss function to avoid generating blurry target images. Long-term residual unit is also explored to help the training of the network. We further apply Auto-Context Model to implement a context-aware deep convolutional adversarial network. Experimental results show that our method is accurate and robust for synthesizing target images from the corresponding source images. In particular, we evaluate our method on three datasets, to address the tasks of generating CT from MRI and generating 7T MRI from 3T MRI images. Our method outperforms the state-of-the-art methods under comparison in all datasets and tasks.

Original languageEnglish
Article number8310638
Pages (from-to)2720-2730
Number of pages11
JournalIEEE Transactions on Biomedical Engineering
Issue number12
Publication statusPublished - 2018 Dec

Bibliographical note

Funding Information:
Manuscript received July 31, 2017; revised December 5, 2017 and February 2, 2018; accepted February 25, 2018. Date of publication March 9, 2018; date of current version November 20, 2018. This work was supported in part by the National Institutes of Health under Grant CA206100 for D. Shen; in part by the National Key Research and Development Program of China under Grant 2017YFC0107600; in part by the National Natural Science Foundation of China under Grant 61473190, 81471733; and in part by the Science and Technology Commission of Shanghai Municipality under Grants 16511101100, 16410722400 for Q. Wang. (Dong Nie and Roger Trullo contributed equally to this work.) (Corresponding author: Qian Wang and Dinggang Shen.) D. Nie is with the Department of Computer Science, Department of Radiology and BRIC, UNC-Chapel Hill, Chapel Hill, NC, 27510 USA (e-mail:

Publisher Copyright:
© 2018 IEEE.


  • Adversarial learning
  • auto-context model
  • deep learning
  • image synthesis
  • residual learning

ASJC Scopus subject areas

  • Biomedical Engineering


Dive into the research topics of 'Medical Image Synthesis with Deep Convolutional Adversarial Networks'. Together they form a unique fingerprint.

Cite this