PEPSI++: Fast and Lightweight Network for Image Inpainting

Yong Goo Shin, Min Cheol Sagong, Yoon Jae Yeo, Seung Wook Kim, Sung Jea Ko

Research output: Contribution to journalArticlepeer-review

51 Citations (Scopus)

Abstract

Among the various generative adversarial network (GAN)-based image inpainting methods, a coarse-to-fine network with a contextual attention module (CAM) has shown remarkable performance. However, due to two stacked generative networks, the coarse-to-fine network needs numerous computational resources, such as convolution operations and network parameters, which result in low speed. To address this problem, we propose a novel network architecture called parallel extended-decoder path for semantic inpainting (PEPSI) network, which aims at reducing the hardware costs and improving the inpainting performance. PEPSI consists of a single shared encoding network and parallel decoding networks called coarse and inpainting paths. The coarse path produces a preliminary inpainting result to train the encoding network for the prediction of features for the CAM. Simultaneously, the inpainting path generates higher inpainting quality using the refined features reconstructed via the CAM. In addition, we propose Diet-PEPSI that significantly reduces the network parameters while maintaining the performance. In Diet-PEPSI, to capture the global contextual information with low hardware costs, we propose novel rate-adaptive dilated convolutional layers that employ the common weights but produce dynamic features depending on the given dilation rates. Extensive experiments comparing the performance with state-of-the-art image inpainting methods demonstrate that both PEPSI and Diet-PEPSI improve the qualitative scores, i.e., the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), as well as significantly reduce hardware costs, such as computational time and the number of network parameters.

Original languageEnglish
Article number9042886
Pages (from-to)252-265
Number of pages14
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume32
Issue number1
DOIs
Publication statusPublished - 2021 Jan

Bibliographical note

Funding Information:
Manuscript received May 21, 2019; revised October 8, 2019 and December 11, 2019; accepted March 2, 2020. Date of publication March 19, 2020; date of current version January 5, 2021. This work was supported by the Institute for Information and communications Technology Promotion (IITP) grant funded by the Korea Government under Grant 2019-0-00268, Development of SW technology for recognition, judgment and path control algorithm verification simulation and dataset generation. (Corresponding author: Sung-Jea Ko.) The authors are with the School of Electrical Engineering Department, Korea University, Seoul 136-713, South Korea (e-mail: ygshin@dali.korea.ac.kr; mcsagong@dali.korea.ac.kr; yjyeo@dali.korea. ac.kr; swkim@dali.korea.ac.kr; sjko@korea.ac.kr).

Publisher Copyright:
© 2012 IEEE.

Keywords

  • Deep learning
  • generative adversarial network (GAN)
  • image inpainting

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'PEPSI++: Fast and Lightweight Network for Image Inpainting'. Together they form a unique fingerprint.

Cite this