Coloring with limited data: Few-shot colorization via memory augmented networks

Seungjoo Yoo; Hyojin Bahng; Sunghyo Chung; Junsoo Lee; Jaehyuk Chang; Jaegul Choo

doi:10.1109/CVPR.2019.01154

Coloring with limited data: Few-shot colorization via memory augmented networks

Seungjoo Yoo, Hyojin Bahng, Sunghyo Chung, Junsoo Lee, Jaehyuk Chang, Jaegul Choo

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

116 Citations (Scopus)

Abstract

Despite recent advancements in deep learning-based automatic colorization, they are still limited when it comes to few-shot learning. Existing models require a significant amount of training data. To tackle this issue, we present a novel memory-augmented colorization model MemoPainter that can produce high-quality colorization with limited data. In particular, our model is able to capture rare instances and successfully colorize them. Also, we propose a novel threshold triplet loss that enables unsupervised training of memory networks without the need for class labels. Experiments show that our model has superior quality in both few-shot and one-shot colorization tasks.

Original language	English
Title of host publication	Proceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019
Publisher	IEEE Computer Society
Pages	11275-11284
Number of pages	10
ISBN (Electronic)	9781728132938
DOIs	https://doi.org/10.1109/CVPR.2019.01154
Publication status	Published - 2019 Jun
Event	32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019 - Long Beach, United States Duration: 2019 Jun 16 → 2019 Jun 20

Publication series

Name	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume	2019-June
ISSN (Print)	1063-6919

Conference

Conference	32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019
Country/Territory	United States
City	Long Beach
Period	19/6/16 → 19/6/20

Bibliographical note

Funding Information:
This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF2016R1C1B2015924).

Funding Information:
Acknowledgements. This work was partially supported bytheNationalResearchFoundationofKorea (NRF) grantfundedbytheKorean government(MSIP)(No. NRF2016R1C1B2015924). We thankall researchers at NAVER WEBTOONCorp.,especiallySungminKang. Jaegul Choo is the corresponding author.

Publisher Copyright:
© 2019 IEEE.

Keywords

Computational Photography
Computer Vision Theory
Deep Learning
Image and Video Synthesis
Vi
Vision + Graphics

ASJC Scopus subject areas

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/CVPR.2019.01154

Cite this

Yoo, S., Bahng, H., Chung, S., Lee, J., Chang, J., & Choo, J. (2019). Coloring with limited data: Few-shot colorization via memory augmented networks. In Proceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019 (pp. 11275-11284). Article 8953467 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2019-June). IEEE Computer Society. https://doi.org/10.1109/CVPR.2019.01154

Coloring with limited data: Few-shot colorization via memory augmented networks. / Yoo, Seungjoo; Bahng, Hyojin; Chung, Sunghyo et al.
Proceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019. IEEE Computer Society, 2019. p. 11275-11284 8953467 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2019-June).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yoo, S, Bahng, H, Chung, S, Lee, J, Chang, J & Choo, J 2019, Coloring with limited data: Few-shot colorization via memory augmented networks. in Proceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019., 8953467, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2019-June, IEEE Computer Society, pp. 11275-11284, 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, United States, 19/6/16. https://doi.org/10.1109/CVPR.2019.01154

Yoo S, Bahng H, Chung S, Lee J, Chang J, Choo J. Coloring with limited data: Few-shot colorization via memory augmented networks. In Proceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019. IEEE Computer Society. 2019. p. 11275-11284. 8953467. (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition). doi: 10.1109/CVPR.2019.01154

Yoo, Seungjoo ; Bahng, Hyojin ; Chung, Sunghyo et al. / Coloring with limited data : Few-shot colorization via memory augmented networks. Proceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019. IEEE Computer Society, 2019. pp. 11275-11284 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition).

@inproceedings{2f4565a4cb154022988371b621b57212,

title = "Coloring with limited data: Few-shot colorization via memory augmented networks",

abstract = "Despite recent advancements in deep learning-based automatic colorization, they are still limited when it comes to few-shot learning. Existing models require a significant amount of training data. To tackle this issue, we present a novel memory-augmented colorization model MemoPainter that can produce high-quality colorization with limited data. In particular, our model is able to capture rare instances and successfully colorize them. Also, we propose a novel threshold triplet loss that enables unsupervised training of memory networks without the need for class labels. Experiments show that our model has superior quality in both few-shot and one-shot colorization tasks.",

keywords = "Computational Photography, Computer Vision Theory, Deep Learning, Image and Video Synthesis, Vi, Vision + Graphics",

author = "Seungjoo Yoo and Hyojin Bahng and Sunghyo Chung and Junsoo Lee and Jaehyuk Chang and Jaegul Choo",

note = "Funding Information: This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF2016R1C1B2015924). Funding Information: Acknowledgements. This work was partially supported bytheNationalResearchFoundationofKorea (NRF) grantfundedbytheKorean government(MSIP)(No. NRF2016R1C1B2015924). We thankall researchers at NAVER WEBTOONCorp.,especiallySungminKang. Jaegul Choo is the corresponding author. Publisher Copyright: {\textcopyright} 2019 IEEE.; 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019 ; Conference date: 16-06-2019 Through 20-06-2019",

year = "2019",

month = jun,

doi = "10.1109/CVPR.2019.01154",

language = "English",

series = "Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition",

publisher = "IEEE Computer Society",

pages = "11275--11284",

booktitle = "Proceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019",

}

TY - GEN

T1 - Coloring with limited data

T2 - 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019

AU - Yoo, Seungjoo

AU - Bahng, Hyojin

AU - Chung, Sunghyo

AU - Lee, Junsoo

AU - Chang, Jaehyuk

AU - Choo, Jaegul

N1 - Funding Information: This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF2016R1C1B2015924). Funding Information: Acknowledgements. This work was partially supported bytheNationalResearchFoundationofKorea (NRF) grantfundedbytheKorean government(MSIP)(No. NRF2016R1C1B2015924). We thankall researchers at NAVER WEBTOONCorp.,especiallySungminKang. Jaegul Choo is the corresponding author. Publisher Copyright: © 2019 IEEE.

PY - 2019/6

Y1 - 2019/6

N2 - Despite recent advancements in deep learning-based automatic colorization, they are still limited when it comes to few-shot learning. Existing models require a significant amount of training data. To tackle this issue, we present a novel memory-augmented colorization model MemoPainter that can produce high-quality colorization with limited data. In particular, our model is able to capture rare instances and successfully colorize them. Also, we propose a novel threshold triplet loss that enables unsupervised training of memory networks without the need for class labels. Experiments show that our model has superior quality in both few-shot and one-shot colorization tasks.

AB - Despite recent advancements in deep learning-based automatic colorization, they are still limited when it comes to few-shot learning. Existing models require a significant amount of training data. To tackle this issue, we present a novel memory-augmented colorization model MemoPainter that can produce high-quality colorization with limited data. In particular, our model is able to capture rare instances and successfully colorize them. Also, we propose a novel threshold triplet loss that enables unsupervised training of memory networks without the need for class labels. Experiments show that our model has superior quality in both few-shot and one-shot colorization tasks.

KW - Computational Photography

KW - Computer Vision Theory

KW - Deep Learning

KW - Image and Video Synthesis

KW - Vi

KW - Vision + Graphics

UR - http://www.scopus.com/inward/record.url?scp=85078777565&partnerID=8YFLogxK

U2 - 10.1109/CVPR.2019.01154

DO - 10.1109/CVPR.2019.01154

M3 - Conference contribution

AN - SCOPUS:85078777565

T3 - Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

SP - 11275

EP - 11284

BT - Proceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019

PB - IEEE Computer Society

Y2 - 16 June 2019 through 20 June 2019

ER -

Coloring with limited data: Few-shot colorization via memory augmented networks

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this