Sound-Guided Semantic Image Manipulation

Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chanyoung Kim, Jinkyu Kim, Sangpil Kim

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    26 Citations (Scopus)

    Abstract

    The recent success of the generative model shows that leveraging the multi-modal embedding space can manipu-late an image using text information. However, manipulating an image with other sources rather than text, such as sound, is not easy due to the dynamic characteristics of the sources. Especially, sound can convey vivid emotions and dynamic expressions of the real world. Here, we propose a framework that directly encodes sound into the multi-modal (image-text) embedding space and manipulates an image from the space. Our audio encoder is trained to pro-duce a latent representation from an audio input, which is forced to be aligned with image and text representations in the multi-modal embedding space. We use a direct latent op-timization method based on aligned embeddings for sound-guided image manipulation. We also show that our method can mix different modalities, i.e., text and audio, which en-rich the variety of the image modification. The experiments on zero-shot audio classification and semantic-level image classification show that our proposed model outperforms other text and sound-guided state-of-the-art methods.

    Original languageEnglish
    Title of host publicationProceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
    PublisherIEEE Computer Society
    Pages3367-3376
    Number of pages10
    ISBN (Electronic)9781665469463
    DOIs
    Publication statusPublished - 2022
    Event2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 - New Orleans, United States
    Duration: 2022 Jun 192022 Jun 24

    Publication series

    NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
    Volume2022-June
    ISSN (Print)1063-6919

    Conference

    Conference2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
    Country/TerritoryUnited States
    CityNew Orleans
    Period22/6/1922/6/24

    Bibliographical note

    Publisher Copyright:
    © 2022 IEEE.

    Keywords

    • Image and video synthesis and generation
    • Self-& semi-& meta- & unsupervised learning

    ASJC Scopus subject areas

    • Software
    • Computer Vision and Pattern Recognition

    Fingerprint

    Dive into the research topics of 'Sound-Guided Semantic Image Manipulation'. Together they form a unique fingerprint.

    Cite this