Multimodal Deep Fusion Network for Visibility Assessment with a Small Training Dataset

Han Wang, Kecheng Shen, Peilun Yu, Quan Shi, Hanseok Ko

    Research output: Contribution to journalArticlepeer-review

    14 Citations (Scopus)

    Abstract

    Visibility is a measure of the transparency of the atmosphere, which is an important factor for road, air, and water transportation safety. Recently, features extracted from convolutional neural networks (CNNs) have obtained state-of-the-art results for the estimation of the visibility range for images of foggy weather. However, existing CNN-based approaches have only adopted visible images as observational data. Unlike these previous studies, in this paper, visible-infrared image pairs are used to estimate the visibility range. A novel multimodal deep fusion architecture based on a CNN is then proposed to learn the robust joint features of the two sensor modalities. Our network architecture is composed of two integrated residual network processing streams and one CNN stream, which are connected in parallel. In addition, we construct a visible-infrared multimodal dataset for various fog densities and label the visibility range. We then compare our proposed method with conventional deep-learning-based approaches and analyze the contributions of various observational and classical deep fusion models to the classification of the visibility range. The experimental results demonstrate that both accuracy and robustness can be strongly enhanced using the proposed method, especially for small training datasets.

    Original languageEnglish
    Article number9225137
    Pages (from-to)217057-217067
    Number of pages11
    JournalIEEE Access
    Volume8
    DOIs
    Publication statusPublished - 2020

    Bibliographical note

    Funding Information:
    This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 61872425 and Grant 61771265. The work of Hanseok Ko was supported by the National Research Foundation of Korea (NRF) under Grant 2019R1A2C2009480.

    Publisher Copyright:
    © 2013 IEEE.

    Keywords

    • Visibility range classification
    • multimodal fusion network
    • visible-infrared image pairs

    ASJC Scopus subject areas

    • General Computer Science
    • General Materials Science
    • General Engineering

    Fingerprint

    Dive into the research topics of 'Multimodal Deep Fusion Network for Visibility Assessment with a Small Training Dataset'. Together they form a unique fingerprint.

    Cite this