TY - JOUR
T1 - Multimodal Deep Fusion Network for Visibility Assessment with a Small Training Dataset
AU - Wang, Han
AU - Shen, Kecheng
AU - Yu, Peilun
AU - Shi, Quan
AU - Ko, Hanseok
N1 - Funding Information:
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 61872425 and Grant 61771265. The work of Hanseok Ko was supported by the National Research Foundation of Korea (NRF) under Grant 2019R1A2C2009480.
Publisher Copyright:
© 2013 IEEE.
PY - 2020
Y1 - 2020
N2 - Visibility is a measure of the transparency of the atmosphere, which is an important factor for road, air, and water transportation safety. Recently, features extracted from convolutional neural networks (CNNs) have obtained state-of-the-art results for the estimation of the visibility range for images of foggy weather. However, existing CNN-based approaches have only adopted visible images as observational data. Unlike these previous studies, in this paper, visible-infrared image pairs are used to estimate the visibility range. A novel multimodal deep fusion architecture based on a CNN is then proposed to learn the robust joint features of the two sensor modalities. Our network architecture is composed of two integrated residual network processing streams and one CNN stream, which are connected in parallel. In addition, we construct a visible-infrared multimodal dataset for various fog densities and label the visibility range. We then compare our proposed method with conventional deep-learning-based approaches and analyze the contributions of various observational and classical deep fusion models to the classification of the visibility range. The experimental results demonstrate that both accuracy and robustness can be strongly enhanced using the proposed method, especially for small training datasets.
AB - Visibility is a measure of the transparency of the atmosphere, which is an important factor for road, air, and water transportation safety. Recently, features extracted from convolutional neural networks (CNNs) have obtained state-of-the-art results for the estimation of the visibility range for images of foggy weather. However, existing CNN-based approaches have only adopted visible images as observational data. Unlike these previous studies, in this paper, visible-infrared image pairs are used to estimate the visibility range. A novel multimodal deep fusion architecture based on a CNN is then proposed to learn the robust joint features of the two sensor modalities. Our network architecture is composed of two integrated residual network processing streams and one CNN stream, which are connected in parallel. In addition, we construct a visible-infrared multimodal dataset for various fog densities and label the visibility range. We then compare our proposed method with conventional deep-learning-based approaches and analyze the contributions of various observational and classical deep fusion models to the classification of the visibility range. The experimental results demonstrate that both accuracy and robustness can be strongly enhanced using the proposed method, especially for small training datasets.
KW - Visibility range classification
KW - multimodal fusion network
KW - visible-infrared image pairs
UR - http://www.scopus.com/inward/record.url?scp=85098072459&partnerID=8YFLogxK
U2 - 10.1109/ACCESS.2020.3031283
DO - 10.1109/ACCESS.2020.3031283
M3 - Article
AN - SCOPUS:85098072459
SN - 2169-3536
VL - 8
SP - 217057
EP - 217067
JO - IEEE Access
JF - IEEE Access
M1 - 9225137
ER -