Learning-based filter selection scheme for depth image super resolution

Depth images that have the same spatial resolution as color images are required in many applications, such as multiview rendering and 3-D texture modeling. Since a depth sensor usually has poorer spatial resolution compared with a color image sensor, many depth image super-resolution methods have been investigated in the literature. With an assumption that no one super-resolution method can universally outperform the other methods, in this paper we introduce a learning-based selection scheme for different super-resolution methods. In our case study, three distinctive mean-type, max-type, and median-type filtering methods are selected as candidate methods. In addition, a new frequency-domain feature vector is designed to enhance the discriminability of the methods. Given the candidate methods and feature vectors, a classifier is trained such that the best method can be selected for each depth pixel. The effectiveness of the proposed scheme is first demonstrated using the synthetic data set. The noise-free and noisy low-resolution depth images are constructed, and the quantitative performance evaluation is performed by measuring the difference between the ground-truth high-resolution depth images and the resultant depth images. The proposed algorithm is then applied to real color and time-of-flight depth cameras. The experimental results demonstrate that the proposed algorithm outperforms the conventional algorithms both quantitatively and qualitatively.