Image super-resolution with Octave convolution for mobile phones

Abstract

Image super-resolution (SR) reconstruction techniques have been widely applied to different fields from daily practice to aviation activities, such as video surveillance, medical diagnosis, and remote sensing applications. The enhancement of deep learning and computer hardware can both advance current image SR reconstruction techniques by constructing a broader and/or deeper network. But the proposed networks require higher computational costs due to overloaded parameters, so it is difficult to directly apply to resource-constrained devices, such as mobile phones. Hence, this study is motivated to develop a lightweight deep learning network for SR reconstruction in mobile phones.

In this context, this study for the first time deploys the Fast Super-Resolution Convolutional Neural Networks (FSRCNN) adapted by octave convolution to construct a new network of FSRCNN\_Octave which can significantly overcome the deficiencies of overloaded parameters in the network, because the octave convolution can reduce the information redundancy of low-frequency features and effectively decrease the number of parameters and computational costs. The FSRCNN\_Octave network is trained with a real-world dataset which includes high-resolution (HR) images and low-resolution (LR) images. A total of ~2,000 HR images are directly captured by digital single-lens reflex cameras; the corresponding LR images are generated using DownSampleGAN (DSGAN). The DSGAN is trained with the image data collected by mobile phones in different scenes and environments covering face and text, indoor and outdoor landscape, and architecture in different illumination conditions. The experimental results show that the FSRCNN\_Octave network presents a higher PSNR and lower floating operation points (FLOPs). Hence, the FSRCNN\_Octave network can efficiently improve the performance of SR reconstruction in mobile phones. Moreover, this study integrates the FSRCNN\_Octave network into mobile phones using Snapdragon Neural Processing Engine (SNPE) to evaluate its practical applicability. The good performances of the FSRCNN\_Octave network are exhibited during the photo acquisition with elevated image quality and faster image processing. This study highlights that the lightweight deep learning network can significantly enhance the image SR reconstruction in mobile phones.\\