A Convolutional Neural Network for Motion-Based Multiframe Super-Resolution Using Fusion of Interpolated Frames

Date of Award


Degree Name

Ph.D. in Electrical Engineering


Department of Electrical and Computer Engineering


Advisor: Russell Hardie


We present two novel multiframe image super-resolution (SR) algorithms that employ convolutional neural networks (CNNs) to generate high-resolution (HR) images from mul- tiple low-resolution (LR) input frames. The first algorithm, named Fusion of Interpolated Frames Network (FIFNET), utilizes motion-based multiframe SR by fusing multiple input frames in a single CNN based on Random and Fixed shifts. The second algorithm, called the Exponential weighted Fusion of Interpolated Frames Network (EWF-FIFNET), presents two variations, Externally Exponential Weighted Fusion-FIFNET (EEWF-FIFNET) and Inter- nally Exponential Weighted Fusion-FIFNET (IEWF-FIFNET) based on affine motion. A custom layer called the Exponential Weighted Fusion (EWF) layer is developed to combine input frames using a technique inspired by the fusion interpolation frame SR framework within the IEWF-FIFNET model. The EWF-FIFNET network utilizes a modified Residual Channel Attention Network architecture with residual in residual (RIR) structures. The proposed algorithms are trained and tested using a realistic observation camera model that incorporates optical and sensor degradation. Affine motion is also incorporated to address a challenging degradation problem. The experimental results show that the proposed algorithms outperform the existing state-of-the-art methods using both simulated and real camera data. It is noteworthy that the real data is not artificially downsampled or degraded, making the proposed algorithms a promising solution for practical applications. This research contributes significantly to the field of multiframe image SR, particularly in motion-based and exponentially weighted fusion approaches using CNNs.


Multiframe super-resolution, convolutional neural network, fusion of interpolated frames, image restoration, subpixel registration

Rights Statement

Copyright © 2023, author.