Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks

Author(s)

Hamed Elwarfalli (0009-0008-0658-3549), University of DaytonFollow
Dylan Flaute (0000-0001-5601-459X), University of DaytonFollow
Russell C. Hardie (0000-0002-1216-3865), University of DaytonFollow

Document Type

Article

Publication Date

1-2024

Publication Source

Sensors

Abstract

Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.

ISBN/ISSN

1424-8220

Document Version

Published Version

Comments

This open-access article is provided for download in compliance with the publisher’s policy on self-archiving. To view the version of record, use the DOI: https://doi.org/10.3390/s24010296

Publisher

MDPI

Volume

24

Peer Reviewed

yes

Issue

1

eCommons Citation

Elwarfalli, Hamed; Flaute, Dylan; and Hardie, Russell C., "Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks" (2024). Electrical and Computer Engineering Faculty Publications. 459.
https://ecommons.udayton.edu/ece_fac_pub/459