5701
Mathematics and Computer Science

SIGNAL ENHANCEMENT AND MANIPULATION USING A SIGNAL-SPECIFIC DEEP NETWORK (No. 1850)

Lead Researcher: Prof. Michal Irani

Summary

Optical and dimensional limitations of images can be partially overcome today using signal enhancement technologies which typically rely on supervised, deep-learning methods. However, as existing methods are restricted to specific training images and distortion types, they provide poor results for any practical case, unless huge set of data-pairs exist and all share the same exact distortion type. The current invention enables signal enhancement of low-resolution images, by exploiting deviations from expected internal patch recurrences detected within the image itself.  More specifically, the approach leverages cross-scale internal repetition of image-specific information, which is trained, at test time, on internal examples extracted solely from the test image. This first unsupervised conventional neural network (CNN)-based super-resolution approach allows for signal enhancement of real-world images acquired under suboptimal, unknown or image-specific conditions and has been shown to outperform state-of-the-art technologies.

Applications

This invention can be applied in a range of images and distortion types requiring enhancement, including:
·         Single video sequences
·         Old photos
·         Noisy images
·         Biological data
·         Medical images (e.g., fMRI)
·         Audio sequences

Advantages

·         Suitable for a wide range of images and data types
·         Applicable for images of any size and any aspect ratio
·         No pre-training requirement
·         Adaptable to images with known or unknown imagining conditions
·         Time-effective training
·         No requirement of side information/attributes
·         No requirement of additional images

Technology's Essence

This invention introduces zero-shot super-resolution (ZSSR), which exploits deep-learning methodologies, without requiring any prior image or training data. The internal recurrence of data within the single input image is exploited for on-line training of a small image-specific convolutional neural network (CNN) in examples extracted from the low-resolution image itself.  The method bears no recurrence patch-size limitation, enabling adaptation of the CNN to different settings for the same image, and requires no external information or prior training. This unsupervised CNN-based super-resolution method has been shown to substantially outperform externally trained state-of-the-art super-resolution of suboptimal, low-resolution images (Figure A).  In cases of ideal imaging conditions, the ZSSR output proved competitive to that of state-of-the-art supervised methods (Figure B).