Signal Enhancement and Manipulation Using a Signal-Specific Deep Network (No. T4-1850)
Lead Researcher: Prof. Michal Irani
Optical and dimensional limitations of images can be partially overcome today using signal enhancement technologies which typically rely on supervised, deep-learning methods. However, as existing methods are restricted to specific training images and distortion types, they provide poor results for any practical case, unless huge set of data-pairs exist and all share the same exact distortion type. The current invention enables signal enhancement of low-resolution images, by exploiting deviations from expected internal patch recurrences detected within the image itself. More specifically, the approach leverages cross-scale internal repetition of image-specific information, which is trained, at test time, on internal examples extracted solely from the test image. This first unsupervised conventional neural network (CNN)-based super-resolution approach allows for signal enhancement of real-world images acquired under suboptimal, unknown or image-specific conditions and has been shown to outperform state-of-the-art technologies.
This invention can be applied in a range of images and distortion types requiring enhancement, including:
- Single video sequences
- Old photos
- Noisy images
- Biological data
- Medical images (e.g., fMRI)
- Audio sequences
- Suitable for a wide range of images and data types
- Applicable for images of any size and any aspect ratio
- No pre-training requirement
- Adaptable to images with known or unknown imagining conditions
- Time-effective training
- No requirement of side information/attributes
- No requirement of additional images
This invention introduces zero-shot super-resolution (ZSSR), which exploits deep-learning methodologies, without requiring any prior image or training data. The internal recurrence of data within the single input image is exploited for on-line training of a small image-specific convolutional neural network (CNN) in examples extracted from the low-resolution image itself. The method bears no recurrence patch-size limitation, enabling adaptation of the CNN to different settings for the same image, and requires no external information or prior training. This unsupervised CNN-based super-resolution method has been shown to substantially outperform externally trained state-of-the-art super-resolution of suboptimal, low-resolution images (Figure A). In cases of ideal imaging conditions, the ZSSR output proved competitive to that of state-of-the-art supervised methods (Figure B).
A) Super-resolution of a suboptimal low-resolution image using the ZSSR approach versus state-of-the-art (SotA) approaches. B) Super-resolution using ZSSR as compared to state-of-the-art methods, of a low-resolution image generated under ideal, supervised conditions.