5701
Mathematics and Computer Science

Signal Enhancement and Manipulation Using a Signal-Specific Deep Network (No. T4-1850)

Lead Researcher: Prof. Michal Irani

Summary

Optical and dimensional limitations of images can be partially overcome today using signal enhancement technologies which typically rely on supervised, deep-learning methods. However, as existing methods are restricted to specific training images and distortion types, they provide poor results for any practical case, unless huge set of data-pairs exist and all share the same exact distortion type. The current invention enables signal enhancement of low-resolution images, by exploiting deviations from expected internal patch recurrences detected within the image itself. More specifically, the approach leverages cross-scale internal repetition of image-specific information, which is trained, at test time, on internal examples extracted solely from the test image. This first unsupervised conventional neural network (CNN)-based super-resolution approach allows for signal enhancement of real-world images acquired under suboptimal, unknown or image-specific conditions and has been shown to outperform state-of-the-art technologies.

Applications

This invention can be applied in a range of images and distortion types requiring enhancement, including:

  • Single video sequences
  • Old photos
  • Noisy images
  • Biological data
  • Medical images (e.g., fMRI)
  • Audio sequences
Advantages
  • Suitable for a wide range of images and data types
  • Applicable for images of any size and any aspect ratio
  • No pre-training requirement
  • Adaptable to images with known or unknown imagining conditions
  • Time-effective training
  • No requirement of side information/attributes
  • No requirement of additional images
Technology's Essence

This invention introduces zero-shot super-resolution (ZSSR), which exploits deep-learning methodologies, without requiring any prior image or training data. The internal recurrence of data within the single input image is exploited for on-line training of a small image-specific convolutional neural network (CNN) in examples extracted from the low-resolution image itself. The method bears no recurrence patch-size limitation, enabling adaptation of the CNN to different settings for the same image, and requires no external information or prior training. This unsupervised CNN-based super-resolution method has been shown to substantially outperform externally trained state-of-the-art super-resolution of suboptimal, low-resolution images (Figure A). In cases of ideal imaging conditions, the ZSSR output proved competitive to that of state-of-the-art supervised methods (Figure B).

Link to the project web-page.

 

(A)  (B) 

A) Super-resolution of a suboptimal low-resolution image using the ZSSR approach versus state-of-the-art (SotA) approaches. B) Super-resolution using ZSSR as compared to state-of-the-art methods, of a low-resolution image generated under ideal, supervised conditions.