How AI Transfers Visual Styles Between Images While Keeping Structure

This patent describes an AI method to generate a new image that combines the structure of one input image with the visual style of another, semantically related image, using a pre-trained Vision Transformer model.

What it covers

This method generates a new image by taking two inputs: a 'source structure image' and a 'target appearance image'. The goal is to create a 'third image' that keeps the original structure from the first image but adopts the visual look from the second. For example, if you provide a picture of a person (source structure) and a picture of a famous painting (target appearance), the system aims to redraw the person in the style of the painting. It achieves this by training a 'generator' (an Artificial Neural Network, or ANN) to minimize differences between the appearance of the generated image and the target appearance, and optionally, to minimize differences between the structure of the generated image and the source structure (Claim 1, Claim 3). A key component is using a pre-trained, fixed Vision Transformer (ViT) model as a 'semantic prior' to understand and correctly match objects between the two input images (Abstract, Claim 8). This allows the generator to be trained effectively with just a single pair of input images.

What it doesn't cover

• —Does not cover style transfer methods that do not use a Vision Transformer (ViT) model as a semantic prior for training the generator.
• —Does not cover image generation where the structure is also created or significantly altered, as it focuses on preserving the 'first structure' (Claim 1).
• —Does not cover methods that require large datasets for training the generator itself, as this patent emphasizes training with only a 'single structure/appearance image pair' (Abstract).
• —Does not cover transferring appearance without semantic awareness, meaning it aims to 'paint' semantically related objects, not just apply a general filter.
• —Does not cover methods that rely solely on adversarial training, as the patent specifies training 'without adversarial training' (Abstract).

The clever bit

The truly clever part is using a pre-trained and fixed Vision Transformer (ViT) model as an 'external semantic prior'. This allows a separate generator to be trained efficiently with only a single pair of input images, without needing extensive datasets or complex adversarial training, while still achieving high-quality, semantically accurate appearance transfer.

Why it matters

This technology is important because it enables high-quality, semantically aware image manipulation with less training data and computational effort. By leveraging pre-trained AI models, it simplifies the process of transferring complex visual styles onto new structures. This can accelerate content creation workflows and open new possibilities for personalized digital experiences across various industries.

Real-world examples

• 1.Virtual try-on applications for clothing or accessories
• 2.Digital fashion design tools
• 3.Content creation for advertising and media
• 4.Personalized avatar generation
• 5.Artistic style transfer applications
• 6.Special effects in film and video production