Making Neural Networks Faster by Skipping Unnecessary Calculations
A method to speed up AI training by keeping data sparse, meaning it ignores zeros to save memory and processing power during both forward and backward passes.
Original patent title: “System and method for bank-balanced sparse activation and joint-activation-weight-sparse training of neural networks”
A method to speed up AI training by keeping data sparse, meaning it ignores zeros to save memory and processing power during both forward and backward passes. Granted to Moffett International Co Ltd Hong Kong in 2022 with 20 claims and 1 forward citation.
Key facts
Coverage
What does this patent actually cover?
This patent describes a way to train neural networks more efficiently by using 'sparse' data structures. Instead of calculating every single value in a neural network, which involves many zeros that don't change the outcome, the system keeps tensors (multidimensional arrays of numbers) sparse by removing or ignoring these zeros. During the forward pass, it generates a dense output but immediately sparsifies it. During the backward pass, it uses these sparse tensors to calculate gradients, ensuring that the training process only focuses on the most significant data points. It specifically uses a 'top-K' activation function to decide which values are important enough to keep, effectively pruning the network during the training process itself.
The gap
What does this patent NOT cover?
- Does not cover training methods that do not use sparse tensors or sparsity-based pruning.
- Does not cover standard dense neural network training where all weights are updated equally.
- Does not cover hardware-specific implementations that do not utilize the described bank-based top-K selection logic.
- Does not cover non-neural network machine learning models like linear regression or decision trees.
These exclusions are unique to PatentBrief — derived from the actual claim language, not patent-office boilerplate.
What made this novel
The patent cleverly applies sparsity not just to the forward pass, but also to the backward pass (gradient calculation), using a bank-based top-K selection to maintain efficiency across the entire training loop.
Schematic visualization of the patent's claim structure. Hand-drawn diagrams in progress for each landmark patent.
Where you've seen this
Real-world examples
Efficient training of deep learning models on edge AI hardware
Pruning-aware neural network training frameworks
Optimized GPU-based model training pipelines
Why it matters
The bigger picture
Training large AI models is incredibly expensive and energy-intensive. By reducing the number of active parameters through sparsity, this method allows for faster training cycles and lower memory requirements, which is essential for deploying large models on hardware with limited resources like mobile devices or edge servers.
Filed
August 16, 2021
Granted
August 30, 2022
Market context
Who's building on this
Companies in this space
The technology is highly relevant to companies developing AI accelerators and specialized hardware, such as NVIDIA, Google (with their TPU architecture), and various edge AI startups focused on model compression and efficient inference.
Market impact
This patent contributes to the broader industry shift toward 'sparse AI,' which is becoming critical as model sizes grow beyond the capacity of traditional memory architectures. It enables more sustainable AI development by reducing the computational overhead required to achieve high-performance results.
Claim 1 — Plain English
What this patent covers
This patent describes a way to train neural networks more efficiently by using 'sparse' data structures. Instead of calculating every single value in a neural network, which involves many zeros that don't change the outcome, the system keeps tensors (multidimensional arrays of numbers) sparse by removing or ignoring these zeros. During the forward pass, it generates a dense output but immediately sparsifies it. During the backward pass, it uses these sparse tensors to calculate gradients, ensuring that the training process only focuses on the most significant data points. It specifically uses a 'top-K' activation function to decide which values are important enough to keep, effectively pruning the network during the training process itself.
The clever bit
The patent cleverly applies sparsity not just to the forward pass, but also to the backward pass (gradient calculation), using a bank-based top-K selection to maintain efficiency across the entire training loop.
What it does not cover
- Does not cover training methods that do not use sparse tensors or sparsity-based pruning.
- Does not cover standard dense neural network training where all weights are updated equally.
- Does not cover hardware-specific implementations that do not utilize the described bank-based top-K selection logic.
- Does not cover non-neural network machine learning models like linear regression or decision trees.
Patent timeline
Application submitted to the patent office
Application published, typically 18 months after filing
Patent officially issued
PatentBrief Score
Impact Score
Early stage
Citation count
6/40
Early citations
Claim breadth
13/20
Broad claimsclaimsThe numbered statements at the end of a patent that legally define what the inventor owns.Read more →
Recency
20/20
Granted within 5 years
Assignee scale
0/20
Independent or smaller assigneeassigneeThe entity that owns the patent — usually the inventor's employer or a company.Read more →
PatentBrief Impact Score — based on citation count, claim breadth, recency, and assignee scale. Not a legal assessment.
Heuristic Value Estimate
What this patent might be worth
$62K – $200K
Midpoint $125K · 15.2 yr remaining · industry ×1.6
Heuristic only — blends forward/backward citation counts, claim scope, time remaining, litigation history, and CPC-derived industry baseline. Real valuations need a professional appraisal.
The original legal language
Original claims
20 claims as filed with the patent office.
Concepts involved
Citations
Patent lineage
Cite this patent
Yan, E. (2022). Making Neural Networks Faster by Skipping Unnecessary Calculations (U.S. Patent No. 11,429,864). U.S. Patent and Trademark Office. https://patentbrief.org/patent/us/11429864/dlss-deep-learning-super-sampling
Auto-generated from the patent record. Double-check author order and the issue date against the official USPTO document before submitting.
Embed
Add this patent to your site
Drop this plain-English patent card into any blog post or article — free, no signup. It always links back to the full breakdown here.
<div data-patentlens-widget data-patent-number="US11429864"></div> <script src="https://patentbrief.org/embed.js" async></script>
Stay in the loop
Get a weekly digest of new patents.
One email per week. No spam. Unsubscribe anytime.
Keep exploring
Related patents you should know
US 4683195 · 1987
How to Make Billions of Copies of a DNA Segment
This patent describes the Polymerase Chain Reaction (PCR), a method to rapidly create many copies of a specific piece of DNA or RNA, enabling its detection and analysis.
Cetus Corp
US 8697359 · 2014
How to Edit Genes in Human Cells Using an Engineered CRISPR System
This patent describes an engineered CRISPR-Cas9 system for precisely cutting DNA in eukaryotic cells to change how genes work, opening the door for gene editing in complex organisms.
Massachusetts Institute of Technology
US 7657849 · 2010
How the iPhone's Slide-to-Unlock Gesture Works
Apple's 2010 patent describes unlocking a device by dragging a specific graphical image across the touchscreen along a predefined path, a gesture that became iconic with the original iPhone.
Apple Inc
US 4733665 · 1988
How Doctors Implant a Permanent Stent Using a Balloon
This patent describes the method for placing a permanent, expandable wire mesh tube inside a blood vessel or other body tube using a balloon-tipped catheter to widen it and keep it open.
Expandable Grafts Partnership
US 4965188 · 1990
How to Make Many Copies of a DNA Piece with Heat
This patent describes the Polymerase Chain Reaction (PCR) method, a technique to make millions of copies of a specific DNA segment using a heat-resistant enzyme and repeated temperature changes.
Cetus Corp
US 4235871 · 1980
How to Encapsulate Active Materials in Lipid Bubbles Efficiently
This patent describes a method for trapping biologically active substances inside tiny, multi-layered fat bubbles called liposomes, using a specific water-in-oil emulsion and gel-forming process to improve how much material gets captured.
Individual
More to explore
More in AI & Machine Learning
US 10452978 · 2019 · Google LLC
How AI Models Understand Language Using 'Attention'
US 6523026 · 2003 · Huntsman International LLC
How Computers Find Hidden Connections Between Different Fields of Knowledge
US 11615208 · 2023 · Capital One Services LLC
How Cloud Systems Automatically Create and Train AI Data Models
US 10402750 · 2019 · Facebook Inc
How Facebook Uses Deep Learning to Predict What You Might Like
New to patents?
Common Questions
Frequently Asked Questions
What does Making Neural Networks Faster by Skipping Unnecessary Calculations cover?
A method to speed up AI training by keeping data sparse, meaning it ignores zeros to save memory and processing power during both forward and backward passes.
Who owns patent US 11429864?
Moffett International Co Ltd Hong Kong owns this patent, granted in 2022.
When does this patent expire?
This patent is expected to expire on August 30, 2042, when the invention enters the public domain.
What is patent US 11429864 cited by?
This patent has been cited by 1 later patents that build on its ideas.
What problem does this patent solve?
Training large AI models is incredibly expensive and energy-intensive. By reducing the number of active parameters through sparsity, this method allows for faster training cycles and lower memory requirements, which is essential for deploying large models on hardware with limited resources like mobile devices or edge servers.
What does this patent NOT cover?
Does not cover training methods that do not use sparse tensors or sparsity-based pruning.
Patent monitoring