How to Make Large Language Models More Confident in Their Answers
Snowflake's patent details a method to make large language models more trustworthy when extracting information from documents by precisely adjusting their confidence scores using a mix of mathematical techniques.
Original patent title: “Post-calibration of large language model confidence scoring via combined techniques”
Snowflake's patent details a method to make large language models more trustworthy when extracting information from documents by precisely adjusting their confidence scores using a mix of mathematical techniques. Granted to Snowflake in 2024 with 33 claims and 10 forward citations, and it is expected to expire in 2043.
Coverage
What does this patent actually cover?
This patent describes a system that improves how reliably large language models (LLMs) report their confidence when processing electronic text documents. First, it receives results from an ML model, including a confidence score (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 1). To address accuracy challenges, it then applies a special recalibration process to these confidence scores. This recalibration involves a combination of techniques: temperature scaling, selecting top-k probabilities from the softmax denominator, and polynomial regression (Claim 4). After adjusting the confidence scores, the system extracts specific pieces of information from the document and stores them in a database, each with its newly adjusted, more accurate confidence score (Claim 1). For example, if an LLM extracts a date from a contract, this method ensures the confidence score for that date is a more accurate reflection of the LLM's certainty.
The gap
What does this patent NOT cover?
- Does not cover improving confidence scores for machine learning models not specifically configured for document processing and understanding of electronic text documents.
- Does not cover calibration methods that do not combine temperature scaling, softmax denominator top-k probability selection, and polynomial regression.
- Does not cover systems that only provide a confidence score without then extracting individual elements of information and storing them with adjusted scores in a database.
- Does not cover ML models that process non-textual data, such as images or audio, for confidence scoring.
- Does not cover methods that improve selective prediction accuracy without also improving calibration quality.
These exclusions are unique to PatentBrief — derived from the actual claim language, not patent-office boilerplate.
Key facts
What made this novel
The noveltynoveltyThe requirement that an invention be different from anything publicly known before its priority date.Read more → lies in the specific combination of three distinct calibration techniques—temperature scaling, softmax denominator top-k probability selection, and polynomial regression—applied together to post-calibrate LLM confidence scores. This combined approach aims to improve both the calibration quality (how well the confidence score matches actual accuracy) and the selective prediction accuracy (how well the model knows when it's likely to be right or wrong) for document processing tasks.
The Patent Drawing
Schematic visualization of the patent's claim structure. Hand-drawn diagrams in progress for each landmark patent.
Where you've seen this
Real-world examples
Automated contract analysis platforms
Financial document processing systems
Legal discovery tools
Healthcare record summarization
Data extraction services in cloud platforms
Why it matters
The bigger picture
Accurate confidence scores are crucial for large language models, especially when they are used to extract sensitive or critical data from documents. If an LLM reports high confidence for an incorrect answer, it can lead to bad decisions. This patent helps ensure that when an LLM extracts information, like financial figures or legal clauses, its stated confidence level is a true reflection of its accuracy, which is vital for secure machine learning systems and data platforms like Snowflake.
Filed
August 16, 2023
Granted
July 9, 2024
Market context
Who's building on this
Companies in this space
Snowflake Inc., the assigneeassigneeThe entity that owns the patent — usually the inventor's employer or a company.Read more →, is actively building on this technology to enhance the reliability of its data processing and AI capabilities, particularly for customers leveraging large language models for data extraction and analysis within their secure data cloud. Other major cloud providers and AI platform companies are also investing heavily in improving the trustworthiness and explainability of LLM outputs.
Market impact
This patent contributes to the growing need for more reliable and trustworthy AI systems, particularly large language models. By providing a robust method for post-calibrating confidence scores, it enables more accurate automated data extraction, which can reduce manual review and improve the quality of data used in critical business processes. This directly impacts industries relying on automated document processing, making LLM-powered solutions more viable and reducing risks associated with AI errors.
Claim 1 — Plain English
What this patent covers
This patent describes a system that improves how reliably large language models (LLMs) report their confidence when processing electronic text documents. First, it receives results from an ML model, including a confidence score (Claim 1). To address accuracy challenges, it then applies a special recalibration process to these confidence scores. This recalibration involves a combination of techniques: temperature scaling, selecting top-k probabilities from the softmax denominator, and polynomial regression (Claim 4). After adjusting the confidence scores, the system extracts specific pieces of information from the document and stores them in a database, each with its newly adjusted, more accurate confidence score (Claim 1). For example, if an LLM extracts a date from a contract, this method ensures the confidence score for that date is a more accurate reflection of the LLM's certainty.
The clever bit
The novelty lies in the specific combination of three distinct calibration techniques—temperature scaling, softmax denominator top-k probability selection, and polynomial regression—applied together to post-calibrate LLM confidence scores. This combined approach aims to improve both the calibration quality (how well the confidence score matches actual accuracy) and the selective prediction accuracy (how well the model knows when it's likely to be right or wrong) for document processing tasks.
What it does not cover
- Does not cover improving confidence scores for machine learning models not specifically configured for document processing and understanding of electronic text documents.
- Does not cover calibration methods that do not combine temperature scaling, softmax denominator top-k probability selection, and polynomial regression.
- Does not cover systems that only provide a confidence score without then extracting individual elements of information and storing them with adjusted scores in a database.
- Does not cover ML models that process non-textual data, such as images or audio, for confidence scoring.
- Does not cover methods that improve selective prediction accuracy without also improving calibration quality.
Patent timeline
Application submitted to the patent office
Application published, typically 18 months after filing
Patent officially issued
Patent enters public domain
PatentBrief Score
Impact Score
Strong
Citation count
21/40
Early citations
Claim breadth
20/20
Very broad protection
Recency
20/20
Granted within 5 years
Assignee scale
0/20
Independent or smaller assigneeassigneeThe entity that owns the patent — usually the inventor's employer or a company.Read more →
PatentBrief Impact Score — based on citation count, claim breadth, recency, and assignee scale. Not a legal assessment.
Heuristic Value Estimate
What this patent might be worth
$123K – $393K
Midpoint $246K · 16.8 yr remaining · industry ×1.6
Heuristic only — blends forward/backward citation counts, claim scope, time remaining, litigation history, and CPC-derived industry baseline. Real valuations need a professional appraisal.
Claim text not yet imported for this patent
The original legal language
Original claims
33 claims as filed with the patent office.
Concepts involved
Citations
Patent lineage
Cite this patent
Szwabe, A. (2024). How to Make Large Language Models More Confident in Their Answers (U.S. Patent No. 12,032,919). U.S. Patent and Trademark Office. https://patentbrief.org/patent/us/12032919/post-calibration-of-large-language-model-confidence-scoring-via-combined-techniq
Auto-generated from the patent record. Double-check author order and the issue date against the official USPTO document before submitting.
Embed
Add this patent to your site
Drop this plain-English patent card into any blog post or article — free, no signup. It always links back to the full breakdown here.
<div data-patentlens-widget data-patent-number="US12032919"></div> <script src="https://patentbrief.org/embed.js" async></script>
Stay in the loop
Get a weekly digest of new patents.
One email per week. No spam. Unsubscribe anytime.
Keep exploring
Related patents you should know
US 4683195 · 1987
How to Make Billions of Copies of a DNA Segment
This patent describes the Polymerase Chain Reaction (PCR), a method to rapidly create many copies of a specific piece of DNA or RNA, enabling its detection and analysis.
Cetus Corp
US 8697359 · 2014
How to Edit Genes in Human Cells Using an Engineered CRISPR System
This patent describes an engineered CRISPR-Cas9 system for precisely cutting DNA in eukaryotic cells to change how genes work, opening the door for gene editing in complex organisms.
Massachusetts Institute of Technology
US 7657849 · 2010
How the iPhone's Slide-to-Unlock Gesture Works
Apple's 2010 patent describes unlocking a device by dragging a specific graphical image across the touchscreen along a predefined path, a gesture that became iconic with the original iPhone.
Apple Inc
US 4733665 · 1988
How Doctors Implant a Permanent Stent Using a Balloon
This patent describes the method for placing a permanent, expandable wire mesh tube inside a blood vessel or other body tube using a balloon-tipped catheter to widen it and keep it open.
Expandable Grafts Partnership
US 4965188 · 1990
How to Make Many Copies of a DNA Piece with Heat
This patent describes the Polymerase Chain Reaction (PCR) method, a technique to make millions of copies of a specific DNA segment using a heat-resistant enzyme and repeated temperature changes.
Cetus Corp
US 4235871 · 1980
How to Encapsulate Active Materials in Lipid Bubbles Efficiently
This patent describes a method for trapping biologically active substances inside tiny, multi-layered fat bubbles called liposomes, using a specific water-in-oil emulsion and gel-forming process to improve how much material gets captured.
Individual
Semantically similar
You might also find these interesting
US 12236201 · 2025 · Snowflake
How AI Models Can Be More Confident in Their Document Answers
US 12299406 · 2025 · Casetext
How AI Checks Documents for Policy Rules and Compliance
US 11989527 · 2024 · Unlikely Artificial Intelligence
How a Computer System Checks and Improves AI Text
US 12073180 · 2024 · Unlikely Artificial Intelligence
How a Computer System Fact-Checks AI Language Models
More to explore
More in Software & Internet
US 4405829 · 1983 · Massachusetts Institute of Technology
How RSA Public-Key Encryption Keeps Digital Messages Secret
US 6285999 · 2001 · Leland Stanford Junior University
How Websites Get Ranked by Importance
US 5960411 · 1999 · Amazon com Inc
How Amazon's One-Click Ordering Works for Online Purchases
US 7669123 · 2010 · Facebook Inc
Displaying Friends' Activities in a Social Network Feed
New to patents?
Common Questions
Frequently Asked Questions
What does How to Make Large Language Models More Confident in Their Answers cover?
Snowflake's patent details a method to make large language models more trustworthy when extracting information from documents by precisely adjusting their confidence scores using a mix of mathematical techniques.
Who owns patent US 12032919?
Snowflake owns this patent, granted in 2024.
When does this patent expire?
This patent is expected to expire on August 16, 2043, when the invention enters the public domain.
What is patent US 12032919 cited by?
This patent has been cited by 10 later patents that build on its ideas.
What problem does this patent solve?
Accurate confidence scores are crucial for large language models, especially when they are used to extract sensitive or critical data from documents. If an LLM reports high confidence for an incorrect answer, it can lead to bad decisions. This patent helps ensure that when an LLM extracts information, like financial figures or legal clauses, its stated confidence level is a true reflection of its accuracy, which is vital for secure machine learning systems and data platforms like Snowflake.
What does this patent NOT cover?
Does not cover improving confidence scores for machine learning models not specifically configured for document processing and understanding of electronic text documents.
Same assignee
More from Snowflake
Patent monitoring