# How to Make Large Language Models More Confident in Their Answers > Snowflake's patent details a method to make large language models more trustworthy when extracting information from documents by precisely adjusting their confidence scores using a mix of mathematical techniques. - **Patent:** US 12032919 - **Original title:** Post-calibration of large language model confidence scoring via combined techniques - **Owner:** Snowflake - **Granted:** 2024 - **Status:** Active - **Times cited:** 10 - **Field:** software, ai_ml, telecommunications, finance, legal_tech ## What it does This patent describes a system that improves how reliably large language models (LLMs) report their confidence when processing electronic text documents. First, it receives results from an ML model, including a confidence score (Claim 1). To address accuracy challenges, it then applies a special recalibration process to these confidence scores. This recalibration involves a combination of techniques: temperature scaling, selecting top-k probabilities from the softmax denominator, and polynomial regression (Claim 4). After adjusting the confidence scores, the system extracts specific pieces of information from the document and stores them in a database, each with its newly adjusted, more accurate confidence score (Claim 1). For example, if an LLM extracts a date from a contract, this method ensures the confidence score for that date is a more accurate reflection of the LLM's certainty. ## What it does NOT cover - Does not cover improving confidence scores for machine learning models not specifically configured for document processing and understanding of electronic text documents. - Does not cover calibration methods that do not combine temperature scaling, softmax denominator top-k probability selection, and polynomial regression. - Does not cover systems that only provide a confidence score without then extracting individual elements of information and storing them with adjusted scores in a database. - Does not cover ML models that process non-textual data, such as images or audio, for confidence scoring. - Does not cover methods that improve selective prediction accuracy without also improving calibration quality. ## The clever bit The novelty lies in the specific combination of three distinct calibration techniques—temperature scaling, softmax denominator top-k probability selection, and polynomial regression—applied together to post-calibrate LLM confidence scores. This combined approach aims to improve both the calibration quality (how well the confidence score matches actual accuracy) and the selective prediction accuracy (how well the model knows when it's likely to be right or wrong) for document processing tasks. ## Real-world examples 1. Automated contract analysis platforms 2. Financial document processing systems 3. Legal discovery tools 4. Healthcare record summarization 5. Data extraction services in cloud platforms ## Why it matters Accurate confidence scores are crucial for large language models, especially when they are used to extract sensitive or critical data from documents. If an LLM reports high confidence for an incorrect answer, it can lead to bad decisions. This patent helps ensure that when an LLM extracts information, like financial figures or legal clauses, its stated confidence level is a true reflection of its accuracy, which is vital for secure machine learning systems and data platforms like Snowflake. ## Frequently asked questions ### What does How to Make Large Language Models More Confident in Their Answers cover? Snowflake's patent details a method to make large language models more trustworthy when extracting information from documents by precisely adjusting their confidence scores using a mix of mathematical techniques. ### Who owns patent US 12032919? Snowflake owns this patent, granted in 2024. ### When does this patent expire? This patent is expected to expire on August 16, 2043, when the invention enters the public domain. ### What is patent US 12032919 cited by? This patent has been cited by 10 later patents that build on its ideas. ### What problem does this patent solve? Accurate confidence scores are crucial for large language models, especially when they are used to extract sensitive or critical data from documents. If an LLM reports high confidence for an incorrect answer, it can lead to bad decisions. This patent helps ensure that when an LLM extracts information, like financial figures or legal clauses, its stated confidence level is a true reflection of its accuracy, which is vital for secure machine learning systems and data platforms like Snowflake. ### What does this patent NOT cover? Does not cover improving confidence scores for machine learning models not specifically configured for document processing and understanding of electronic text documents. **Full plain-English explainer:** https://patentbrief.org/patent/us/12032919/post-calibration-of-large-language-model-confidence-scoring-via-combined-techniq **Original patent:** https://patents.google.com/patent/US12032919 --- _Source: PatentBrief — https://patentbrief.org. Patent facts are from public records; the plain-English explanation is PatentBrief's._ ## Related patents Semantically similar inventions in the PatentBrief corpus: - [How AI Models Can Be More Confident in Their Document Answers](https://patentbrief.org/patent/us/12236201/enhanced-machine-learning-model-accuracy-through-post-hoc-confidence-score-calib) — This patent describes a system for making machine learning models more accurate by adjusting how confident they are in the data they extract from documents, after the initial extraction. - [How AI Checks Documents for Policy Rules and Compliance](https://patentbrief.org/patent/us/12299406/large-language-model-artificial-intelligence-text-evaluation-system) — This patent describes a system where an AI model helps evaluate parts of a document against specific rules or policies given in plain language, then identifies problems or suggests fixes to ensure compliance. - [How a Computer System Checks and Improves AI Text](https://patentbrief.org/patent/us/11989527/computer-implemented-methods-for-the-automated-analysis-or-use-of-data-including-11989527) — This patent describes a method where a separate computer system, using structured knowledge, checks and corrects the output from a large language model to make it more accurate and consistent. - [How a Computer System Fact-Checks AI Language Models](https://patentbrief.org/patent/us/12073180/computer-implemented-methods-for-the-automated-analysis-or-use-of-data-including-12073180) — This patent describes a method for a separate computer system to fact-check and improve the output of a large language model by translating its text into a structured, machine-readable format. - [How a Computer System Improves Large Language Model Output with Structured Data](https://patentbrief.org/patent/us/12067362/computer-implemented-methods-for-the-automated-analysis-or-use-of-data-including-12067362) — This patent describes a method where a separate computer system analyzes and refines the initial output from a large language model (LLM) by translating it into a highly structured, machine-readable language to reduce bias and improve accuracy.