# How AI Models Can Be More Confident in Their Document Answers > This patent describes a system for making machine learning models more accurate by adjusting how confident they are in the data they extract from documents, after the initial extraction. - **Patent:** US 12236201 - **Original title:** Enhanced machine learning model accuracy through post-hoc confidence score calibration - **Owner:** Snowflake - **Granted:** 2025 - **Status:** Active - **Times cited:** 0 - **Field:** software, ai_ml, telecommunications, finance, ecommerce ## What it does A machine learning (ML) system first receives results from an AI model, which include a confidence score and electronic documents (Claim 1). It then performs "document understanding" to pull out specific "data points" from these documents, like a date or an amount (Claim 1). The system connects the initial confidence score to these extracted data points. Next, it uses a "post-hoc calibration (PHC) solution set" (Claim 1) to adjust these confidence scores. This calibration step involves methods like "temperature scaling" or "polynomial regression" (Claim 3) to make sure the confidence score truly matches how accurate the AI's prediction is (Claim 1). After adjusting, the system extracts the final pieces of information, like a specific value, and outputs these values along with their newly adjusted, more reliable confidence scores (Claim 1). For example, if an AI extracts a shipping address from a contract, this patent helps ensure the AI's reported confidence level for that address is truly accurate. ## What it does NOT cover - Does not cover improving ML model accuracy during the initial training or prediction phase, only after results are generated. - Does not cover ML models that do not involve "document understanding" or extracting data points from electronic documents. - Does not cover methods of improving ML model performance that do not involve calibrating confidence scores. - Does not cover calibration methods that are not "post-hoc" or do not use techniques like temperature scaling, softmax top-k, or polynomial regression. - Does not cover general model improvements that are not specifically related to aligning confidence scores with prediction accuracy. ## The clever bit The clever bit is recognizing that an AI's initial confidence score might not truly reflect its accuracy, and then providing specific methods to recalibrate that confidence after the AI has already made its predictions and extracted data from documents. This ensures the reported confidence is a more honest reflection of how likely the extracted information is to be correct. ## Real-world examples 1. Snowflake Cortex (Snowflake's AI services) 2. AI-powered invoice processing systems 3. Automated contract review software 4. Intelligent document processing platforms 5. Large language models used for data extraction ## Why it matters Many AI models, especially large language models, can be overly confident in their predictions, even when they are wrong. This patent matters because it provides a structured way to make an AI's stated confidence level more truthful. This is crucial for applications where relying on AI-extracted information, like financial figures or legal clauses from documents, requires high accuracy and trustworthiness. By ensuring the confidence score truly reflects the likelihood of correctness, businesses can better decide when to trust AI outputs versus when human review is needed. ## Frequently asked questions ### What does How AI Models Can Be More Confident in Their Document Answers cover? This patent describes a system for making machine learning models more accurate by adjusting how confident they are in the data they extract from documents, after the initial extraction. ### Who owns patent US 12236201? Snowflake owns this patent, granted in 2025. ### When does this patent expire? This patent is expected to expire on May 29, 2044, when the invention enters the public domain. ### What problem does this patent solve? Many AI models, especially large language models, can be overly confident in their predictions, even when they are wrong. This patent matters because it provides a structured way to make an AI's stated confidence level more truthful. This is crucial for applications where relying on AI-extracted information, like financial figures or legal clauses from documents, requires high accuracy and trustworthiness. By ensuring the confidence score truly reflects the likelihood of correctness, businesses can better decide when to trust AI outputs versus when human review is needed. ### What does this patent NOT cover? Does not cover improving ML model accuracy during the initial training or prediction phase, only after results are generated. **Full plain-English explainer:** https://patentbrief.org/patent/us/12236201/enhanced-machine-learning-model-accuracy-through-post-hoc-confidence-score-calib **Original patent:** https://patents.google.com/patent/US12236201 --- _Source: PatentBrief — https://patentbrief.org. Patent facts are from public records; the plain-English explanation is PatentBrief's._ ## Related patents Semantically similar inventions in the PatentBrief corpus: - [Improving AI Predictions by Factoring in How Much You Trust Each Model](https://patentbrief.org/patent/us/20230044102/ensemble-machine-learning-models-incorporating-a-model-trust-factor) — This patent describes a method for making an AI system more accurate by combining several individual AI models and giving more weight to the ones considered more trustworthy. - [How to Automatically Detect and Fix Changes in AI Model Data](https://patentbrief.org/patent/us/10599957/systems-and-methods-for-detecting-data-drift-for-data-used-in-machine-learning-m) — This patent describes a system that automatically notices when the real-world data an AI model sees changes, causing its predictions to become less accurate, and then fixes the model. - [How AI Checks Documents for Policy Rules and Compliance](https://patentbrief.org/patent/us/12299406/large-language-model-artificial-intelligence-text-evaluation-system) — This patent describes a system where an AI model helps evaluate parts of a document against specific rules or policies given in plain language, then identifies problems or suggests fixes to ensure compliance. - [How a Computer System Checks and Improves AI Text](https://patentbrief.org/patent/us/11989527/computer-implemented-methods-for-the-automated-analysis-or-use-of-data-including-11989527) — This patent describes a method where a separate computer system, using structured knowledge, checks and corrects the output from a large language model to make it more accurate and consistent. - [How a Computer System Improves Large Language Model Output with Structured Data](https://patentbrief.org/patent/us/12067362/computer-implemented-methods-for-the-automated-analysis-or-use-of-data-including-12067362) — This patent describes a method where a separate computer system analyzes and refines the initial output from a large language model (LLM) by translating it into a highly structured, machine-readable language to reduce bias and improve accuracy.