How AI Models Can Be More Confident in Their Document Answers

This patent describes a system for making machine learning models more accurate by adjusting how confident they are in the data they extract from documents, after the initial extraction.

What it covers

A machine learning (ML) system first receives results from an AI model, which include a confidence score and electronic documents (Claim 1). It then performs "document understanding" to pull out specific "data points" from these documents, like a date or an amount (Claim 1). The system connects the initial confidence score to these extracted data points. Next, it uses a "post-hoc calibration (PHC) solution set" (Claim 1) to adjust these confidence scores. This calibration step involves methods like "temperature scaling" or "polynomial regression" (Claim 3) to make sure the confidence score truly matches how accurate the AI's prediction is (Claim 1). After adjusting, the system extracts the final pieces of information, like a specific value, and outputs these values along with their newly adjusted, more reliable confidence scores (Claim 1). For example, if an AI extracts a shipping address from a contract, this patent helps ensure the AI's reported confidence level for that address is truly accurate.

What it doesn't cover

• —Does not cover improving ML model accuracy during the initial training or prediction phase, only after results are generated.
• —Does not cover ML models that do not involve "document understanding" or extracting data points from electronic documents.
• —Does not cover methods of improving ML model performance that do not involve calibrating confidence scores.
• —Does not cover calibration methods that are not "post-hoc" or do not use techniques like temperature scaling, softmax top-k, or polynomial regression.
• —Does not cover general model improvements that are not specifically related to aligning confidence scores with prediction accuracy.

The clever bit

The clever bit is recognizing that an AI's initial confidence score might not truly reflect its accuracy, and then providing specific methods to recalibrate that confidence after the AI has already made its predictions and extracted data from documents. This ensures the reported confidence is a more honest reflection of how likely the extracted information is to be correct.

Why it matters

Many AI models, especially large language models, can be overly confident in their predictions, even when they are wrong. This patent matters because it provides a structured way to make an AI's stated confidence level more truthful. This is crucial for applications where relying on AI-extracted information, like financial figures or legal clauses from documents, requires high accuracy and trustworthiness. By ensuring the confidence score truly reflects the likelihood of correctness, businesses can better decide when to trust AI outputs versus when human review is needed.

Real-world examples

• 1.Snowflake Cortex (Snowflake's AI services)
• 2.AI-powered invoice processing systems
• 3.Automated contract review software
• 4.Intelligent document processing platforms
• 5.Large language models used for data extraction