{
  "patent_number": "US 12236201",
  "country": "US",
  "title": "How AI Models Can Be More Confident in Their Document Answers",
  "original_title": "Enhanced machine learning model accuracy through post-hoc confidence score calibration",
  "summary": "This patent describes a system for making machine learning models more accurate by adjusting how confident they are in the data they extract from documents, after the initial extraction.",
  "what_it_does": "A machine learning (ML) system first receives results from an AI model, which include a confidence score and electronic documents (Claim 1). It then performs \"document understanding\" to pull out specific \"data points\" from these documents, like a date or an amount (Claim 1). The system connects the initial confidence score to these extracted data points. Next, it uses a \"post-hoc calibration (PHC) solution set\" (Claim 1) to adjust these confidence scores. This calibration step involves methods like \"temperature scaling\" or \"polynomial regression\" (Claim 3) to make sure the confidence score truly matches how accurate the AI's prediction is (Claim 1). After adjusting, the system extracts the final pieces of information, like a specific value, and outputs these values along with their newly adjusted, more reliable confidence scores (Claim 1). For example, if an AI extracts a shipping address from a contract, this patent helps ensure the AI's reported confidence level for that address is truly accurate.",
  "what_it_does_not_cover": [
    "Does not cover improving ML model accuracy during the initial training or prediction phase, only after results are generated.",
    "Does not cover ML models that do not involve \"document understanding\" or extracting data points from electronic documents.",
    "Does not cover methods of improving ML model performance that do not involve calibrating confidence scores.",
    "Does not cover calibration methods that are not \"post-hoc\" or do not use techniques like temperature scaling, softmax top-k, or polynomial regression.",
    "Does not cover general model improvements that are not specifically related to aligning confidence scores with prediction accuracy."
  ],
  "filed": "2024-05-29",
  "granted": "2025-02-25",
  "expires": "2044-05-29",
  "status": "active",
  "holder": "Snowflake",
  "holder_url": "https://patentbrief.org/company/snowflake",
  "inventors": [
    {
      "name": "Andrzej Szwabe",
      "url": "https://patentbrief.org/inventor/andrzej-szwabe"
    }
  ],
  "times_cited": 0,
  "tags": [
    "software",
    "ai_ml",
    "telecommunications",
    "finance",
    "ecommerce"
  ],
  "abstract": "Examples provide enhanced machine learning model accuracy through post-hoc confidence score calibration. A machine learning (ML) system receives results generated by an ML model, the results comprising at least one confidence score and electronic documents. The ML system processes the results generated by the ML model comprising performing document understanding by extracting data points from the electronic documents. The ML system associates the confidence score with the extracted data points and calibrates a confidence score associated with the extracted data points using a post-hoc calibration solution set. The ML system implements confidence scoring recalibration comprising aligning the confidence score with prediction accuracy and adjusting the generated confidence score by the recalibration. Based on adjusting the confidence score, the ML system extracts an individual element of information from the electronic documents comprising an extracted value. The ML system generates an output comprising the extracted values and an adjusted confidence score.",
  "url": "https://patentbrief.org/patent/us/12236201/enhanced-machine-learning-model-accuracy-through-post-hoc-confidence-score-calib",
  "markdown_url": "https://patentbrief.org/patent/us/12236201/enhanced-machine-learning-model-accuracy-through-post-hoc-confidence-score-calib/md",
  "google_patents_url": "https://patents.google.com/patent/US12236201",
  "relatedPatents": [
    {
      "patentNumber": "20230044102",
      "countryCode": "US",
      "title": "Improving AI Predictions by Factoring in How Much You Trust Each Model",
      "url": "https://patentbrief.org/patent/us/20230044102/ensemble-machine-learning-models-incorporating-a-model-trust-factor"
    },
    {
      "patentNumber": "10599957",
      "countryCode": "US",
      "title": "How to Automatically Detect and Fix Changes in AI Model Data",
      "url": "https://patentbrief.org/patent/us/10599957/systems-and-methods-for-detecting-data-drift-for-data-used-in-machine-learning-m"
    },
    {
      "patentNumber": "12299406",
      "countryCode": "US",
      "title": "How AI Checks Documents for Policy Rules and Compliance",
      "url": "https://patentbrief.org/patent/us/12299406/large-language-model-artificial-intelligence-text-evaluation-system"
    },
    {
      "patentNumber": "11989527",
      "countryCode": "US",
      "title": "How a Computer System Checks and Improves AI Text",
      "url": "https://patentbrief.org/patent/us/11989527/computer-implemented-methods-for-the-automated-analysis-or-use-of-data-including-11989527"
    },
    {
      "patentNumber": "12067362",
      "countryCode": "US",
      "title": "How a Computer System Improves Large Language Model Output with Structured Data",
      "url": "https://patentbrief.org/patent/us/12067362/computer-implemented-methods-for-the-automated-analysis-or-use-of-data-including-12067362"
    }
  ]
}