How to Automatically Detect and Fix Changes in AI Model Data

This patent describes a system that automatically notices when the real-world data an AI model sees changes, causing its predictions to become less accurate, and then fixes the model.

What it covers

The system detects when the data used by a machine learning model changes over time, a problem called 'data drift'. It does this by first receiving model training data and generating a predictive model. Then, it takes new model input data to generate predicted data. Crucially, it receives 'event data' (real-world outcomes) and compares a 'data profile' of the predicted data to a data profile of the event data (Claim 1). If these profiles differ significantly, indicating data drift, the system then automatically corrects the model. For example, a credit card fraud detection model might be trained on past transaction data. If new types of fraud emerge, the system would compare the model's fraud predictions (predicted data) with actual confirmed fraud cases (event data) to see if the patterns have shifted. If drift is detected, the system could retrain the model using the newer event data (Claim 13) or adjust its internal settings, called hyperparameters (Claim 9).

What it doesn't cover

• —Does not cover detecting data drift without comparing the *profile* of predicted data to the *profile* of real-world event data, as specified in Claim 1.
• —Does not cover systems that only detect data drift but do not automatically initiate a correction of the model, as 'correcting the model' is a required step in Claim 1.
• —Does not cover detecting data drift based solely on changes in the input data *before* it's processed by the model, without considering the model's predictions or real-world outcomes (event data).
• —Does not cover detecting data drift in models that are not 'predictive models', as the claims consistently refer to this specific type.
• —Does not cover methods of drift detection that do not involve receiving 'event data' for comparison with predicted data.

The clever bit

The novelty lies in the system's ability to automatically compare the *data profile* of a model's predictions with the *data profile* of actual real-world outcomes (event data) to detect drift, and then to automatically correct the model. This moves beyond simply monitoring model performance to understanding *why* performance might be dropping and taking proactive steps to fix it.

Why it matters

Machine learning models are increasingly used in critical applications, from finance to healthcare. Their performance can degrade significantly if the real-world data they encounter changes from what they were trained on. This patent addresses a fundamental challenge in maintaining reliable AI systems by automating the detection and correction of such changes. This capability is essential for ensuring that AI models remain accurate and trustworthy over time, especially in dynamic environments.

Real-world examples

• 1.Credit card fraud detection systems
• 2.Loan application risk assessment models
• 3.Algorithmic trading systems
• 4.Predictive maintenance for industrial equipment
• 5.Recommendation engines on streaming services
• 6.Medical diagnostic AI tools