How to Check if a Large Language Model's Answer is True

Amazon's patent details a system for verifying large language model responses by having a second AI translate the answer into formal logic, which an automated theorem prover then checks for validity or invalidity.

What it covers

This patent describes a method for verifying answers from a large language model (LLM). First, a system obtains a 'conversational text' which includes a user's question (query) and the answer from a first LLM. Then, a *second* large language model translates this answer into 'first-order logic,' which is a formal way to represent statements and relationships (Claim 1). An automated theorem prover, specifically a 'satisfiability modulo theories solver,' then checks this logic translation. It determines if the answer is 'valid' by checking if its negation is impossible (unsatisfiable), and if it's 'invalid' by checking if the answer itself is impossible (Claim 1). Finally, the system outputs whether the answer is valid, invalid, or neither, to a user interface, database, or report.

What it doesn't cover

• —Does not cover verifying LLM answers without translating them into first-order logic.
• —Does not cover verification methods that do not use an automated theorem prover or satisfiability modulo theories solver.
• —Does not cover systems where the first LLM directly provides the first-order logic translation without a second LLM.
• —Does not cover verifying answers that are not derived from a query-response conversational text from an LLM.
• —Does not cover systems that only determine validity or invalidity without also identifying 'neither valid nor invalid'.

The clever bit

The novelty lies in using a *second* large language model to translate a natural language answer into a formal 'first-order logic' representation. This allows a traditional, rigorous automated theorem prover to then formally check the truthfulness of the answer, bridging the gap between flexible AI output and strict logical verification.

Why it matters

Large language models sometimes generate incorrect or misleading information, a problem often called 'hallucination.' This patent offers a structured, automated way to check the factual accuracy of LLM outputs. By providing a formal verification process, it aims to make LLMs more reliable for critical applications where accuracy is paramount, such as in scientific research, legal analysis, or financial advice.

Real-world examples

• 1.AI assistants providing factual summaries of documents
• 2.Generative AI tools for legal research
• 3.Medical diagnostic support systems using LLMs
• 4.Automated code generation and verification
• 5.Customer service chatbots providing policy information