LLM Misinformation Scanner
Language models state false things in the same confident register as true things. Can a system flag that before a person acts on it?
Try the product here.
The working prototype will be added with the final content.
Context
Model outputs are fluent by construction. Fluency is what makes a wrong answer expensive — there is no surface signal that separates a well-grounded sentence from an invented one, so the reader carries the entire verification burden.
The work began in Carnegie Mellon's Applied Generative AI Lab and sits in the same space as AI safety, reliability, and evaluation: not preventing generation, but attaching a usable trust signal to it.
Why it matters
Once an LLM sits inside a workflow, the failure is rarely the model being wrong. It is the model being wrong and nobody noticing until downstream. A confidence score is only useful if it is legible — a number a reviewer can act on, with the claim that caused it.
The space of detection methods
These are published research directions, not results I am claiming. The product question was which of them survives contact with a real workflow.
- Claim decomposition — split a response into atomic, independently checkable statements.
- FActScore-style verification — score each atomic claim against retrieved evidence and aggregate.
- Semantic entropy — sample multiple generations and measure disagreement in meaning rather than in wording.
- Verifier-based evaluation — a separate model judges the claim, with the checker decoupled from the generator.
What I am building first, and what I deferred
Building first. Claim decomposition plus evidence-backed verification. It is the most inspectable path — every score traces to a specific claim and a specific source, which is what makes the output reviewable.
Deferred. Sampling-based semantic entropy. Strong signal in the literature, but the multi-generation cost and the lack of a citable reason behind each score make it a poor first surface for a user who has to act on the output.
Deferred. A fully learned end-to-end scorer. Nothing to point at when it is wrong, and no calibration story before there is evaluation data of my own.
Major tradeoffs
- Interpretability over raw benchmark score — a slightly weaker detector whose output can be audited beats a stronger one that cannot.
- Retrieval quality is the ceiling. A verifier is only as good as the evidence it is handed, so retrieval is treated as part of the product, not as plumbing.
- Latency budget forces the design. Per-claim verification is parallelisable; sampling-based methods are not, at the same cost.
Where it stands
In active development. The current focus is the decomposition and verification pipeline and an evaluation harness of my own, so that any accuracy claim I make later is measured rather than borrowed.
What I'd do next
- Build a held-out evaluation set and report calibration, not just accuracy.
- Test whether reviewers actually change decisions when shown claim-level scores.
- Revisit semantic entropy as a second-pass signal on low-confidence spans only.