How accurate is your AI fact‑checker? We tested five.
Behind every fact-check verdict is a language model making judgment calls. We ran 157 real social-media posts through five different models and adjudicated where they disagreed. What we found changed how we think about the cost–accuracy tradeoff — and about whether “offline” AI is a viable substitute at all.
§ 1 · The QuestionWhich model should be doing the work?
Perclaim is a fact-checking application that reads social-media posts and tells you what’s true, false, misleading, or unverifiable. Behind that verdict is a language model — currently GPT-5.4-mini, with OpenAI’s hosted web search tool pulling in current-events context. But it’s not the only option. Other models claim better accuracy, lower costs, or the appeal of running entirely on your own hardware.
So we tested five of them, head-to-head, on the same 157 posts. Same input, same task, same scoring system. The corpus was political-heavy (Trump administration policy, Iran war coverage, election claims) because that’s what users actually submit. Some posts were text-only. Most carried images. The hardest were image-only posts where a screenshot of a Truth Social post or a quote-card meme carried the entire claim.
The five models were:
Claude Sonnet 4.6 — Anthropic’s mid-tier frontier model with the strongest available web search
Claude Haiku 4.5 — Anthropic’s smaller, cheaper frontier model with web search
GPT-5.4-mini — OpenAI’s current Perclaim production model
gpt-oss:120b — an open-weight reasoning model running locally, no web access
gemma4:26b — an open-weight multimodal model running locally, no web access
§ 2 · The MethodTreating the best model as ground truth
Fact-checking evaluations have a chicken-and-egg problem: to score a model’s accuracy you need ground-truth answers, but producing those requires human fact-checkers spending hours on each contested political claim. Inter-rater agreement on charged political content is itself noisy.
Our shortcut: use the most accurate model in the field as a proxy for ground truth, then check it. When Claude Sonnet 4.6 disagreed with GPT-5.4-mini on factually consequential cases, we manually reviewed each one with web-search verification. Sonnet was correct in 6 cases, in tie/legitimate-calibration on 5 cases, and wrong in 0 cases. That gave us enough confidence to use Sonnet’s verdicts as a comparison anchor for the other models.
A caveat we own
Using Sonnet as the reference biases this study toward “models that look like Sonnet rank well.” It’s a proxy, not absolute truth. We report the decomposition of disagreements in detail below so calibration differences are visible separately from substantive ones.
§ 3 · The HeadlineTwo tiers, sharp divide
Here’s how each candidate agreed with Sonnet — first the raw verdict-match rate, then a more useful number: the rate at which the candidate reached the same factual conclusion, allowing for one-notch calibration differences (like calling something “MIXED” where Sonnet called it “MOSTLY_TRUE”) and category swaps (calling something “OPINION” instead of “UNVERIFIABLE”).
Effective concordance with Sonnet 4.6
After collapsing 1-notch calibration shifts and orthogonal category swaps
Haiku 4.5 Anthropic, w/ web search
96.5%
GPT-5.4-mini OpenAI, w/ web search
92.7%
gpt-oss:120b Open-weight, no retrieval
76.8%
gemma4:26b Open-weight, no retrieval
63.2%
The two retrieval-equipped frontier models — Haiku and GPT — cluster tightly with Sonnet, reaching the same factual conclusion 90+% of the time. The two offline models fall off a cliff, dropping to 63–77%. There is no intermediate tier in our sample. The divide is categorical.
§ 4 · The FindingThe retrieval cliff is real
Here’s the part of the data that genuinely surprised us. Models that disagreed with Sonnet were doing so for fundamentally different reasons depending on whether they had web search.
For the frontier models (Haiku, GPT-5.4-mini): disagreements were mostly about calibration. Both models found the same facts. They just labeled them differently — calling a post “MIXED” where Sonnet called it “MOSTLY_TRUE,” or “MISLEADING” where Sonnet said “MOSTLY_FALSE.” Same factual reasoning, slightly different scoring rubric.
For the offline models (gpt-oss, gemma4): disagreements were about whether real events occurred at all. When a post mentioned something that happened after the model’s training cutoff, the offline model couldn’t verify it. So it defaulted to “FALSE.” Over and over.
95%
Of gpt-oss’s “FALSE” verdicts landed on rows where the retrieval-equipped reference identified real, documented events.
Three real examples from the study:
The Vivek Ramaswamy primary win. Sonnet correctly identified that Ramaswamy won the 2026 Ohio GOP gubernatorial primary against Amy Acton (Democratic nominee). gpt-oss insisted: “Ramaswamy is a presidential candidate with no primary win, he is not running for Ohio governor, and Amy Acton is not on the ballot.” gpt-oss was using 2024 training data and couldn’t update.
The FireAid $100M concert. Sonnet confirmed the FireAid wildfire-aid concert raised approximately $100M, citing a House Judiciary Committee report. gemma4 declared: “There is no evidence that a ‘Pacific Palisades Fire Aid’ concert raised $100 million... No such large-scale fund or corresponding official statements exist in the public record.” The concert is a real, widely-covered event.
Virginia Democrats’ redistricting spending. Sonnet confirmed the approximately $62M figure from VPAP data. gemma4 said: “No official financial disclosures or reports from credible news organizations corroborate such an enormous expenditure.” VPAP data is public and well-documented.
When a fact-checker can’t look things up, it can’t tell “claim about a real event” from “fabricated claim.” It just denies both.
This isn’t a quirk to be tuned out with better prompting. It’s a structural limitation. An offline fact-checker is reading a news feed through training-data memories that may be a year or two stale. For current-events fact-checking, that’s disqualifying.
§ 5 · Five Days LaterThe cheap model that almost wasn’t
The original study flagged one model as the most frustrating result: Haiku 4.5. On accuracy it was essentially indistinguishable from Sonnet — zero factual errors against the reference on every substantive disagreement we adjudicated — at a fraction of Sonnet’s cost. But it had a disqualifying flaw: roughly one in eleven responses came back as unparseable output. Sometimes malformed JSON; more often the model declining to answer in conversational prose instead of returning a structured verdict. An 8.9% failure rate is not something you put in front of users.
We treated that as an engineering problem rather than a capability gap — and five days after the original study, we fixed it. The fix was a parser-side recovery cascade: when a response fails to parse cleanly, a first pass tries to extract the verdict structurally even from broken JSON; if that finds nothing, a second pass recognizes a prose refusal and converts it into a clean “UNVERIFIABLE” verdict with the refusal preserved. No model change, no retraining — just better handling of the output the model already produces.
Haiku 4.5 null-verdict rate
Across three replays — original study, prompt fix, recovery cascade
Original 14 May, pre-fix
8.9%
Prompt fix 19 May, intermediate
5.9%
Recovery cascade 19 May, final
1.6%
The failure rate dropped from 8.9% to 1.6% — an 82% reduction. At that level, Haiku is reliable enough to deploy. It now runs in production as Perclaim’s cross-vendor backup: if the primary model has a transient error, the check fails over to Haiku rather than to a second model from the same vendor — so an outage or a systematic quirk at one provider doesn’t take the whole system down.
§ 6 · Five Months LaterThe next generation, checked against real fact-checkers
Models move fast. By October 2026 there was a new generation from both vendors: gpt-6-luna from OpenAI, and Claude Haiku 5.5 and Claude Opus 5.5 from Anthropic. Haiku 4.5, our backup, was also nearing retirement. So we ran the evaluation again — and this time we fixed the two biggest weaknesses of the May study.
First, real ground truth. Instead of trusting one model as the yardstick, we collected 32 claims that professional fact-checkers (PolitiFact, Snopes, Lead Stories, FactCheck.org, AFP, AP and others) had already ruled on, and scored every model against those published verdicts. Second, no memorization. All 32 were fact-checked between July and early October 2026, after the newest models’ training data ends, so no model could simply remember the verdict. They had to search and reason, exactly as they do for a real Perclaim user.
Agreement with professional fact-checkers
32 claims fact-checked July–October 2026 · same scoring as § 3
Claude Haiku 5.5 Anthropic, w/ web search
93.8%
GPT-5.4-mini OpenAI, current primary
90.6%
gpt-6-luna OpenAI, w/ web search
90.6%
Claude Opus 5.5 Anthropic, reference
90.6%
On a set this small, one claim moves a score by about three points, so those four numbers are effectively a tie. The models separated on something more important: how often they got a fact wrong. We adjudicated every substantive disagreement across both the 32 new claims and a re-run of 114 posts from the May corpus — 146 checks per model.
0
Factual errors from gpt-6-luna across all 146 checks. GPT-5.4-mini made two; Claude Haiku 5.5 made one.
GPT-5.4-mini got a Senate vote backwards (it said a senator voted against a bill he voted for) and misdated a ceasefire. It also missed the same fabricated celebrity quote it missed in May, calling it MIXED instead of FALSE.
Claude Haiku 5.5 confused two people and called a real event FALSE. Its other habit is caution: when a quote has no source, it tends to answer “UNVERIFIABLE” rather than “FALSE.” That’s not wrong, but it’s less useful than a clear FALSE with the reason. Unlike Haiku 4.5, it returned a usable verdict on every single check.
gpt-6-luna made no factual errors and caught the fabricated quotes the others missed, at about the same cost as GPT-5.4-mini. Its catch is speed: it takes roughly three to four times as long per check.
Claude Opus 5.5 was our new reference model. It was accurate, but at more than ten times the per-check cost of the production model it didn’t beat the cheaper models on the fact-checker set.
What changed in production: Claude Haiku 5.5 is now the backup, replacing Haiku 4.5. And gpt-6-luna is in a limited live trial as the next primary. If its speed holds up on real traffic, it will replace GPT-5.4-mini for everyone; until then GPT-5.4-mini stays the primary.
§ 7 · The ModelsWhat each one is good for
Combining accuracy, cost, and reliability data, here’s where each model landed. Per-check cost is shown relative to the production model — we run GPT-5.4-mini at roughly a nickel a check, and everything else is expressed as a multiple of that.
Production primary
GPT-5.4-mini
OpenAI · with hosted web search
92.7%
Effective concordance with Sonnet
1×
Per-check cost (baseline)
0%
Failure rate
The model running Perclaim today, across both the free and Pro tiers. Reliable, fast, cost-efficient, and in May never factually wrong against Sonnet on the manually-adjudicated hard cases. The October re-test (§ 6) found two factual errors in 146 checks, which is why gpt-6-luna is now on trial to replace it.
Production backup
Claude Haiku 5.5
Anthropic · with dynamic-filter web search
93.8%
Agreement with fact-checkers
~1–2×
Per-check cost (rel.)
0%
Failure rate
The backup since October 2026 (§ 6). The highest agreement with professional fact-checkers on the post-cutoff set, a usable verdict on every one of 146 checks, and a different vendor from the primary for resilience. It made one factual error, and it leans toward “UNVERIFIABLE” on unsourced quotes.
In trial
gpt-6-luna
OpenAI · with hosted web search
0
Factual errors in 146 checks
~1×
Per-check cost (rel.)
~3–4×
Time per check (rel.)
The most accurate model in the October re-test: no factual errors and it caught fabricated quotes the others missed, at the same cost as the current primary. It is slower, so it is running a limited live trial before it becomes the primary for everyone.
Retired backup
Claude Haiku 4.5
Anthropic · with web search
96.5%
Effective concordance with Sonnet
~2×
Per-check cost (rel.)
1.6%
Failure rate (post-fix)
The biggest surprise of the May evaluation. On accuracy, indistinguishable from Sonnet — zero factual errors against the reference on the substantive disagreements we adjudicated. Once its output-reliability problem was fixed (§ 5), it served as the cross-vendor backup until Haiku 5.5 replaced it in October 2026.
Reference model
Claude Sonnet 4.6
Anthropic · with dynamic-filter web search
100%
Reference baseline
~6×
Per-check cost (rel.)
0%
Failure rate
The most accurate model we tested, and the yardstick the others are measured against. On 11 head-to-head substantive disagreements with GPT-5.4-mini, Sonnet was correct in 6 and never wrong. It cites specific fact-checkers (Lead Stories, PolitiFact, Snopes) with specific dates. We keep it as our evaluation reference rather than a production model — at several times the per-check cost, it doesn’t fit a sustainable consumer price point, and Haiku delivers comparable judgment for far less.
Not viable today
gpt-oss:120b
Open-weight · local hardware · no retrieval
76.8%
Effective concordance with Sonnet
power
Cost is electricity only
6%
Hard failure rate
A capable reasoning model with no way to learn about events after its training cutoff. 95% of its “FALSE” verdicts in this study were on posts that contained real, documented events. Cost is essentially free at runtime, but the systematic false-negative problem disqualifies it for current-events fact-checking. Would require a separate web-retrieval system to be built before it could be considered.
Not viable today
gemma4:26b
Open-weight · local hardware · no retrieval
63.2%
Effective concordance with Sonnet
power
Cost is electricity only
17%
Total failure rate
Multimodal but offline. Same retrieval problem as gpt-oss, plus a much higher overall failure rate driven by JSON parse failures (22 of 152 rows) and hard errors. Same conclusion: not viable without a separate web-retrieval system.
§ 8 · The Honest BitWhat this study isn’t
We’re publishing this because we think it’s a useful picture, but it has real limits.
It’s a sample of 157 posts. Topically heavy on US politics (Trump administration, Iran, elections). Models may calibrate differently on science, health, finance, or local news.
Sonnet isn’t ground truth. It’s the most accurate model among the five we tested. Using it as the reference biases the matrix toward models that calibrate similarly. We show the decomposition (calibration vs. substantive disagreement) so this is visible.
The adjudicator was another LLM. Manual case-by-case review was performed by an AI assistant with web-search verification. The October update partly answers this by scoring against professional fact-checkers’ published verdicts (§ 6), but the substantive disagreements were still adjudicated by an AI assistant.
Each model was run once. Some fraction of observed disagreement is run-to-run sampling noise, not a real difference. A multi-replay variance study is a future work item. (The Haiku follow-up made this concrete: of eleven failures in one run, seven recovered on their own in the next, with no intervention.)
The October fact-checker set is small. 32 claims is enough to catch factual errors but not to rank models that are within a few points of each other. The October re-run of the May posts has another caveat: those posts predate the Claude 5.5 models’ training cutoff, so Haiku 5.5 and Opus 5.5 may already know some answers there. That’s why the post-cutoff set is the fairer test.
Cost numbers are eval-derived. They come from evaluation runs, not production traffic, and we report them only as rough relative multiples rather than precise figures.
§ 9 · What This MeansFor Perclaim users, in plain terms
If you’re using Perclaim today, the model checking your posts is GPT-5.4-mini — a fast frontier model with web search that agrees with professional fact-checkers about 90% of the time and runs at a cost that lets us hold the free tier open. If it ever has a transient problem, your check fails over to Claude Haiku 5.5 — a different vendor’s model — so a hiccup at one provider doesn’t leave you without an answer. We’re trialing gpt-6-luna as the next primary because it made no factual errors in our October tests; the question we’re still answering is whether its slower checks are worth the wait.
We looked hard at whether a more expensive model should sit behind a Pro tier, and decided against it: the most accurate model we tested costs several times more per check without being more accurate on the cases that actually matter — it’s tighter on calibration, not on facts. So Pro and free run the same strong model. What Pro buys you is higher usage limits and the surrounding features, not a different brain behind the verdict. We’d rather put a frontier model in front of everyone than gate accuracy behind a paywall.
And if you’ve wondered whether an “open-source” or “self-hosted” AI could do this work: not yet. Not on current events. Until we build the architecture for a model to search the web independently of OpenAI or Anthropic, the offline candidates will continue to deny real events they have no way to learn about. We’re working on it.