Behind every fact-check verdict is a language model making judgment calls. We ran 157 real social-media posts through five different models and adjudicated where they disagreed. What we found changed how we think about the cost–accuracy tradeoff — and about whether “offline” AI is a viable substitute at all.
Perclaim is a fact-checking application that reads social-media posts and tells you what’s true, false, misleading, or unverifiable. Behind that verdict is a language model — currently GPT-5.4-mini, with OpenAI’s hosted web search tool pulling in current-events context. But it’s not the only option. Other models claim better accuracy, lower costs, or the appeal of running entirely on your own hardware.
So we tested five of them, head-to-head, on the same 157 posts. Same input, same task, same scoring system. The corpus was political-heavy (Trump administration policy, Iran war coverage, election claims) because that’s what users actually submit. Some posts were text-only. Most carried images. The hardest were image-only posts where a screenshot of a Truth Social post or a quote-card meme carried the entire claim.
The five models were:
Fact-checking evaluations have a chicken-and-egg problem: to score a model’s accuracy you need ground-truth answers, but producing those requires human fact-checkers spending hours on each contested political claim. Inter-rater agreement on charged political content is itself noisy.
Our shortcut: use the most accurate model in the field as a proxy for ground truth, then check it. When Claude Sonnet 4.6 disagreed with GPT-5.4-mini on factually consequential cases, we manually reviewed each one with web-search verification. Sonnet was correct in 6 cases, in tie/legitimate-calibration on 5 cases, and wrong in 0 cases. That gave us enough confidence to use Sonnet’s verdicts as a comparison anchor for the other models.
Using Sonnet as the reference biases this study toward “models that look like Sonnet rank well.” It’s a proxy, not absolute truth. We report the decomposition of disagreements in detail below so calibration differences are visible separately from substantive ones.
Here’s how each candidate agreed with Sonnet — first the raw verdict-match rate, then a more useful number: the rate at which the candidate reached the same factual conclusion, allowing for one-notch calibration differences (like calling something “MIXED” where Sonnet called it “MOSTLY_TRUE”) and category swaps (calling something “OPINION” instead of “UNVERIFIABLE”).
The two retrieval-equipped frontier models — Haiku and GPT — cluster tightly with Sonnet, reaching the same factual conclusion 90+% of the time. The two offline models fall off a cliff, dropping to 63–77%. There is no intermediate tier in our sample. The divide is categorical.
Here’s the part of the data that genuinely surprised us. Models that disagreed with Sonnet were doing so for fundamentally different reasons depending on whether they had web search.
For the frontier models (Haiku, GPT-5.4-mini): disagreements were mostly about calibration. Both models found the same facts. They just labeled them differently — calling a post “MIXED” where Sonnet called it “MOSTLY_TRUE,” or “MISLEADING” where Sonnet said “MOSTLY_FALSE.” Same factual reasoning, slightly different scoring rubric.
For the offline models (gpt-oss, gemma4): disagreements were about whether real events occurred at all. When a post mentioned something that happened after the model’s training cutoff, the offline model couldn’t verify it. So it defaulted to “FALSE.” Over and over.
Three real examples from the study:
This isn’t a quirk to be tuned out with better prompting. It’s a structural limitation. An offline fact-checker is reading a news feed through training-data memories that may be a year or two stale. For current-events fact-checking, that’s disqualifying.
The original study flagged one model as the most frustrating result: Haiku 4.5. On accuracy it was essentially indistinguishable from Sonnet — zero factual errors against the reference on every substantive disagreement we adjudicated — at a fraction of Sonnet’s cost. But it had a disqualifying flaw: roughly one in eleven responses came back as unparseable output. Sometimes malformed JSON; more often the model declining to answer in conversational prose instead of returning a structured verdict. An 8.9% failure rate is not something you put in front of users.
We treated that as an engineering problem rather than a capability gap — and five days after the original study, we fixed it. The fix was a parser-side recovery cascade: when a response fails to parse cleanly, a first pass tries to extract the verdict structurally even from broken JSON; if that finds nothing, a second pass recognizes a prose refusal and converts it into a clean “UNVERIFIABLE” verdict with the refusal preserved. No model change, no retraining — just better handling of the output the model already produces.
The failure rate dropped from 8.9% to 1.6%— an 82% reduction. At that level, Haiku is reliable enough to deploy. It now runs in production as Perclaim’s cross-vendor backup: if the primary model has a transient error, the check fails over to Haiku rather than to a second model from the same vendor — so an outage or a systematic quirk at one provider doesn’t take the whole system down.
Combining accuracy, cost, and reliability data, here’s where each model landed. Per-check cost is shown relative to the production model— we run GPT-5.4-mini at roughly a nickel a check, and everything else is expressed as a multiple of that.
The model running Perclaim today, across both the free and Pro tiers. Reliable, cost-efficient, and on manually-adjudicated hard cases, never factually wrong against Sonnet — just slightly less precise on calibration. This is the production default.
The biggest surprise of the evaluation. On accuracy, indistinguishable from Sonnet — zero factual errors against the reference on the substantive disagreements we adjudicated. Once its output-reliability problem was fixed (§ 5), it became the obvious cross-vendor backup: nearly Sonnet-grade judgment, a fraction of the cost, and a different vendor than the primary for resilience.
The most accurate model we tested, and the yardstick the others are measured against. On 11 head-to-head substantive disagreements with GPT-5.4-mini, Sonnet was correct in 6 and never wrong. It cites specific fact-checkers (Lead Stories, PolitiFact, Snopes) with specific dates. We keep it as our evaluation reference rather than a production model — at several times the per-check cost, it doesn’t fit a sustainable consumer price point, and Haiku delivers comparable judgment for far less.
A capable reasoning model with no way to learn about events after its training cutoff. 95% of its “FALSE” verdicts in this study were on posts that contained real, documented events. Cost is essentially free at runtime, but the systematic false-negative problem disqualifies it for current-events fact-checking. Would require a separate web-retrieval system to be built before it could be considered.
Multimodal but offline. Same retrieval problem as gpt-oss, plus a much higher overall failure rate driven by JSON parse failures (22 of 152 rows) and hard errors. Same conclusion: not viable without a separate web-retrieval system.
We’re publishing this because we think it’s a useful picture, but it has real limits.
If you’re using Perclaim today, the model checking your posts is GPT-5.4-mini — a frontier model with web search. Our evaluation says that’s a defensible choice: it reaches the same factual conclusions as the best available model 93% of the time, makes no factual errors against that reference on hard cases, and runs at a cost that lets us hold the free tier open. If it ever has a transient problem, your check fails over to Claude Haiku 4.5 — a different vendor’s model, with near-identical accuracy in our tests — so a hiccup at one provider doesn’t leave you without an answer.
We looked hard at whether a more expensive model should sit behind a Pro tier, and decided against it: the most accurate model we tested costs several times more per check without being more accurate on the cases that actually matter — it’s tighter on calibration, not on facts. So Pro and free run the samestrong model. What Pro buys you is higher usage limits and the surrounding features, not a different brain behind the verdict. We’d rather put a frontier model in front of everyone than gate accuracy behind a paywall.
Read the full research paper: Cross-Vendor Concordance Study (PDF)
And if you’ve wondered whether an “open-source” or “self-hosted” AI could do this work: not yet. Not on current events. Until we build the architecture for a model to search the web independently of OpenAI or Anthropic, the offline candidates will continue to deny real events they have no way to learn about. We’re working on it.