djinn six icon
djinn six®

The Sixth Layer

← Back to blog
ProbeSix

Wrong on Sunday right on Monday: why one passing test tells you so little about AI

Allen Clark

Allen Clark

Wrong on Sunday right on Monday: why one passing test tells you so little about AI

Ask an AI the same question on Sunday and again on Monday. Should you get the same answer?

Most of us would say yes without thinking. For years we have tested software on just that assumption. A test that passed yesterday, on the same code, passes today. That's the whole basis of a regression suite, a quarterly penetration test and an annual audit.

This month, a run of evidence from very different places says that assumption doesn't hold for AI. Not in a fringe way, and not only for obscure models. And it matters for security more than for anything else.

What Full Fact found

Full Fact, the UK's fact-checking charity, has published the first report from its Public Interest AI Trust Benchmark, under the title "Wrong on Sunday, right on Monday". Between 15th July and 22nd September a team of fact checkers and data scientists asked five widely used chatbots (Gemini, ChatGPT, Grok, DeepSeek and Claude) the same 22 questions every day. They collected 7,625 answers and checked them against references their own fact checkers had researched.

Most answers were right. The failures are what make the report worth reading:

  • The same question got different answers on different days. Asked when England last won a major football trophy, several models repeatedly named the 2022 Euros as the women's most recent win, overlooking the 2025 tournament.
  • The world changed and some models didn't notice. DeepSeek was still naming Sir Keir Starmer as Prime Minister and Labour leader as late as September.
  • Superseded figures were presented as current. Several chatbots gave a weekly asylum support rate of £8.86, replaced by £9.95 in June 2025. Gemini 3.1 Pro and Claude gave the outdated figure outright in some answers.
  • Proposals were reported as settled. Models described a proposal for asylum seekers to contribute £10,000 towards their living costs as though it had been adopted, and said bank cards were already accepted as voter ID.
  • Sources were wrong in ways no human would be. Asked about the MP for a Northampton postcode, Gemini 3.1 Pro cited a 2012 PDF of a local American newspaper.
  • Figures came from the wrong table. ChatGPT gave a figure for people in Scotland who could not speak English well that actually came from a separate table broken down by whether respondents were transgender.

A huge takeaway from this is a line that I'd want to put on every AI risk register now: every answer from the LLM arrived in the same assured tone, whether it was right, out of date or premature. In Full Fact's words:

"confidence is no guide to reliability."

The same pattern, in the research literature

Full Fact were testing facts. The October issue of Communications of the ACM shows the same gap in systems that act, not just answer.

Kept running is not the same as correct. In "Toward Agentic Runtime Healing", Sun, Zhu, Xu, Du, Li and Lo let an LLM write and run code on the fly whenever a program hit an error nobody had planned for. Across their benchmarks, GPT-4 kept the program running in 73.0% of cases and produced the correct result in just 38.7%. In the gap between those two numbers, the system carried on, looked healthy, and gave the wrong answer. When the authors examined the faulty fixes, some quietly skipped the rest of the original logic or dropped a return value. To their credit, they ask the obvious question themselves, "What if the LLM deletes my database?". Their answers are a whitelist of permitted behaviours checked before anything runs, a sandbox and a restricted toolkit.

Leaderboards are not your system. Michael Stonebraker and Peter Baile Chen of MIT compare public text-to-SQL leaderboards where accuracy runs above 80% and in some cases above 90%, with what happened on real company data warehouses, including MIT's own 1,400-table system. A plain LLM scored zero, with retrieval, prompt engineering and agents it was raised to the 10% range. Even when they handed the model the exact tables and joins it needed, it still only reached the 30% range. Their explanation was that public benchmark data may well be in the training data and real systems have years of schema rot, local jargon and harder questions.

Failures don't sit in one place. Ian Foster's Opinion column on agency argues that when an agent fails, the cause may be spread across a sequence of locally reasonable decisions that add up to a bad outcome. In his atricle he suggests that verifying such systems may need probabilistic guarantees and safety envelopes that bound the worst case, not classical pass or fail.

From the most experienced voice in the room. Nancy Leveson, who has worked on safety and security for more than 40 years, has just taken over CACM's Inside Risks column. Her list of open problems includes hallucination, sycophancy, jailbreaking and Big Tech overclaiming what its models can do. Her blunter point is about timing: for systems of today's complexity, a high level of assurance after development is impossible.

Why this is a security problem, not just an accuracy one

In my last two pieces I made two arguments.

  • Pass or fail is not enough: a right answer can rest on the wrong reasoning, and a single verdict hides that. Full Fact found exactly this: models quoting a near-correct figure taken from the wrong table. A pass/fail mark scores that as right.
  • $50 a breach : AI agents have made attacker patience almost free, with agents working a single target for nearly a day at around $5 an attempt.

Put those together with this month's evidence and the conclusion is uncomfortable. If a model's factual answer can change from one day to the next, there is no reason to assume its refusal never does. A guardrail that holds 99 times in 100 is a guardrail that fails once. Against a human attacker who gives up after a few tries, that might be acceptable. Against an agent that will ask the same question a thousand different ways overnight, it isn't. Inconsistency is attack surface. The attacker only needs the bad day once.

The runtime healing paper also adds a second half of the problem, when an AI system fails, it often doesn't look like a failure. It keeps running in the same confident tone Full Fact describes and the wrong result flows downstream.

Check the checker

There is another lesson in the Full Fact report and for anyone who tests AI for a living it may be critical.

To score thousands of answers, Full Fact used an AI model to break each one into individual facts. They checked it by hand and it was right 97.8% of the time which is a really good number. But in one case the scoring step itself recorded that a million people living in the UK do not speak any English at all, a claim the report calls "both wrong and harmful." The chatbot never said it, the original answer made clear the figure included people who could not speak English well so it was the grader that created the error.

More and more AI testing, including security testing, now uses another model as the judge. That's often the only practical way to assess thousands of conversations., but the judge is an AI system too with all the properties above.

If you can't say how often your grader is wrong, measured against careful human labels, you don't know what your test results mean. Agreement between two models is not the same thing because they can agree and both be wrong.

What to do about it

For anyone deploying AI in a regulated environment, or assessing someone who is:

  1. Ask more than once. Run the same test repeatedly, on different days and with small, boring variations in wording, language and format. Treat a change of answer as a finding, not noise.
  2. Separate "it carried on" from "it was right". Measure outcomes, not just availability. A system that never crashes and is wrong a third of the time is not reliable.
  3. Test where it runs. A model's published scores describe someone else's test set. Your system prompt, your data, your tools and your users are what matter.
  4. Check that it knows what day it is. Test whether time-sensitive facts are current and whether the system distinguishes what is in force from what has merely been proposed.
  5. Measure the grader. If a model marks your tests, measure it against human judgement before you trust its verdicts, and keep measuring.
  6. Constrain what an agent can do, not just what it says. Permitted actions, sandboxes and restricted toolkits, the controls the runtime healing authors propose, are the questions an assessor should be asking.
  7. Make it continuous. A point-in-time pass on a system that answers differently tomorrow is a snapshot, not an assurance.

None of this means AI can't be trusted. It means trust has to be earned the way we earn it for every other control in a regulated firm: by showing that it behaves the same way, every time, under pressure. This month's evidence says we can't assume that. We have to test for it.

Sources

← More insights from djinn six
    Wrong on Sunday right on Monday: why one passing test tells you so little about AI | djinn six