Stanford researchers retested 50 published findings about large language models on 20 current models and found that 44% had expired and another 34% had weakened. The study, “The shelf life of LLM findings” by Owen Queen and James Zou, sorts each claim as expired, weakened, persists, or strengthened, according to how much of the original effect remains as of September 2026. The lesson is that conclusions about what AI can’t do can go stale quickly, because the systems being studied keep changing.
What they tested. The authors hand-picked 50 published findings on reasoning, hallucination, bias, sycophancy, jailbreaks, prompt injection, and evaluation, reproduced each experiment, and ran it on 20 models from 8 developers released between July 2024 and September 2026. They used 20 samples per query instead of the original papers’ 50.
How fast claims aged. Across all 50 claims, the estimated share that had expired was 4% six months after publication, 16% after one year, and 39% after two years. Among claims with identifiable dates, the researchers estimated an average expiration time of 15.7 months. That figure excludes claims that haven’t expired, so it isn’t an expected lifespan for all LLM findings.
Some areas aged faster than others. Of the nine reasoning claims, five expired, four weakened, and none persisted or strengthened. The Decoder reported that many reasoning findings had faded, while some behavioral tendencies, including sycophancy, persisted. The study also includes a case study on fabricated citations, where the current frontier is at or approaching zero. By one outside count, the oldest model in the panel, GPT-4o mini, has 5 of the 50 findings in the expired zone, while GPT-6.1 Sol has 30.
What it doesn’t mean. The figures don’t say 78% of LLM research is false. They describe this sample of 50 hand-picked, widely cited claims after retesting on newer models. A finding can accurately describe the models, setup, and period originally studied and still not apply to later systems. The authors also stress that this is 50 chosen claims, not the whole literature. “Weakened” isn’t “expired” either: the effect shrank but didn’t vanish.
What the study doesn’t establish. The materials don’t explain why each finding persisted, weakened, or expired, and they don’t show that newer models are better across the board. The study is also a preprint. One commentator expects it to be peer-reviewed, but it hasn’t been yet. Axios has separately reported that AI systems can advance faster than the research processes used to evaluate them, so publication lag is part of the problem.
Why it matters for readers. A paper that says a model can’t do something is a snapshot of the models tested, under those conditions, at that date. Before relying on one, check which models it used, when the experiment ran, and whether anyone has retested it. The same caution applies to vendor benchmarks and to articles like this one.
Bottom line: The study’s practical lesson is to treat findings about AI as dated claims, not permanent facts. Stanford’s retest points toward a maintenance approach: keep the original result, state its scope clearly, and revisit it as the models change.
