AI Made It Possible · Evidence Test

Did AI Really Figure That Out? How to Test Novelty, Generalization and Memorization

If an AI gives us an answer no person in the room could produce, that is impressive. But what exactly does it prove?

The answer depends on what the system had already seen, what tools it could use, whether the task was genuinely new, and whether the result survives independent verification.

The basic problem: performance is visible, process is harder to see

Imagine two students get the same answer right.

One saw the answer yesterday and remembered it. The other had never seen the problem and worked it out from first principles.

The score is identical. The evidence about intelligence is not.

AI evaluation has the same problem at a much larger scale because modern models are trained on enormous amounts of data. A system may have seen the exact problem, a close variation, a discussion of the answer, a code patch, a textbook explanation, or thousands of structurally similar examples.

Why François Chollet's distinction matters

François Chollet's work on measuring intelligence argues that skill on a task is not the same thing as intelligence itself. A system can become extremely skilled if it receives enough prior knowledge, training data and exposure.

His alternative focuses on skill-acquisition efficiency: how effectively a system can learn and adapt when faced with new problems.

That is a useful distinction because a model that has seen millions of examples can look brilliant on familiar tasks while still struggling when the structure of the problem changes.

Benchmark contamination is not a minor technical detail

A benchmark is supposed to test what a model can do.

But if the benchmark, its answers, or close variants appear in training data, the test can become partly a memory test.

That concern is no longer theoretical. In 2026, OpenAI stopped relying on SWE-bench Verified as a frontier coding measure after finding signs that models could reproduce task-specific details that suggested contamination. OpenAI's broader evaluation guidance now recommends preferring private or newly created tasks and checking whether models can reproduce benchmark-specific information before solving.

Google DeepMind has gone even further by piloting double-blind evaluations designed to keep proprietary test items hidden from the model developer and prevent future optimization against the test.

Why private and newly created tasks are stronger evidence

ARC-AGI uses private evaluation tasks specifically to reduce leakage and overfitting. Its core idea is to give systems a small number of examples and then ask them to infer the rule needed to solve a new case.

Humanity's Last Exam was built for a related reason: many older academic benchmarks had become too easy for frontier models. Its questions are designed to be original, difficult and resistant to simple internet lookup or database retrieval.

OpenAI's FrontierScience work similarly holds back part of its scientific evaluation set to help track contamination and uses original research-level tasks designed by scientists.

Retrieval is not fake intelligence—but it changes the claim

This is where I think public discussions often become unfair in both directions.

If an AI agent is allowed to search the web, use a calculator, query a database and run code, then using those tools is part of the system's capability. Humans use tools too.

But if we want to know whether the underlying model can reason through a new problem from limited information, then giving it unrestricted retrieval changes the experiment.

Both can be valuable. They are simply measuring different things.

A research assistant that knows how to find the right paper may be extremely useful. A model that can independently derive a new principle is making a stronger cognitive claim.

What would count as stronger evidence of real generalization?

Novel discovery is a stronger test than exam performance

There is a difference between answering a difficult question and producing knowledge humanity did not already have.

If an AI proposes a new theorem, drug candidate, material, engineering design or scientific mechanism, and independent experts verify that it works, the evidence becomes much stronger.

Even then, we should still ask how the result was produced. But a validated discovery that could not simply have been retrieved from existing literature is much harder to dismiss as memorization.

What if the strategy does not look human?

There is another possibility that becomes more important as systems improve: the AI may reach a correct result using a strategy that no person explicitly taught it and that does not resemble the way a human expert would approach the problem.

That does not automatically prove a new form of intelligence. Training can produce internal representations and problem-solving shortcuts that are difficult for humans to interpret even when the underlying behavior still grows from patterns in human-created data.

But it does change the test.

If a system repeatedly solves genuinely new problems using an approach that was not present in the examples it was trained on, transfers that approach to other unfamiliar problems, and produces independently verifiable results, then “it only copied us” becomes a weaker explanation.

The important evidence is not that the method looks alien. It is that the method works, transfers, and survives tests designed to rule out memorization and retrieval.

What would make the evidence stronger?

Novel problems. The task should be created after training or otherwise kept outside likely exposure.

Transfer. The strategy should work across multiple unfamiliar tasks rather than one lucky example.

Interpretability where possible. Researchers should inspect internal mechanisms instead of relying only on verbal explanations generated by the model.

External verification. A surprising method matters more if independent experts can confirm that it produces a correct theorem, design, prediction or scientific result.

Resistance to prompting changes. If the capability disappears whenever the wording changes, it may be shallower than it first appeared.

The clone question gets harder here

This connects directly to the problem raised in the superintelligence article.

Suppose an AI has absorbed so much human knowledge that it can combine our ideas faster, more broadly and more accurately than any individual person.

At some point, that recombination may produce results no single human has produced before.

Does that count as intelligence?

I think the better question is more specific:

What kind of intelligence has been demonstrated?

A system can be derivative in origin and still become generative in consequence. Human knowledge itself is cumulative. We also learn from previous people, absorb patterns, imitate, recombine and eventually create.

So “it was trained on human data” cannot, by itself, settle the argument.

The stronger test is whether the system can continue adapting when the answer is not already embedded in the material it learned from.

Beginner's glossary

Sources and further reading

• François Chollet — On the Measure of Intelligence
• ARC Prize — ARC-AGI-2 Technical Report
• Nature — Humanity's Last Exam
• Google DeepMind — Double-blind AI evaluations
• OpenAI — Why SWE-bench Verified no longer measures frontier coding capabilities
• OpenAI — Trustworthy third-party evaluations
• OpenAI — FrontierScience

Research review: September 23, 2026. This article distinguishes observable performance from the harder question of how that performance was produced.