Learning, Abduction, and Scientific Discovery

What is missing when a solver succeeds?

A system can solve a difficult problem and still leave the central scientific choice to someone else. The choice may be the variables, the representation, the admissible explanations, or the definition of a successful result. This does not diminish the solution. It changes what that solution demonstrates.

Tom Zahavy’s position paper LLMs can’t jump uses scientific invention to question a narrow account of intelligence as induction and deduction. Its central concern is how a system chooses new premises, not just how well it reasons from supplied premises. Read that as a research thesis. The paper does not prove that transformers, language models, or learned predictors cannot create concepts.

The useful question is operational: what must an experiment show to distinguish framework revision from excellent search within an existing framework? This note follows the historical motivation, the proposed role of interactive world models, and the evaluations that could make the disagreement testable. The title is a challenge to investigate, not a theorem to assume.

Deduction, induction and abduction

Deduction preserves truth when the premises are true and the inference is valid. It does not create empirical truth by itself. Induction generalizes from experience; a future case can defeat a useful generalization. Abduction proposes an explanation that would make an observation intelligible. A plausible explanation is not thereby verified.

These operations interact. An abductive proposal creates premises; deduction exposes their consequences; observations support or challenge them; induction summarizes regularities. A learned model can participate in more than one operation. Calling a training objective predictive does not settle which reasoning operations the trained system can perform.

Peirce’s account motivates the distinction, while Magnani’s account gives a role to manipulating representations and imagined situations. Neither distinction is an architectural classification of current AI systems.

Follow a premise from invention to test

The paper uses an Einstein-inspired sequence: experience E, a conceptual jump J, axioms A, and consequences S. The return from S to experience matters as much as the jump. A new premise that explains nothing, predicts nothing, and admits no test is not made scientific merely by being novel.

The diagram separates roles, not a mandatory linear schedule. A failed consequence can change a hypothesis before any physical experiment. An unexpected observation can change the representation of the problem. A successful prediction can justify retaining an approximation in one regime while rejecting it in another.

The paper’s own first diagram is an AI-generated reconstruction, not a facsimile of Einstein’s letter; its caption notes generated inaccuracies. The scene here is an original conceptual diagram and makes no claim to reproduce that historical document.

General relativity: why change the framework?

General relativity is useful here because the problem was not simply “fit a different coefficient.” Newtonian gravity already explained a great deal. The relation between inertial and gravitational mass had strong empirical support. Mercury’s unexplained perihelion precession was a tension, but it was not the only conceptual pressure on the theory.

One could respond to an anomaly by adding a missing object, as in the proposed planet Vulcan. A deeper response changes what gravity means. In the general-relativistic framework, spacetime geometry and free motion replace a Newtonian force description. The conceptual change reorganizes several constraints rather than only lowering one residual.

For this note, the sequence is motivation → candidate principle → mathematical consequences → comparison with experience. Do not turn a retrospective story about Einstein into a universal account of discovery. The final theory also depended on mathematical work, earlier physics, observational constraints and difficult revisions.

The case matters for AI because a benchmark often fixes the objects and language that a solver may use. Changing that language is different from discovering a strong answer expressed in it. Both achievements can be valuable; they require different evidence.

A conceptual jump is not a jump without constraints

The conceptual tensions included action at a distance versus a field description, the status of inertial frames, the equivalence of gravitational and inertial effects, and compatibility with earlier successful physics. Conservation, covariance and the weak-field Newtonian limit constrained acceptable proposals.

Thus “little new data” does not mean “no signal.” A candidate can be ruled out by internal inconsistency, a wrong limiting case, or a conflict with an established observation. A constraint is evidence against some candidates even when it is not a large training dataset or a differentiable loss.

Premise generation and premise checking must remain distinct. Search can help formalize a proposed principle and eliminate failures. That does not explain, by itself, where the principle came from. Conversely, an inspiring principle without successful formalization and checks is not a finished theory.

The engineering analogue is explicit: record which assumptions a system inherits and which it proposes to replace. Otherwise a successful derivation can be mistaken for an autonomous choice of its starting concepts.

The distinction is between proposing these requirements and finding mathematical objects that satisfy them. This comparison is not a sufficient recipe for deriving unique field equations. Einstein Online explains the link between matter, geometry and geodesic motion ↗.

Thought experiments can expose a missing distinction

An elevator thought experiment compares free fall, support in a gravitational field, and acceleration. Manipulating these imagined situations can expose a relation that is difficult to see in a list of observations. This is the point of manipulative abduction: changing a scenario or representation can help generate an explanation.

State the domain carefully. The equivalence comparison is local. It does not make every gravitational field globally identical to uniform acceleration; spatial variation and tidal effects require further distinctions. An illustration that omits those effects must not claim global equivalence.

This suggests an AI experiment: vary mass, support, acceleration, reference frame and field conditions; then ask which relations persist. The experiment should specify what its environment actually represents. A convincing video of an elevator is not evidence that its simulator has the right physical invariants.

The historical thought experiment motivates the proposal. It is not a benchmark showing that an AI world model has rediscovered general relativity.

Where prediction and compression help—and where they do not settle the question

Prediction and compression are productive scientific tools. Sparse regression and system identification can recover compact equations when the measurements, variables, candidate terms and operating regime are suitable. Good prediction can reveal structure; a compact representation can make a structure easier to use.

The difficulty is treating these successes as a complete explanation of invention. Low error in a sampled regime may leave competing causal explanations unresolved. A sparse observation set can support several models. A new concept may require a different vocabulary, and an initially awkward formulation may become simple only after later mathematical work.

Simplicity also depends on representation. Counting symbols in one language can penalize a theory that becomes short in another. A claim that the final theory is elegant does not establish that an inventor could select it by minimizing that exact criterion at the start.

The argument therefore challenges a narrow account of invention as fixed-language compression. It does not show that compression cannot participate in invention. Predictive fit, invariance, consistency, causal discrimination and explanatory structure can be complementary criteria. Information Theory and Compression owns the coding and prediction connection; this note owns the scientific boundary question.

An interactive world model is a proposal, not a solution

The proposed direction is an interactive world model: notice a tension, construct a counterfactual situation, intervene, inspect what remains invariant, propose hypotheses, formalize them, and test their consequences. The critical distinction is passive prediction versus controlled intervention—not simply language versus video.

A language model with tools can interact with an environment. A video model can remain a passive predictor. The medium alone does not establish grounding, intervention semantics, or scientific validity. Action-controllable systems such as Genie illustrate interaction, but that capability is not a certification of physical laws.

An environment needs a declared domain and independently checked behavior. If a learned simulator supplies both the imagined evidence and its judgment, a model can confirm the simulator’s errors. Separate internal plausibility from external validation, and test transfer to observations or interventions the simulator did not manufacture.

The proposal is promising precisely where representations can be manipulated and consequences tested. Its missing component is the bridge from useful latent regularities to explicit, falsifiable principles. The paper does not supply a general algorithm that closes that gap.

Observation does not specify an intervention

Consider two supplied causal explanations. In A, X causes Y. In B, an unobserved U causes both X and Y. Let U be a fair binary variable. Both explanations make X and Y equal in passive observations. Observing X=1 therefore does not distinguish them.

Setting X is a different operation. Under A, setting X changes Y. Under B, setting X does not change U, so Y still follows U. The scene adds the same 10% measurement-flip probability to both explanations so that outcomes remain noisy rather than falsely decisive.

Try observing X=1 first: changing the selected outcome leaves the prior odds unchanged because both hypotheses give the same likelihood. Then set X=1 or X=0. Now the likelihood ratio can move the odds. A surprising noisy outcome can support the alternative hypothesis; no single result proves a law.

This illustrates why an intervention can be informative when passive fit is not. It does not demonstrate automated abduction: both hypotheses and their structural laws were supplied by us. Inventing a third explanation remains a separate task.

Separate generating a hypothesis from weighing its evidence

With equal prior odds, setting X=1 and measuring Y=1 gives likelihoods .9 and .5. The posterior probability of A is $\frac{0.9}{0.9+0.5}=\frac{9}{14}\approx0.6429$. A passive observation of X=1 and measured Y=1 instead gives .9 under both explanations, so the posterior stays .5.

Expected information gain averages uncertainty reduction over the possible outcomes before observing one. The uncertainty after a particular outcome can increase when the evidence makes previously unequal hypotheses harder to distinguish. Do not interpret every single outcome as a positive gain.

The probabilities are exact only within the toy’s declared two-hypothesis model. If both hypotheses are wrong, normalizing their probabilities still produces a winner. A scientific workflow needs ways to detect model misspecification and expand the hypothesis class, not only a posterior over a closed list.

Bayesian updating weighs candidates; it does not specify how candidates are generated. Strong priors learned from pretraining also make “only a few examples” an inadequate argument that induction is impossible. Probability and Distributions gives the general conditioning rules.

From an invariant to a usable axiom

A latent pattern is not yet an axiom. The system must name quantities, define measurement procedures, declare the applicable domain and express a relation that yields consequences. It must also distinguish a true invariant from an artifact of its simulator, dataset or coordinate choice.

For example, a representation might remain stable across a set of controlled transformations. The next questions are: what physical or computational quantity does it represent; which transformations are valid; what changes would break the relation; and what independently measurable prediction follows? Without that bridge, “the representation contains the law” is difficult to test.

Formalization can fail for informative reasons. A statement may be dimensionally inconsistent, fail in an expected limit, or provide no discriminating prediction. These failures should feed back into hypothesis generation rather than be hidden by an aggregate score.

The bridge also depends on the field. Executable computer-science ideas have a different testing substrate from observational astronomy or laboratory physics. There is no reason to assume one route from latent variables to formal principles will serve every domain.

Who supplied the scientific task?

AlphaProof and AlphaEvolve make the boundary concrete. Formal proof search operates against a specified statement, formal language and checker. Program evolution operates against an initial implementation and an automatic evaluation setup. These systems can produce genuinely novel, difficult and useful solutions; “supplied framework” is not a synonym for trivial work.

A clear report distinguishes proposal authorship, hypothesis-class choice, ontology choice, experimental design and evaluation. “AI scientist” alone leaves those distinctions unresolved. An autonomous agent can still inherit the scientific question and all of its validation assumptions.

Use Agent Training and Environments for interaction and environment contracts, and Building and Controlling Reasoning Models for reasoning budgets and verification. Neither a longer trace nor more tool calls establishes framework invention by itself.

Evaluation feedback does not require a gradient

Do not equate feedback with a gradient. AlphaEvolve’s outer loop uses LLM-generated code, automatic evaluation and evolutionary selection. Its evaluation function can return scalar metrics without differentiating through candidate generation. Some evolved programs can contain gradient-based learning; that does not make the outer search gradient descent.

This correction matters for the scientific argument. A shortage of labeled examples is different from a shortage of usable constraints. A non-differentiable checker, measured outcome, consistency test or limiting-case check can still guide a search.

The relevant limitation is narrower: a supplied evaluator can favor the wrong objective or exclude the representation that a new explanation needs. Identify that boundary instead of asserting that every current solver requires a dense error signal or derivative.

What changes when the evaluator is part of the hypothesis?

The ordinary loop proposes a candidate, evaluates it and improves it. A meta-loop can also question the variables, the candidate language, the premises and the evaluator. This is where a proposed account of scientific invention differs from routine optimization.

Changing an evaluator is not inherently progress. A system can make a failing candidate appear successful by weakening a test. Keep independent evidence, previous failures and non-negotiable constraints visible. A new evaluation rule needs justification and prospective predictions, not merely a higher self-reported score.

A useful log records both kinds of change: “parameter revised within the same model” and “representation or premise revised.” It should explain which tension motivated the revision and which new test can distinguish the revision from its predecessor.

This gives the research thesis a measurable target without demanding a metaphysical definition of creativity. Track what changed, who supplied it, and whether the change survived a test it was not selected to pass.

Turn the thesis into discriminating tests

Historical reconstruction needs controlled knowledge access. A model pretrained on later science can retell a discovery without recreating the conditions that made it difficult. A nominal cutoff or withheld prompt is not enough if the solution remains in pretraining, tools, examples or the evaluator.

Prospective tasks can reduce that problem: freeze the available information, record candidate explanations before new evidence arrives, and compare predictions under independent tests. Report failures as well as successes. Include baselines that search the same supplied hypothesis class.

These tests still do not prove a general ability to invent. They make specific claims more credible and identify where human design remains essential. Treat these questions as evaluation contracts, not as announced capabilities.

What the argument does not establish

First, the paper supplies no impossibility theorem for transformers. Showing a conceptual distinction does not show that a trained architecture cannot implement the operation. A rigorous negative claim would require a defined capability, assumptions and an argument that excludes it.

Second, general relativity is one historical case, not a complete sample of science. Data-rich experimental discovery, mathematical proof, engineering search and theory revision can require different combinations of prediction, manipulation and abstraction.

Third, sparse data are not absent information. Pretraining priors, background knowledge, consistency, invariance, conservation and previous observations constrain candidates. The claim that two to five examples cannot support induction overlooks those priors and assumptions.

Fourth, grounding is neither established as necessary nor shown to be sufficient for invention. A tool-using language model could interact meaningfully; an interactive visual model could learn the wrong physics. The proposal must be tested at the level of intervention and validation contracts.

Finally, the induction–abduction boundary can blur in practice. A system may recombine prior structures to propose a useful explanation. Decide what your experiment tests rather than using one label to dismiss or certify the whole process. The strongest reading of the paper is a demand for better scientific evaluations, not a verdict that present architectures can never invent.

A small experimental loop with explicit contracts

A small loop can make responsibilities visible without pretending to solve automated science. Candidate generation, formalization, experiment selection and external measurement are separate interfaces. Each must document what it receives from a human, pretrained model or simulator.

tension = compare(evidence, current_theory)
candidates = propose(tension, assumptions)
for hypothesis in candidates:
    contract = formalize(hypothesis)
    if not check_consistency_and_limits(contract):
        continue
    test = choose_discriminating_test(contract, alternatives)
    prediction = predict(contract, test)
    observation = measure_independently(test)
    record(prediction, observation, provenance)
revise_with_failures_visible()

This is an interface sketch, not a runnable invention algorithm. The hardest operations are intentionally named, not hidden. In particular, measure_independently must not simply ask the same generator to confirm its prediction. Record uncertainty, measurement noise, resource cost and failed tests.

Keep a useful separation in your own research notes: established mechanism, historical interpretation, proposed capability, and experimental evidence. A proposal can be worth studying before it is demonstrated. Labeling it accurately makes that value clearer, not smaller.

Sources, provenance and connected topics

The author copy and proceedings record can differ; cite the version used for a specific claim. Zahavy supplies the research thesis. The other references provide context on abduction, interaction, formal verification and search. They do not collectively establish that a world model can invent scientific axioms.

• Zahavy (2026), Position: LLMs can’t jump — official ICML/PMLR record ↗; author PDF ↗; original PhilSci-Archive reference ↗.
• Peirce (1934), Collected Papers: background on abduction, induction and deduction.
• Magnani (2009), Abductive Cognition: background on manipulative abduction.
• Bruce et al. (2024), Genie: Generative Interactive Environments ↗. Action control is not validated scientific physics.
• AlphaProof (2025), Olympiad-level formal mathematical reasoning with reinforcement learning ↗. Formal proof search against a specified checker.
• Novikov et al. (2025), AlphaEvolve ↗. Automatic evaluation and evolutionary code search.
• Schmidhuber (2008), Driven by Compression Progress ↗. The compression/curiosity connection, not a proof that all invention reduces to fixed-language compression.

Continue with Information Theory and Compression for prediction and coding, Probability for evidence updates, Agents for environment contracts, and Reasoning Models for verification and inference-time search.