TasteVal Finds AI Can Optimise Experiments Faster—But That Is Not All of Research Taste

AI can write code, launch trials and compare results at a speed no human laboratory can match. The harder question is whether it can decide which experiment deserves to run next—and whether a faster answer is actually a better scientific decision.

A new preprint called TasteVal tries to measure that narrower capability. Its authors call it “experimental research taste”: the ability to design experiments and update decisions from their outcomes when the research problem has already been chosen. The results are striking, but the boundary around the benchmark matters as much as the headline.

What TasteVal is trying to measure

Research taste is often treated as something intangible: knowing which question is worth pursuing, which result is signal rather than noise and when a line of inquiry has become a dead end. TasteVal turns one part of that judgement into a measurable loop.

The evaluated model acts as a Researcher. It proposes an experiment, receives the result, interprets it and chooses what to try next. A separate fixed Coder agent implements the experiment. This separation is designed to stop coding ability from dominating the score.

• Eight newly constructed, open-ended AI research and development tasks are used.
• Tasks cover text and vision, including data curation, model training, fine-tuning, preference modelling and adversarial alignment.
• Each run receives up to 40 serial H100 GPU hours and 120 wall-clock hours.
• Twenty models released between 2023 and 2026 are evaluated with six seeds per model per task.
• Twenty-four human experts provide the comparison baseline, with at least two experts attempting each task.

The tasks themselves are withheld to reduce contamination. That protects future testing, but it also means outside researchers cannot inspect the full benchmark or independently reproduce every task-level judgement from the public paper alone.

The headline result is about efficiency, not wisdom

TasteVal’s main measure is a compute multiplier. If an expert requires 20 GPU hours to reach a score that a model reaches in 10, the model receives a 2× multiplier for that comparison. This is serial experiment compute: time spent learning from one experiment before choosing the next. It is not a claim that half as many GPUs can replace an expert laboratory.

The strongest evaluated model, Opus 5.5, reached a reported compute multiplier of 2.30 relative to the expert baseline, with a wide 95% confidence interval of 1.15 to 4.37. The authors also report that the multiplier for frontier models doubled approximately every three months after December 2025. Yet the final-performance measure—how good the eventual result was after the full budget, ignoring compute used—showed no comparable break and doubled on a much slower estimated 14.6-month trend.

The careful reading is not “AI has become a better scientist than humans”. It is “on eight constrained AI-R&D tasks with fast feedback, the best tested model reached the expert endpoint with less serial experimental compute”.

What the benchmark measures—and what it leaves outside

TasteVal measures	TasteVal does not establish
Choosing the next experiment after a problem is fixed	Choosing which scientific problem is socially or intellectually worth solving
Learning from quick, low-noise validation feedback	Managing ambiguous evidence, slow replication or conflicting objectives
Serial experimental efficiency on one H100	Performance in a large, parallel frontier-training programme
Performance against selected human expert baselines	Superiority to every expert, team or institution
Optimisation under an explicit scoring function	Original theory, scientific significance or safe deployment judgement

This boundary is not a minor footnote. The authors explicitly say the benchmark does not measure problem selection. Its tasks provide clean progress signals and usually fast scoring. Real frontier research can involve uncertain goals, delayed evidence, hardware constraints, security reviews, organisational coordination and results that resist a single numerical score.

Fast optimisation can still be shallow

The paper’s most useful caution concerns novelty. The authors classify model-proposed experiments as tuned, composed, modified or novel. They conclude that the strongest submissions did not rely on genuinely novel ideas or non-obvious cross-domain adaptation. Progress came mainly from better shallow and moderate optimisation: tuning, combining and adapting methods within a well-defined search space.

That can still be economically important. Many research programmes spend substantial resources on deciding which ablation, data mixture or training configuration to test next. An agent that eliminates weak options earlier can save real compute. But efficient hill-climbing is not the same as discovering a new mountain.

Models and experts handled uncertainty differently

TasteVal also reports a difference in how models and humans treated noisy improvements. Strong human runs measured run-to-run spread early and discarded gains smaller than that variation. One leading model rarely repeated an experiment with a different seed and tended to bundle several training runs into one proposal.

This is a practical warning for any AI-assisted laboratory: a system may improve the displayed metric while remaining weak at deciding whether the change is scientifically meaningful. A leaderboard gain without replication, uncertainty estimates or a plausible mechanism should remain a candidate result, not a conclusion.

Cost comparisons need the same discipline

The authors estimate the best model’s average run cost at about $282, compared with $9,123 for the human baseliners when salary and experiment compute are placed on a common accounting basis. This is an informative operational comparison, not a universal wage-replacement ratio. It depends on assumed labour rates, API pricing, task design, the fixed Coder agent and which costs are included.

A complementary METR study of optimisation expenditure found that agents could beat humans at low budgets on a NanoGPT speedrun while still showing modest gains relative to the larger body of human work. METR also found that raw trajectories overstated progress until results were revalidated. Together, the studies suggest that cost-performance curves are more useful than one headline price.

More experiments do not automatically mean more progress

OpenAI’s September 2026 research-acceleration report describes rising agent use, more experiments and more code inside its research organisation. It also says these indicators are easy to measure but hard to interpret, because the least automatable parts of research become the next bottlenecks. High-level planning remained a small share of agent activity, and more than half of successful four-to-eight-hour tasks required at least one human intervention.

That distinction matters beyond frontier laboratories. A small research team can double its experiment count while making no better decisions if it lacks a preregistered question, reliable data, independent review or a stopping rule. Activity is an input. Progress requires validated knowledge.

A practical protocol for AI-assisted experimentation

1. Write the research question before the optimisation target. State what decision the result should inform and who could be affected.
2. Define the smallest meaningful effect. Decide in advance what improvement would justify another run, a scale-up or a change in practice.
3. Separate proposing from approving. Let the AI suggest experiments, but require a named human to approve budget, data access and safety boundaries.
4. Reserve compute for replication. Do not allow the agent to spend the whole budget chasing new variants without repeating promising results.
5. Keep failed experiments visible. Record rejected ideas, seeds, configurations and negative results so the system does not rediscover the same dead ends.
6. Test transfer. Re-run the result under a different dataset, environment or implementation before treating it as robust.
7. Audit the conclusion. Ask whether the written interpretation is supported by the experiment, or whether it quietly extends beyond the measured task.

Questions to ask when a benchmark claims expert-level research

• Was the problem chosen by the system, or supplied in advance?
• Was progress fast and objectively measurable, or slow and contestable?
• Were humans and models given equivalent information, tools and budgets?
• Is the baseline the average expert, the best expert, or a particular recruited group?
• Were results replicated across seeds and independently revalidated?
• Are tasks public enough to audit, yet protected from contamination?
• Does the result show novelty, reliable optimisation, or both?
• Which conclusion comes from evidence, and which comes from extrapolation?

Why the distinction matters

If AI systems become better at selecting experiments, laboratories may receive more useful evidence per unit of compute. That could shorten development cycles and free researchers from repetitive search. It could also accelerate unsafe work if experimental efficiency rises faster than governance, monitoring and human capacity to interpret the results.

The correct response is neither to dismiss the benchmark nor to turn it into a prophecy. TasteVal offers a serious measurement of a previously vague capability. Its result should motivate stronger replication, broader tasks and explicit tests of problem selection, novelty and real-world messiness—not the claim that scientific judgement has been solved.

The research takeaway

AI may be getting markedly better at navigating experiments once the destination and score are visible. Human researchers still carry the harder responsibility of deciding whether the destination matters, whether the evidence is trustworthy and whether the journey should continue at all.