An AI agent can open a scientific application, move through menus and produce a file that looks finished. That is not the same as producing a scientifically correct result. A polished plot can encode the wrong parameters; a molecular drawing can be chemically invalid; a segmentation mask can outline the wrong tissue; and a simulation can complete without answering the question that was asked.
This distinction sits at the centre of OSWorld-Science, a 62-page preprint submitted on 30 September 2026. The research team built a benchmark for computer-using AI agents working with specialist scientific software. It is an ambitious test of whether agents can move beyond answering questions and carry out research workflows. Its most useful lesson, however, is methodological: evaluate the artefact, not the performance of activity.
What OSWorld-Science actually tests
The benchmark contains 146 tasks spanning seven scientific domains and 14 recorded software configurations. The work includes molecular drawing and synthesis planning, pathology and radiology image analysis, statistical computing, fluid simulation, geographic information, linguistics and neuroscience. Twelve visual-language models were evaluated through a shared agent harness.
This is not a text-only question set. Every task requires the agent to observe scientific software through screenshots. Most require clicking or typing, and many permit a terminal as well as the graphical interface. A run may involve loading data, choosing settings, entering parameters, manipulating a structured scientific object and exporting a result.
The benchmark then checks the resulting state and files. Its task-specific evaluators inspect outputs such as molecular structures, segmentation masks, plots and numerical results. They can award partial credit when only part of the required outcome is present. That is an important improvement over grading an agent by whether it reached a particular screen or announced that it had finished.
A visible workflow can conceal an invisible error
Scientific software contains more state than a screenshot can show. A graph may look plausible while using the wrong column. An image viewer may display the correct scan with the wrong coordinate system. A simulation may run successfully after a unit conversion was omitted. Even the correct numerical value can be misleading if it was produced from an incorrect method.
The paper formalises this problem by separating the agent’s observations from the underlying environment state. The agent sees instructions, screenshots and recent interaction history, while the evaluator can inspect latent state and generated files. This matters because visual progress is weak evidence when the research claim depends on data provenance, parameters, transformations or exact artefact contents.
The overall score hides a jagged capability profile
The paper reports a highest mean task-specific score of 73.7% under its evaluation settings. That number includes partial credit and counts runs that ended without a valid outcome as zero; it is not a binary claim that nearly three quarters of tasks were completed perfectly. The authors also report large variations by domain.
For example, the leading models were much stronger on some biology and statistics subsets than on chemistry. One model that performed well overall scored about 6% in chemistry while reaching 100% on the two linguistics tasks. Two examples cannot establish a general linguistic capability, and a strong aggregate cannot erase a weak chemistry result. The appropriate conclusion is not that one agent is “best at science”, but that performance depends heavily on the scientific domain, application and workflow.
This fits the wider picture in Stanford HAI’s 2026 AI Index technical-performance chapter. Capability is advancing quickly, yet model performance remains jagged across tasks and benchmarks. A headline score compresses that irregular surface into one number.
Most failures were not confidently wrong answers
The trajectory analysis is especially revealing. Across roughly 1,500 released runs, 402 received full credit. Among the remaining analysed runs, the most common failure was not a finished but incorrect artefact. It was no graded artefact at all: 667 runs ended without the required deliverable, and 613 of those exhausted the interaction budget.
That changes how failure should be monitored. A system that returns nothing, leaves an empty template or stops after 100 actions poses a different operational risk from one that produces a plausible but scientifically wrong result. Both matter, but they require different controls.
| Failure mode | What it can look like | What to verify |
|---|---|---|
| No deliverable | The agent keeps working, times out or declares completion without the required file | Required file exists, opens and contains the expected object |
| Invalid artefact | A file is present but malformed, unreadable or structurally invalid | Schema, file integrity and application-level validation |
| Wrong content | The output looks complete but encodes the wrong data, parameters or result | Inputs, transformations, units, reference values and method |
| Partial completion | Some steps or outputs are correct, but the scientific goal is not satisfied | Explicit acceptance criteria for every required component |
More reasoning is not automatically more reliable
A tempting response to failure is to increase the model’s reasoning effort or context window. The study’s controlled ablations do not support treating either change as a universal fix. On a 23-task pathology subset, none of the tested non-default settings was significantly different from its default after correction for multiple comparisons.
Higher reasoning effort sometimes solved tasks that lower settings missed, but it also caused regressions. In one reported pattern, a task completed by a short script at medium effort became a 100-step graphical exploration at higher effort. A longer history window shortened some trajectories and rescued two long graphical tasks, yet a still larger window lost ground on other tasks.
The practical point is not that reasoning or memory never helps. It is that configuration changes must be tested against the exact workflow, with cost and failure modes recorded. “Use more reasoning” is a hypothesis, not a validation method.
The agent is more than the model
OSWorld-Science also treats the surrounding harness as part of the experiment. The harness determines what screenshots and history the model receives, how its commands are executed, how long it may continue, what happens after an API failure and which tools it can call. Two systems using the same model can therefore behave differently.
The earlier OSWorld benchmark made a similar point for general desktop tasks by providing controlled starting states and execution-based evaluators. Its maintainers later repaired reported examples and introduced OSWorld-Verified, a reminder that the benchmark itself is infrastructure that must be maintained.
When an organisation evaluates a scientific agent, it should record at least the model version, prompt, tool permissions, software version, initial state, step and token limits, retry policy, data files and evaluator version. Without those details, a score is difficult to reproduce and impossible to diagnose.
A practical artefact-first evaluation
A small research team does not need to reproduce a 146-task academic benchmark. It can borrow the benchmark’s logic and build a narrow, high-value test around its own work.
- Choose real tasks. Select workflows that matter to a decision, publication or experiment rather than generic demonstrations.
- Freeze the starting state. Use known input files, documented software versions and controlled settings.
- Define the required artefacts. State exactly which files, plots, tables, masks, structures or numerical values must be delivered.
- Build independent checks. Validate structure, units, parameters and reference values outside the agent’s own narration.
- Separate failure classes. Record no deliverable, invalid file, wrong result, partial result and full success independently.
- Repeat the run. One success cannot reveal variability. Use several runs when the workflow is stochastic.
- Measure complete cost. Include model usage, retries, software licences, human review and the time required to repair failures.
- Keep an audit trail. Store prompts, actions, generated artefacts, evaluator outputs and human overrides.
Illustrative example: validating an automated statistical workflow
Consider a hypothetical agent asked to import a dataset, fit a regression model and export a report. A weak evaluation might check whether a PDF appears. An artefact-first evaluation would inspect the data columns used, treatment of missing values, model formula, coefficient table, diagnostics, confidence intervals, exported code and any transformation applied to the source data.
The team might score the task in components: 20% for correct data import, 20% for the specified model, 20% for diagnostics, 20% for numerically correct outputs and 20% for a reproducible export. This is an illustrative scoring scheme, not a validated standard. Its value is that it makes partial completion visible and prevents a polished document from receiving full credit when the underlying analysis is wrong.
What the study cannot establish
OSWorld-Science is a preprint, not peer-reviewed evidence of clinical or scientific effectiveness. Its task distribution is uneven: chemistry contributes 43 tasks while linguistics contributes only two. The paper evaluates particular model-and-harness combinations under defined budgets, so its rankings should not be treated as permanent product ratings.
The authors release the question sets, container images, code and many results, and say that numerical figures are generated programmatically from archived evaluation files. However, they do not publish every trajectory in this version because some tested software requires licences. Reproducibility is therefore substantial but not complete for every component.
Nor does success on these benchmark tasks prove that an agent can design a valid study, interpret ambiguous evidence, notice an unstated confounder or accept responsibility for a scientific conclusion. Operating software is one layer of research competence.
The deeper lesson: verify outcomes, not motion
Computer-using agents will make it easier to automate specialist software. That can widen access and reduce repetitive work. It also creates a seductive failure mode: movement through a complex interface looks like progress, especially when the agent narrates its plan confidently.
OSWorld-Science offers a better standard. Define the scientific state that should exist at the end, inspect the artefact directly and classify how the run failed when it did. The most trustworthy agent will not be the one that clicks fastest. It will be the system whose outputs can be independently reproduced, checked and rejected when the evidence does not hold.
Sources and further reading
- OSWorld-Science preprint and version history
- OSWorld-Science full HTML paper
- OSWorld-Science project page
- OSWorld benchmark and verified-evaluation documentation
- Stanford HAI 2026 AI Index: Technical Performance
Discover more from Marychuks.com AI, Psychology, Business & CreativeVerse
Subscribe to get the latest posts sent to your email.
