OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software

Abstract

Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires interpreting specialized interfaces, manipulating scientific objects, and producing verifiable results. We thus introduce OSWorld-Science, a benchmark and evaluation environment that combines scientifically meaningful tasks, artifact-based evaluation, and an efficient agent harness for studying computer use in the scientific domain. The benchmark contains 12 VLMs and 146 high-quality tasks across several scientific domains and software configurations, covering workflows such as molecular drawing and retrosynthesis, pathology image analysis, statistical computing, and physical simulation. Tasks are developed through expert proposals and iterative human–AI co-design, with selection guided by scientific value and difficulty. Task-specific execution-based evaluators inspect application states and generated artifacts, including molecular structures, segmentation masks, plots, and numerical results, and award partial credit for incomplete outcomes. Our special harness integrates model adapters, interaction-loop control, and trajectory logging to support comparisons of models and interaction strategies. This combination makes OSWorld-Science a testbed for examining how agents coordinate visual interpretation with graphical and command-line actions. Our results shows that current state-of-the-art VLMs with a strong harness still face challenges in addressing key questions in the scientific domains. We also analyze the benchmarking results across multi-linguistics, reasoning efforts, context length and other factors and derive several important conclusions and directions to assist future development. Overall, we provide an integrated framework connecting expert-defined scientific goals to verifiable software outcomes, enabling systematic evaluation of both agent capabilities and harness design in scientific workflows. We also welcome public contributions now: Link!

Overview of OSWorld-Science in three panels: 1, task and environment (expert proposal, co-worker design, filtering by diversity, difficulty and scientific value, multilingual and multi-domain tasks); 2, harness and agent loop (VM configurations, loop controller, VLM adapter, screenshots and actions); 3, evaluation and analysis (step-wise and binary grading, behavior experiments, discussion). — Overview of OSWorld-Science: expert task design and filtering, the evaluation harness and agent loop, and the grading and behavior analyses.

Score, Tokens and Cost

Twelve models, one run per task at their default reasoning effort. Pick a task set, then compare the mean score with output tokens or cost per task.

X

Y

Partial = mean task score with partial credit; Binary = share of tasks scored 1; VOID runs count as 0. Output tokens include thinking tokens. Cost and output tokens come from the results workbook (its recorded cost, or its input and output tokens at list prices); where it has no usage (Statistics output tokens, radiology, linguistics) they come from the released trajectory records. Kimi K3's Bio cost is its reported total, counted as output tokens. Cost and tokens are averaged over tasks with recorded usage; a model without recorded usage for a set is left off that axis and shown as "--" in the table.

Model	Output tokens / task	Partial score	Binary score

Task Statistics

146 tasks in 7 scientific domains and 14 software configurations, proposed by domain experts and filtered for diversity, difficulty and scientific value.

Interaction Demands

All 146 tasks are observed through screenshots of the application. 122 of them (83.6%) also allow the command line, while 24 (16.4%) are GUI-only (ASKCOS retrosynthesis, PDF reading and Praat). Four software configurations, QuPath, ChemDraw, ASKCOS and SAS/R, carry 85 tasks.

CLI support by task (146 tasks)

Task Domain Distribution

Chemistry and physics carry half of the tasks; medicine, statistics and biology most of the rest. The inner ring is the domain, the outer ring the software each task runs in. Hover a segment or a logo for its count.

Software counts follow the graded task list. The capability tags and operation templates follow the task annotation sheet, which files one structure-recognition task under ASKCOS.

Required Capabilities

Besides screen observation, every task is annotated with the interaction primitives it needs: clicking in 124 tasks, typing in 101, drawing in 21, explicit screenshot capture in 21 and table analysis in 1. Tags are non-exclusive, so the 268 annotations exceed the task count.

Operation Templates

Each task follows one of 10 operation templates that describe what the agent must read and how it must act. Hover a template for its meaning.

Counts are over 146 tasks. Each task carries exactly one template. Two names are lightly reworded from the task annotations, which read “read sound file, text, click, type” and “read pdf, analyze table, out”.
Template	Tasks	Share
Read text, click, typeRead on-screen text or a data file, then drive the application with clicks and typed input (SAS/R, OpenFOAM, ANSYS, CIAO with DS9).	51	34.9%
Read pathology images, click, typeInspect a whole-slide image in QuPath, then annotate, measure or export through the GUI or scripting.	23	15.8%
Read PDF, take screenshot, drawFind a structure in a paper PDF, capture it, and redraw it in ChemDraw.	21	14.4%
Understand molecule, clickReason about a target molecule and navigate ASKCOS retrosynthesis results.	21	14.4%
Read signal file, text, click, typeLoad EEG recordings in EEGLAB, read the accompanying text, and run an analysis with typed parameters.	10	6.8%
Understand molecule, click, typeWork with protein structures or NMR spectra in PyMOL or Mnova, typing identifiers and parameters.	8	5.5%
Read geographic images, click, typeDigitize or analyze raster and vector layers in QGIS.	6	4.1%
Read MRI images, click, typeReview radiology series in Weasis or 3D Slicer and record findings.	3	2.1%
Read sound file, text, clickAcoustic inspection of recordings in Praat, driven by clicks only.	2	1.4%
Read PDF, analyze table, outputExtract a table from a PDF and produce a derived result.	1	0.7%

Leaderboard

Key Findings

Score and Token Budget

X

Y

Higher Scores Do Not Require the Most Tokens

Claude Fable 5.1 leads with a mean score of 73.7% over the 146 tasks while producing about 21K output tokens per task.

Claude Opus 5 reaches 57.9% with about 80K.
GPT-6 Astra (57.4%) and GPT-5.6 sol (50.1%) are the most frugal of the top four, at about 12K and 14K.
Gemini 3.1 Pro spends about 157K output tokens per task for 17.2%.
In dollars, GPT-5.6 luna reaches 22.6% for $0.21 per task, while Kimi K3 costs $10.31 for 30.2%.

Token and cost averages cover all 146 tasks for every model.

1 / 3

Case Studies

10 case studies from the paper, one per application. Each compares runs on the same or a closely related task, shows the paper's figure for it and names what separated the outcomes. The first is the case study of the main text; the others come from the appendix.

Chemistry · Medicinal and natural-product chemistry · PDF/image viewer + terminal (pdftotext, RDKit, Open Babel)

Checks That Share One Assumption

TasksLenacapavir SAR table extraction; Archangiumide structure reconstruction · Main text

SuccessClaude Opus 5Lenacapavir SAR: 17/17 rows correct, score 1.0, 70 steps
FailureClaude Opus 5Archangiumide (unreported run; the scored run reached 1.0 in 43 steps): stereochemistry and InChIKey wrong, graded 0.25, 68 steps

Two five-stage workflow diagrams with raw trajectory frames: the successful Lenacapavir SAR extraction (locate table, render and inspect, decompose, orthogonal checks, 17/17 correct) and a failed, unreported Archangiumide reconstruction (complex drawing, repeated crops, manual reconstruction, self-consistency only, wrong stereochemistry). — Contrasting Claude Opus 5 chemistry workflows with representative raw trajectory frames beside each workflow diagram. (a) and (b): the successful Lenacapavir run moves from source-table localization to focused stereochemical inspection and final validation of the complete 17-row CSV. (c) and (d): a failed, unreported Archangiumide run (the scored run reached 1.0) converts coordinate-guided reconstruction into a chemically valid, internally consistent structure, yet its own round-trip check still reveals a stereochemical mismatch. The decisive difference is independent source-grounded validation rather than trajectory length or the number of checks.

Summary

Two Claude Opus 5 chemistry runs with nearly identical horizons end with opposite outcomes. On the Lenacapavir SAR task the run located Table 2 with pdftotext, rendered the page at 200–1200 dpi, separated the shared scaffold from the 17 variable R¹ substituents, and before writing the CSV cross-checked the assay column and footnotes, parsed every SMILES with RDKit and verified the intended CIP assignments: all 17 rows were correct (score 1.0 after 70 steps). On an unreported Archangiumide run (the scored run reached 1.0 in 43 steps) it spent most of 68 steps on enlarged crops, coordinate grids and hand-mapped coordinates, then validated its own hypothesis by round-tripping the SMILES through CML/Open Babel and recomputing the formula and InChIKey; the graph and formula were correct but the stereochemistry and InChIKey were wrong (graded 0.25), because every downstream conversion reproduced the same inverted stereocenter.

Takeaway

Long trajectories and many validation actions are insufficient when all checks share one latent assumption; orthogonal validation against the source, not self-consistency, separates the two runs.

Physics · X-ray astronomy · CIAO + SAOImage DS9

Where the Number Comes From

TaskBackground-subtracted net counts (netcounts) · Appendix

SuccessGPT-6 AstraReads NET_COUNTS from the output file, score 1.0, 9 steps
FailureGPT-5.6 terraEnters 1427 − BG_ERR = 1410.7212, score 0.0, 44 steps

Two five-stage workflow diagrams with DS9, terminal and spreadsheet frames: GPT-6 Astra reads NET_COUNTS 1424.9204178903 from the output file; GPT-5.6 terra enters 1410.7212 from the visible COUNTS and BG_ERR rows. — Contrasting GPT-6 Astra and GPT-5.6 terra workflows on netcounts, with representative raw trajectory frames beside each workflow diagram. (a) and (b): the successful run reads its regions back from DS9. It then reads NET_COUNTS from the output file named at the bottom of the Net Counts window (green boxes) and writes 1424.9204178903. (c) and (d): the failed run sees the same window with the NET_COUNTS row scrolled out of view. It enters 1427 − BG_ERR = 1410.7212 (red boxes) and later cites its own entry as the correct value. The decisive difference is the origin of the reported number, not the setup, the route, or the trajectory length. Host names in the terminal prompts are masked.

Summary

Both runs define the same two regions, tag the annulus as background and see the same DS9 Net Counts window, in which the NET_COUNTS row is scrolled out of view. GPT-6 Astra reads its regions back with xpaget, reads NET_COUNTS from the FITS table that the analysis wrote (1424.9204178903), writes it to the workbook and asserts the stored value (score 1.0 after 9 steps). GPT-5.6 terra types 1410.7212, which is 1427 minus BG_ERR, the 1σ uncertainty of the background rather than the area-scaled background of 2.08 counts, and later validates the saved cell against itself; the answer is 14.2 counts from the reference against a tolerance of 1.5 (score 0.0 after 44 steps).

Takeaway

A number read from a GUI result window is not sufficient evidence until its definition has been checked; agents should trace each reported number to the tool's full-precision output before committing it.

Physics · Computational fluid dynamics · ANSYS Fluent

A Zero-Valued Observable

TaskDrag adjoint of a cylinder at Re = 40 · Appendix

SuccessKimi K3Selects the wall after reading 0 N; adjoint converges at iteration 27, score 1.0, 86 steps
FailureGPT-5.6 terraKeeps the unselected zone; zero-source adjoint stops at iteration 1, score 0.33, 100 steps

Two five-stage workflow diagrams with Fluent dialog and console frames: Kimi K3 reads 0 N, selects the wall zone and gets 1271.7444 N and 27 adjoint iterations; GPT-5.6 terra keeps the unselected zone and its adjoint stops at iteration 1. — Contrasting two trajectories on the adjoint task FT-008a, with raw frames from each run to the right of its workflow diagram. (a) and (b): Kimi K3 evaluates its drag observable, reads 0 N, selects the wall zone it had left unselected, re-evaluates to 1271.7444 N, and runs the adjoint to convergence at iteration 27 (score 1.0, 86 steps). (c) and (d): GPT-5.6 terra reaches the same dialog with the same 0 N reading, concludes that the observable is correctly configured, obtains 1271.7444 N from the stored drag report without reconciling the two values, and runs a zero-source adjoint that stops at iteration 1 (score 0.33, 100 steps). The decisive difference is whether an implausible value may overrule the agent's reading of its setup, not the trajectory length or the number of checks.

Summary

FT-008a asks for the drag on a cylinder at Re = 40 and for the number of iterations the drag adjoint needs to converge (27). Kimi K3 and GPT-5.6 terra both create the force observable without selecting the wall zone and reach the same screen: the observable dialog with wall unselected above a console line reading “Observable Value [N]: 0”. Kimi K3 lets that reading overturn its belief within one step, selects the wall, re-evaluates to 1271.7444 N and runs the adjoint to convergence at iteration 27 (score 1.0, 86 steps). GPT-5.6 terra calls 0 N “not a credible total cylinder drag” yet concludes that the observable is correctly configured, starts the adjoint without reconciling the two drag values, and reports 1 iteration against the reference 27 (score 0.33, 100 steps).

Takeaway

A check helps only if its result may overrule the agent's reading of its own setup: a physically impossible value, such as zero drag on a bluff body, is evidence against the setup.

Physics · Computational fluid dynamics · OpenFOAM (terminal, foamToVTK)

Correct Physics, Lost Digits

TasksFlat-plate boundary layer, wall-resolved k–ω SST; Turbulent plane channel, wall-resolved k–ω SST · Appendix

SuccessGPT-5.6 solRejects Float32 VTK arrays, exports 12-digit native fields, 4/4 gates, score 1.0, 35 steps
FailureGPT-5.6 solExports 6-digit VTK arrays, declares DONE with 59 steps unused, score 0.125, 41 steps

Two five-stage workflow diagrams with terminal frames: the flat-plate run rejects Float32 arrays and exports 12-digit native fields (EXPORT_OK, 4/4 gates); the plane-channel run exports 6-digit arrays and declares DONE (score 0.125). — Contrasting GPT-5.6 sol OpenFOAM workflows, with representative raw terminal frames beside each workflow diagram. (a) and (b): the successful flat-plate run (OF-012) inspects the array declarations in its VTK (.vtu) output, rejects the Float32 arrays as short of full precision (step 33), reads the 12-digit native field files (step 34), and checks row counts and physical bounds before EXPORT_OK (score 1 after 35 steps). (c) and (d): the failed plane-channel run (OF-004) sees the same kind of declaration (step 37), builds its CSV files from 6-digit ASCII VTK arrays and checks their line counts, array lengths, and header (step 38), and declares DONE although its own volume-weighted bulk velocity differs from U_b in the 8th significant figure (step 41; score 0.125 after 41 steps). The decisive difference is validation of the delivered precision against the stated requirement, not the correctness of the solve or the number of checks. Host names in the terminal prompts are masked.

Summary

Two GPT-5.6 sol runs on wall-resolved k–ω SST tasks both computed the correct converged solution, and both statements require at least 8 printed significant digits in every value. On the flat-plate task (OF-012) the run inspected the array declarations of its VTK output, rejected the Float32 arrays as insufficient for full precision, parsed the 12-digit native fields instead, and passed all four gates (score 1.0 after 35 steps). On the plane-channel task (OF-004) it built its CSV files from foamToVTK -ascii arrays printed with 6 digits, checked line counts, array lengths and the header, and declared DONE with 59 steps unused although its own bulk velocity differed from U_b in the 8th significant figure (score 0.125 after 41 steps).

Takeaway

A correct solve and several checks are insufficient when no check is held against the tolerance it should test; agents should compare the delivered numbers, not only the file shapes, with the stated precision.

Medicine · Digital pathology · QuPath

Install Through the Application

TaskRegister an attention map onto a reference slide (needs the Image Alignment extension) · Appendix

SuccessGPT-6 AstraInstalls via QuPath, confirms the menu entry, score 1.0, 31 steps, 2,568 output tokens
FailureClaude Opus 5Copies the jar into a guessed directory, 6 relaunches, VOID at the 100-step cap, 26,879 output tokens
SuccessClaude Fable 5.1Same shell-copy strategy first, then QuPath's own settings and extension manager; score 1.0 at step 84

Two five-stage workflow diagrams with QuPath desktop frames: GPT-6 Astra installs the extension through QuPath and exports the aligned image (31 steps, score 1.0); Claude Opus 5 copies the jar into a guessed directory and the Analyze menu stays unchanged (100 steps, VOID). — Contrasting workflows on QP_004_T1 (register an attention map onto a reference slide; requires installing the Interactive Image Alignment extension). Top: GPT-6 Astra installs through the application, confirms the menu entry exists, applies the affine transform, and switches to the command finder when menus stall — 31 steps, score 1.0. Bottom: Claude Opus 5 copies the jar into a guessed directory from the shell and relaunches QuPath six times. Panels (c2) and (c3) show the same empty Analyze menu at step 19 and step 98: the disconfirming evidence was available 81 steps before the budget ran out. The run is scored VOID. The decisive difference is verifying the precondition through the interface that owns it, not trajectory length or effort. The figure is reproduced from the paper; its panel (c) counts nine relaunches and says Opus never opens Analyze, while the released trajectory shows six relaunches and several visits to the Analyze menu.

Summary

On QP_004_T1 the Interactive Image Alignment extension must first be installed into QuPath, and only GPT-6 Astra and Claude Fable 5.1 solve the task. GPT-6 Astra hands the downloaded jar to QuPath itself, confirms the new entry in the Analyze menu before doing any science, applies the six affine values and switches to the command finder when a submenu stalls (score 1.0 after 31 steps and 2,568 output tokens). Claude Opus 5 copies the jar into a guessed directory from the shell, relaunches QuPath six times, and never lets the empty Analyze menu it saw at step 19, and again at step 98, overturn its belief; it is scored VOID at the 100-step cap after 26,879 output tokens. Claude Fable 5.1 is the middle case: after eighteen steps of the same shell-copy strategy it types the path into QuPath's own user-directory setting (step 19), later copies the jar into the folder QuPath's extension manager reports, and finishes at step 84.

Takeaway

Agents should verify a prerequisite through the interface that owns it before spending budget on the work that depends on it, and treat a negative check as disconfirming evidence rather than a cue to retry.

Statistics · Biostatistics · SAS Studio / R

Interface Overhead and Task Completion

TaskReusable group-test routine for two growth-study files · Appendix

SuccessGPT-6 AstraLeaves SAS Studio for local R at step 7, score 1.000, 12 steps, 2.8 minutes
FailureKimi K3Correct Wilcoxon output in SAS Studio, no file delivered, score 0.080, 85 steps, 73 minutes

Three strip plots of score, steps used and input tokens per step for 214 shell-mode runs, 10 domain-application runs and 16 GUI-text-editor runs on the statistics tasks. — Outcomes and costs by application type across the statistics runs. Each point represents one run, and the horizontal marker shows the median. Steps and tokens use logarithmic axes. Using SAS Studio or RStudio is associated with more steps but no detectable score difference. Using a GUI text editor is associated with both more steps and lower scores. The two runs that used both a domain application and an editor are assigned to the domain-application group.

The paper's screenshot panels for this case (SAS Studio in a browser session) are not reproduced; the statistics section's outcome chart is shown instead.

Summary

stat_group_tests asks for one reusable, parameterized routine that selects the test from the study design and sample size, applies unchanged to two growth-study files and reports each statistic to ten significant figures; SAS and R are both permitted. GPT-6 Astra signed in to SAS Studio, found it still loading at step 4, inspected the inputs locally and switched to shell mode at step 7, scoring 1.000 in 12 steps and 2.8 minutes. Kimi K3 uploaded both files to SAS Studio, wrote and submitted a SAS macro and obtained the correct Wilcoxon statistics at step 82, but the run ended during a repair after all 85 steps, 73 minutes and 11.64 M input tokens, with no output file (score 0.080).

Takeaway

Statistical correctness and successful delivery are distinct capabilities: interface overhead can turn correct work into a failed outcome, so tasks meant to measure application operation should make that application necessary.

Biology · NMR spectroscopy · Mnova (MestReNova)

Clicks in the Wrong Coordinate Frame

TaskAssign the ¹H spectrum of 6-bromoindole from the raw FID · Appendix

SuccessClaude Opus 5Licenses Mnova through its dialogs and reads the spectrum, score 0.75, 25 steps
FailureGPT-5.6 lunaClicks one point beside the wizard 39 times, VOID at the 100-step cap
In betweenGPT-5.6 solStates the scaling mismatch at step 65, later falls back and ends on the wizard, VOID at the 100-step cap

Two five-stage workflow diagrams with Mnova frames: Claude Opus 5 installs and licenses Mnova and runs the multiplet analysis (25 steps, score 0.75); GPT-5.6 luna clicks beside the Registration Wizard's Install button until the budget runs out (100 steps, VOID). — Contrasting workflows on nmr_6bromoindole_assignment (assign the ¹H spectrum of 6-bromoindole from the raw FID in Mnova; requires accepting Mnova's license agreement, choosing Install in its Registration Wizard and importing a license file). Top: Claude Opus 5 installs Mnova, clicks Accept and Install where they are, and Mnova itself confirms the license (a2) before the run analyzes the spectrum (a3) — 25 steps, score 0.75. Bottom: GPT-5.6 luna clicks one point beside the Registration Wizard 39 times; scaled by 1.41, the point is the Install button (red cross: the click; green circle: Install). Panels (c1) and (c2) show the same unchanged wizard at step 50 and step 96; the run turned to the keyboard only at step 97, three steps before the budget ran out. The run is scored VOID. The decisive difference is whether the clicks land in the screen's coordinate frame, not knowledge of NMR. Institution names in the license file names and the license host ID are masked.

Summary

Mnova opens the Bruker FID only after the agent accepts its license agreement, chooses Install in the Registration Wizard and imports a license file; eight of the twelve models end VOID, and only four deliver an assignment. Claude Opus 5 clicks each button where it is, Mnova confirms the license, and the run replaces its recalled answer with the automatic multiplet analysis (score 0.75 after 25 steps and 5,575 output tokens). GPT-5.6 luna clicks within two pixels of one point beside the wizard 39 times in steps 50–97; scaled by 1.41, the ratio between the 1920×1080 screen and the 1365×768 image the model sees, that point is exactly the Install button. It reaches Install from the keyboard only at step 97 and is scored VOID at the 100-step cap after 10,282 output tokens.

Takeaway

The failure is at the interaction layer, not in NMR: what separates the runs is whether clicks land in the screen's coordinate frame and whether an unchanged dialog is allowed to change the agent's coordinates.

Biology · EEG · EEGLAB (command line)

Trial Alignment and an Event-Pairing Audit

TasksRT–P3 trial-aligned inference; Forensic event-pairing audit · Appendix

Partial creditClaude Opus 5RT–P3: trial accounting and effect estimates agree with the reference, score 0.945
Partial creditClaude Opus 5Audit: correct mapping, but 6 surplus assignments reported where 12 rows are affected, score 0.77

EEG case figure: epoch and analysis windows; mean Pz amplitudes for 36 fast and 36 slow trials (difference 0.93 microvolts); event timing showing invalid pairings; six shared response events affecting 12 rows versus 6 reported; terminal output on the right. — EEG analysis and event-pairing audit by Claude Opus 5 on two tasks. (a) Epoch and analysis windows for trial-aligned RT–P3 inference. (b) Mean Pz amplitudes over 300–500 ms for 36 fast and 36 slow trials; horizontal bars indicate group means. (c) Actual event timing for the first four stimuli in the forensic task. The candidate crosses a trial boundary and reuses a response preceding a later stimulus; the corrected mapping leaves unmatched stimuli unanswered. (d) Six shared response events affect 12 stimulus rows, whereas the agent reports six surplus assignments. Right: original terminal observations with relevant output highlighted.

Summary

On the RT–P3 task, Claude Opus 5 preserved the alignment of stimuli, responses and EEG epochs through preprocessing and artifact rejection, retained 72 clean responded trials split equally into fast and slow groups, and agreed with the reference on trial accounting and core effect estimates (slow–fast difference 0.93 µV, bootstrap 95% CI spanning zero). On a separate forensic task, the same model correctly reconstructed the stimulus–response mapping but incompletely described the candidate script's errors: six response events were each assigned to two stimuli, affecting 12 rows, and the model reported the six surplus assignments as the number of reused-response rows.

Takeaway

Reconstructing the right mapping is not the same as auditing it completely: the model counted six surplus assignments where twelve stimulus rows were affected.

Geographic information · Remote sensing · QGIS

Finished-Looking Maps, Incomplete Coverage

TasksBuilding footprint digitization; Road centerline digitization · Appendix

SuccessGPT-6 AstraRoads: one script, score 0.98, 4 steps
SuccessClaude Opus 5Roads: measures and repairs gaps, score 0.90, 88 steps
FailureClaude Fable 5.1Roads: five segments, part of the network missing, score 0.76, 10 steps
FailureGPT-6 AstraFootprints: 10 of 12 buildings, including 6 sheds; misses the two smallest
FailureClaude Opus 5Footprints: mainly the large buildings, reported as complete
FailureClaude Fable 5.1Footprints: mainly the large buildings, reported as complete

Aerial imagery with delivered geometries in yellow: building footprints (top row) and road centerlines (bottom row) for the input image, Claude Opus 5, Claude Fable 5.1 and GPT-6 Astra. — Delivered geometries (yellow) on two digitization tasks. Columns from left to right: the input image, Claude Opus 5, Claude Fable 5.1, and GPT-6 Astra. Top row: building footprints (GEO_004). The scene contains 4 large buildings and 8 small sheds beside the parking lots, and only GPT-6 Astra digitizes most of the sheds. Bottom row: road centerlines (GEO_005). Claude Opus 5 and GPT-6 Astra cover the full network, and Claude Fable 5.1 leaves part of it out.

Summary

On road-centerline digitization (GEO_005), GPT-6 Astra writes the network with one script in 4 steps and scores 0.98, Claude Opus 5 measures the road positions from the image and repairs gaps over 88 steps for 0.90, and Claude Fable 5.1 draws five correct segments in 10 steps, reports the task as complete and scores 0.76 because part of the network is missing. On building-footprint digitization (GEO_004) the scene contains 4 large buildings and 8 small sheds, and eight runs declare the task done after part of the work: seven of them digitize mainly the large buildings, while GPT-6 Astra digitizes 10 of the 12 buildings, including 6 sheds, and misses the two smallest. None of the nine runs that deliver footprints on GEO_004 checks whether all buildings are covered.

Takeaway

Trajectory length says little about method, and a check that the output exists and has the expected structure does not reveal a map that misses part of the scene.

Linguistics · Phonetics · Praat

Acoustic Landmark versus Interface Control

TaskMark the release burst of a prevoiced word (tolerance 5 ms) · Appendix

SuccessGPT-6 AstraRejects the transient, 3.1 ms after the reference, score 1.000, 9 steps
FailureClaude Opus 5Precise control, wrong event: 5.5 ms before the reference, score 0.10, 41 steps
FailureClaude Sonnet 5Right description, lost time axis: 51.7 ms late, score 0.10, 93 steps

Three rows, each pairing a Praat editor view with an enlarged waveform: GPT-6 Astra's boundary just after the reference release, Claude Opus 5's boundary on the earlier transient, and Claude Sonnet 5's view that starts after the release. — Three runs locating the release of bun in the neg task. Each row pairs the run's Praat view with an enlargement of the boxed waveform region. Where visible, the black line marks the reference release, the red line marks the inserted boundary, and the gray triangle marks the earlier transient within the prevoicing. (a) GPT-6 Astra rejects the transient and places the boundary 3.1 ms after the reference, within tolerance. Its score is 1.000. (b) Claude Opus 5 inspects a 16 ms window through the Select dialog but marks the transient 5.5 ms before the reference. Its score is 0.10. (c) The window viewed by Claude Sonnet 5 begins 6 ms after the reference. The run treats the edge of its pink selection as the release and places the boundary 51.7 ms late. Its score is 0.10.

Summary

The task asks for the release burst of a prevoiced word within 5 ms, and a brief broadband transient 5.5 ms before the true release is a distractor. GPT-6 Astra rejects the transient after a single zoom and places the boundary 3.1 ms after the reference, finishing in 9 steps (score 1.000). Claude Opus 5 controls the time axis precisely, opening Praat's Select dialog 11 times and inspecting windows as narrow as 10 ms, but marks the transient 5.5 ms before the reference (score 0.10). Claude Sonnet 5 describes the target correctly but issues 63 zoom commands across 93 steps, loses the release after pressing Praat's in button, and places the boundary at the edge of its own selection, 51.7 ms late (score 0.10).

Takeaway

Placing a boundary through a graphical editor takes two separable skills, identifying the intended acoustic landmark and controlling the interface precisely enough to mark it; GPT-6 Astra showed both, Claude Opus 5 only the second and Claude Sonnet 5 only the first.

Failures and Successes, Side by Side

Five tasks, five applications. For each, one run that earned full credit and one that did not, from the same task package, the same starting state and the same grader. Frames are the screenshots the model saw at that step; captions quote the model's own plan where it wrote one.

QuPath 0.5.1 · Medicine · Digital pathology · Task QP_001_T1-2

One intermediate-size nucleus ellipse inside ROI-A

TaskA breast-resection whole-slide image is open in a QuPath project that has one rectangular annotation, ROI-A. The agent inspects the clearly separable nuclei in ROI-A and draws exactly one roughly circular ellipse of intermediate nuclear size, completely inside ROI-A. It then classifies the ellipse as “Medium Cell Size”, leaves ROI-A unchanged and saves the project. The grader reads the saved project, checks that ROI-A is unchanged and that there is exactly one ellipse, then checks the ellipse's class, shape, position and diameter.

Successful run

Claude Fable 5.1

PASSScore 1.00

27 of 100 steps · GUI share 96%

One 19 × 19 px ellipse, classified Medium Cell Size, inside ROI-A (allowed diameter 10.2–27.0 px).

Claude Fable 5.1, screenshot before its action at step 8 — Step 8 *· 1 / 7*Fable first inspects the project files from a terminal and launches QuPath from it (steps 1–4). The slide is now open at 0.02×, with ROI-A labeled in the middle of the tumor.

1 / 7

Model narrationShort plans before most actions (steps 21, 22 and 26 are code only).

Failed run

GPT-5.6 luna

FAIL · wrong contentScore 0.00

37 of 100 steps · GUI share 72%

Grader: “'Medium Cell Size' is not assigned to an EllipseROI”.

GPT-5.6 luna, screenshot before its action at step 7 — Step 7 *· 1 / 6*Luna launches QuPath from a terminal and opens the project through the file chooser (steps 1–6). The slide appears at 0.02×, with ROI-A labeled in the center.

1 / 6

Model narrationTwo short code comments (steps 7–8); every other response is code only, so these captions describe the logged actions and the screen.

Analysis

Both runs reach the same QuPath panels and both type the string “Medium Cell Size”; the difference lies in where that string ends up and at what scale the ellipse is drawn. Fable zoomed until all of ROI-A was in view (2.65×) and drew a nucleus-sized 19 × 19-px circle inside it. It created the class with Add class, applied it with Set selected, and confirmed the Classification row before saving. Luna drew at 0.08×, where a 25-pixel screen drag produced a 311 × 311-px ellipse (15,844 µm²), larger than ROI-A (240 × 240 px) and about 1.2 mm away from it. It then wrote the label into the annotation's Name field, never created the class, and kept pressing Set selected with nothing to apply. The grader finds ROI-A unchanged and exactly one ellipse, then stops at the class check: no ellipse carries the classification, so size and position are never scored.

Takeaway

In QuPath a name is not a classification, and a 25-pixel drag at 0.08× is not nucleus-sized. The successful run worked at the scale of the object and read back the exact field the grader checks.

ANSYS Fluent 2026 R1 · Physics · Computational fluid dynamics · Task FV-001a

Laminar channel flow: whole-wall shear and peak x-velocity

TaskSet up and solve steady laminar flow between two parallel plates (Re = 100) in ANSYS Fluent from a supplied ten-cell-high 2-D mesh. The specification is serial, double precision, laminar, ρ = 1 kg/m³ and μ = 0.003 Pa·s, with a uniform 1 m/s inlet, a 0 Pa outlet, the coupled solver and absolute residual criteria of 1e-10. The reference run stops itself at iteration 104. Two numbers go into report.json: the area-weighted average x wall shear over the whole wall (5/9 of the score) and the maximum x-velocity in the fluid (4/9). Each must be within 0.1% of the reference.

Successful run

GPT-5.6 sol

PASSScore 1.00

75 of 100 steps · GUI share 95%

tau_wall_x = 0.05894787 Pa and u_max = 1.4708195 m/s, both within 2e-8 relative of the reference.

GPT-5.6 sol, screenshot before its action at step 3 — Step 3 *· 1 / 7*The Fluent Launcher shows one solver process and 3D selected. Sol switches to 2D, ticks Double Precision and starts the solver.

1 / 7

Model narrationOne-line plans as code comments until the reports; steps 68–74 are code only.

Failed run

GPT-5.6 luna

FAIL · wrong contentScore 0.00

99 of 100 steps · GUI share 85%

tau_wall_x = 0.659303 Pa (reference 0.05895) and u_max = 1.6178159 m/s (reference 1.47082); both keys fail.

GPT-5.6 luna, screenshot before its action at step 23 — Step 23 *· 1 / 7*After several attempts (steps 12–22), the material editor shows the required density of 1 and viscosity of 0.003.

1 / 7

Model narrationShort code comments on most steps up to step 69, none afterwards.

Analysis

Both runs set up the same case through the same Fluent panels and ask for the same two report types (Sol through console commands, Luna through the report dialogs). Sol set the three residual criteria to 1e-10. The solve then stopped exactly where the task said it would (iteration 104, continuity 9.77e-11), and its two reports matched the reference in every printed digit (within 2e-8, relative). Luna never reached the residual monitor, so its solve stopped at iteration 40 on the default 1e-3 criterion. That alone does not explain its numbers: Fluent reported 0.659 Pa and 16.18 m/s, both about 11 times the reference, in a channel fed at 1 m/s. No screenshot shows the inlet value after Luna retyped it (steps 32, 33 and 37). The console also labels the second number Velocity Magnitude, not x-velocity. Instead of revisiting the setup, Luna spent 30 steps (69–98) rewriting report.json and shifting decimal points, and it ended with both values wrong.

Takeaway

A solver that says “converged” has not checked the answer. The successful run set the stopping criterion the task specified, and its solve stopped at the iteration the task predicted; the failed run treated a 16 m/s peak in a channel fed at 1 m/s as a formatting problem and moved on.

QGIS 3.44 · Geoscience · Remote sensing · Task rareplanes-aircraft-point-annotation

Mark every aircraft with exactly one point

TaskAn aerial image of an airfield is open in QGIS with an empty point layer, aircraft_points. The agent marks every fully visible aircraft with exactly one point, including parked or stored aircraft. It then uses QGIS to export the layer to a GeoPackage with one layer of 2-D points in EPSG:4326. The reference contains 15 aircraft. The run passes only if each aircraft carries exactly one point and there are no extra points.

Successful run

GPT-5.6 terra

PASSScore 1.00

11 of 100 steps · GUI share 100%

15 points exported: 15 true positives, 0 false positives, 0 missed (F1 1.0).

The released task file lists 60 steps; these runs used the 100-step limit applied to all geoscience runs.

GPT-5.6 terra, screenshot before its action at step 2 — Step 2 *· 1 / 7*Editing is switched on (step 1). In one action Terra presses Ctrl+. (Add Point Feature) and clicks 15 times, once per aircraft, with every position read from this screenshot.

1 / 7

Model narrationNone: the run logs code only, so these captions describe the logged actions and the screen.

Failed run

Claude Sonnet 5

FAIL · wrong contentScore 0.97

39 of 100 steps · GUI share 95%

16 points exported: 15 true positives plus 1 extra point (F1 0.968; a pass needs an exact match).

Continuous F1, as in the results table; the task passes only at F1 = 1, so the run's verdict is FAIL (binary score 0).

The released task file lists 60 steps; these runs used the 100-step limit applied to all geoscience runs.

Claude Sonnet 5, screenshot before its action at step 2 — Step 2 *· 1 / 7*Sonnet sees the whole scene: “I can count approximately 15 aircraft in this parking area.”

1 / 7

Model narrationPlans and self-checks before every action.

Analysis

Both runs find the same aircraft and export a valid EPSG:4326 GeoPackage from inside QGIS (Sonnet through the toolbar's “Export aircraft points” button, Terra also through Save Features As). Terra placed all 15 points in one action, reading every position from a single screenshot. Sonnet placed them in five batches and clicked (1434, 550) twice, as “Aircraft 9” and again as “Aircraft 12”, which left 16 points. The duplicate cannot be seen on the canvas, but Sonnet had two signals of it: its own comments run up to “Aircraft 16”, and ogrinfo prints Feature Count: 16. Having estimated “approximately 15” at the start, it still accepted 16 as matching the image, so its verification confirmed the file's format but never tested the count against its own estimate.

Takeaway

Verification helps only when the result is compared with something. Sonnet ran the right check and then explained away the one number that contradicted its own count.

Praat · Linguistics · Phonetics · Task praat_vot_plosive1

Word-initial voice onset time for bat, Pat, bit and pit

TaskIn Praat's TextGrid editor, the agent annotates word-initial positive voice onset time (VOT) for bat, Pat, bit and pit in Plosive1.wav. Each interval runs from the supplied burst-release boundary to the onset of sustained periodic voicing (the first regular glottal cycle, not isolated transients) and is labeled VOT. The annotation must be done through the GUI and saved as a text TextGrid. Each word's start, end and duration must lie within 2 ms (bat, bit) or 5 ms (Pat, pit) of a private reference.

Successful run

GPT-6 Astra

PASSScore 1.00

27 of 100 steps · GUI share 96%

VOTs of 10.2, 85.5, 12.9 and 102.5 ms (bat, Pat, bit, pit); all four words pass.

GPT-6 Astra, screenshot before its action at step 2 — Step 2 *· 1 / 7*The maximized TextGrid editor shows the four words, and both tiers share the supplied boundaries. Astra hides the IPA palette, which widens the time axis, and zooms into bat on its second try (step 3).

1 / 7

Model narrationOne-line plans as code comments.

Failed run

Claude Fable 5.1

FAIL · wrong contentScore 0.55

47 of 100 steps · GUI share 100%

bat and bit end 4.1 and 4.4 ms late (tolerance 2 ms); Pat and pit pass. Score 0.55.

Claude Fable 5.1, screenshot before its action at step 6 — Step 6 *· 1 / 7*Bat in a 38-ms view. Fable puts the onset at the rising edge of the pulse “peaking near x≈565”, which it takes to be the first regular ≈108 Hz cycle.

1 / 7

Model narrationPlans and measurements before every action.

Analysis

Both runs work in the same waveform-and-spectrogram view and keep the supplied release boundaries, so the only difference is where each places voicing onset. Astra placed every boundary in a 60–76 ms window, where the periodic train is visible as a whole, at the first cycle of that train. Fable also looked at each word in a 43–131 ms view, but placed its boundaries after zooming further, to 38 ms (bat), 10.8 ms (Pat), 10.2 ms (bit) and 34.5 ms (pit); the two narrowest show only a cycle or two. In every word it waited for a larger pulse: in bat it called the stretch before it “a low-amplitude irregular region” already in a 72-ms view, and in Pat, bit and pit it set aside a first, smaller pulse as a transient. All four of its VOTs came out 4.1–5.4 ms longer than Astra's. In bit the zoom decided the outcome: at 43.5 ms Fable put the onset at “~5.879 s”, next to Astra's 5.8786 s, and at 10.2 ms it moved it to 5.8831 s. Against the private reference, bat and bit end 4.1 and 4.4 ms late, beyond their 2-ms tolerance, while Pat and pit stay within 5 ms. Fable's interface slips cost steps, not points.

Takeaway

More zoom is not more precision. In a 43-ms view Fable read bit's voicing onset where Astra did; at a 10-ms zoom it took that first, smaller cycle for an isolated pulse and ended 4.4 ms late. In bat it waited for a larger pulse even at 72 ms.

SAOImage DS9 with CIAO (DAX) · Physics · X-ray astronomy · Task chandra-ds9-lightcurve-4479-01

Background-subtracted light curve of a Chandra source

TaskThe level-2 event file of Chandra ObsID 4479 is loaded in DS9. With a given source circle and background annulus, the agent measures the source's background-subtracted count rate in 5000-s time bins. It reports the time offset (dt) of the peak bin, the dt of the first bin above 0.015 counts/s, the peak rate to four significant figures, and the highest rate with 1000-s bins. The answers go into answer.xlsx, and DS9 must stay open with both regions. Each answer is worth a quarter of the score.

Successful run

Kimi K3

PASSScore 1.00

77 of 150 steps · GUI share 93%

45000, 35000, 0.05475 and 0.06862: all four answers pass.

Kimi K3, screenshot before its action at step 38 — Step 38 *· 1 / 7*Earlier attempts had the radius fields revert (steps 32–37); now “Generate worked — the Radius list now shows 67.073171 and 105.69106.” The annulus is centered on the same physical position as the source circle, (3442.9172, 3711.2736).

1 / 7

Model narrationPlans on about half of the steps; the other responses are code only.

Failed run

GPT-5.6 terra

FAIL · wrong contentScore 0.50

42 of 100 steps · GUI share 76%

45000 and 35000 pass; 0.05406 and 0.06775 are both 1.3% low. Score 0.5.

The task allows 150 steps; this model was run with a 100-step limit, and it stopped on its own at step 42.

GPT-5.6 terra, screenshot before its action at step 10 — Step 10 *· 1 / 6*Terra writes both regions to a region file from a terminal (step 8) and loads it with Region > Open. The source circle and the dashed background annulus appear.

1 / 6

Model narrationNone: the run logs code only, so these captions describe the logged actions and the screen.

Analysis

Both runs found the same peak bin (dt = 45,000 s) and the same first bin above 0.015 counts/s (35,000 s). Only the rate values differ. Kimi drew both regions through DS9's dialogs, marked the annulus as background and ran the DAX Lightcurve (Time) tool twice, then read NET_RATE from the tool's output. Terra spent 21 steps in the DAX menus without running that tool. It rebuilt the light curve in a terminal from generic dmextract time histograms, dividing background-subtracted counts by the nominal bin width and passing over dmextract's hint to use opt=ltc1. Both of its rates come out at 0.987 times Kimi's (0.05406/0.05475 and 0.06775/0.06862). That factor is the observation's dead-time correction (DTCOR = 0.98734 in the event file's header), which the light-curve tool folds into each bin's exposure, so the four-significant-figure answers miss.

Takeaway

Rebuilding a standard tool by hand is a quiet risk. The bins and the peak were right, but the dead-time correction the GUI tool applies was lost. The only warning was dmextract's hint to use opt=ltc1, printed after each of Terra's four dmextract calls, and Terra passed over it.

A step is one model call and the action it returned; the frame shown for step N is the screenshot the model saw before that action. GUI share is the fraction of the run's action steps that operate the application GUI. Score is the grader score in [0, 1]; PASS means full credit.

Trajectory Showcase

Ten tasks in five applications (QuPath, ANSYS Fluent, SAOImage DS9, Praat and QGIS), one fully credited run each, from seven models. Every frame is the agent's own screenshot; open the task showcase to step through a full run.

Task showcase

Citation

@misc{dai2026osworldscience,
  title  = {{OSWorld-Science}: A Benchmark of Computer Use Agents for Learning and Using Scientific Software},
  author = {Dai, Dingyuan and Qi, Heli and Liu, Lei and Li, Yinxi and Chen, Baiding and Dou, Zijun and Zeng, Qingcheng and Kang, Qi and Sun, Oliver and Wang, Eric and Zhou, Bo and Wang, Haixin and Du, Yufan and Bo, Shi and Lin, Ruihan and Yuan, Mengqi and Lu, Dunjie and Dillmann, Steven and Shi, Yiming and Su, Tina and Xin, Xin and Liu, Minghao and Wang, Xi and Huang, Xu and Zhang, Ge and Nie, Pengyu and Yang, Zhen and Tang, Jie and Li, Juanzi and Xuan, Weihao and Liu, Tianyu},
  year   = {2026},
  note   = {Preprint},
  url    = {https://github.com/larryinx/OSWorld-Science}
}

OSWorld-Science A Benchmark of Computer Use Agents for Learning and Using Scientific Software

Abstract

Score, Tokens and Cost

Task Statistics

Interaction Demands

Task Domain Distribution

Required Capabilities

Operation Templates

Leaderboard

Key Findings

Higher Scores Do Not Require the Most Tokens

Difficulty and Rankings Depend on the Domain

Most Failures Leave Nothing to Grade

Case Studies

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Summary

Failures and Successes, Side by Side

Claude Fable 5.1

GPT-5.6 luna

Analysis

GPT-5.6 sol

GPT-5.6 luna

Analysis

GPT-5.6 terra

Claude Sonnet 5

Analysis

GPT-6 Astra

Claude Fable 5.1

Analysis

Kimi K3

GPT-5.6 terra

Analysis

Trajectory Showcase

Citation