Skip to main content

WorkspaceBench: Measuring Agents That Operate OpenBB Workspace

· 28 min read

A few weeks ago at NY Tech Week I gave a talk on centralizing dashboards and agentic workflows in OpenBB Workspace. At the end I talked about how I'm seeing more analysts and quants using Claude Code and Codex to drive their work - and so we need to adapt the workspace to make it agent first.

And this is why we had just announced Workspace MCP, which enables agents to work with the workspace by reading data, building dashboards, and assembling whole apps.

If agents are going to drive this workspace for real analysts, we (OpenBB) need to find a way to measure how well they can navigate our software. And by navigate I mean being able to inspect data, call right tools, change widget parameters, build a dashboard, add new widgets on the dashboard, leverage skills, etc..

This has lead me to build WorkspaceBench.

It includes the harness, a deterministic workspace simulator, 300 different synthethically generated scenarios, graders, and everything else. The Stark Industries mock enterprise apps, which I used for the demo above, is where the benchmark's scenarios come from — the same widgets and apps, turned into tasks.

Why a benchmark​

The workspace was built on a lot of feedback from users (and a few millions in R&D). An agent driving it over Workspace MCP leaves (kinda) similar traces to a person and acts on the same durable state - the dashboards, widgets, and apps.

But the feedback loop is different.

Humans drive with a mouse, and you can infer a lot from it: hesitation, rage clicks, even aha moments. When I see how a user utilizes the workspace, I can immediately spot paper cuts or ways to improve the product, without the user even raising these. And then I can always ask the user directly about their experience - well, not always, but often!

An agent doesn't give me this. It won't hover around like a human would and I can't ask that agent what it was trying to do at that time. Although, the actions (via MCP tools) are legible, they state exactly what it's doing, but without knowing what it was trying to accomplish, I can't tell the overall goal.

A benchmark fixes that half: I wrote the task, so intent is known by construction. And once intent is pinned down, I can run multiple rollouts (from different models) to understand how they get that task done; and as a byproduct where they succeed or fail.

There are many benches in the wild. Some relevant ones:

Each of these allows labs to represent via a single number how good their model is at that specific domain. (Debatable if this is a good thing or not, particularly with test-time scaling)

With WorkspaceBench I'm trying to assess how a model can operate our financial workspace with the MCP tools we offer.

A few of the things I'm looking to get out of this:

  • Finding workspace bugs. Our own test suite catches most of those already, so I don't expect to run into many here - but you never know.
  • Where the tool surface can be improved — the schemas, names, and error messages they models trip on. This is the biggest lever, because fixing the MCP surface once makes every agent better at the same time. At the same time you don't want to overfit the tools to a small 8b model, as this can impact performance of a bigger model.
  • Which models I can honestly recommend. When a user asks which agents can drive their workspace best, I'd love to be able to point to some hard data rather than vibes.
  • What to train on. With every run we have a full graded trace, and those traces could potentially be used to post-train a small, cheap model to drive the workspace. Could we push to HuggingFace a small agent that clients can run in their own environment to navigate their workspace.

Then, this also can work as a way to catch regressions as the product surface increases with new features, or we add more MCP tools.

A chat answer is very limiting​

For financial workflows, chat interface will often be enough. But not always. A lot of times you need to see artifacts that are representing real data in the backend that you can trust, or comparing two time-series side by side, or check how your portfolio is doing overall, or just look into numbers and letting your human brain do pattern recognition on its own.

If you work from that assumption, then you want to measure how the AI can manipulate your workspace state and not the tokens output. For instance - did the widget land on the right dashboard, under the. correct tab, with the right ticker and parameters, without breaking the layout that already existed?

So WorkspaceBench grades the final state, deterministically (i.e. exact widget identifiers, exact data arguments, layout bounds, overlap checks). τ-bench takes the same position for tool agents: grade the state, not the chat output - there are other benchmarks for that, and it will be highly dependent on the workflow you are interested in (imo each firm should actually have their own private evals to measure models based on what actually matters to them).

So in a nutshell:

WorkspaceBench measures whether an agent can operate the workspace, not whether it can reason about markets or financial data.

What one task looks like​

Here's an example of an earnings review on AAPL:

Prep the earnings review for AAPL: add the price performance and latest news widgets, then write a note with the key takeaways — EPS, revenue, and the move.


Here is an agent doing exactly that, live in the Workspace:

When it stops, the grader ignores everything the agent said in chat and reads only what it built:

Then it checks:

  • Correct widgets with the right data arguments
  • Laid out inside the grid with no overlaps,
  • Note that at least mentions the required terms - AAPL, EPS, revenue (a presence check, not a values check!)
  • Clean trace — i.e. schema fetched before creating anything, zero invalid calls.

Either every check passes or the scenario fails; there is no partial credit. This is your typical rubric-graded benchmark (itemized rubric per task), except that in this case it's a verifiable reward rather than an LLM judge interpretation.

This is just one example.

The rest of this post is how I scaled this to 300, and how models perform on these.

Setup​

The six models​

I picked the following models for testing:

  • GPT-5.5 — the most advanced model available
  • Claude Sonnet 5 — a recent release, and fairly affordable (and more affordable than Claude Fable 5)
  • GLM-5.2 — the best open-weights model right now
  • gpt-4.1-mini — a solid, cheap model, and what I used as the gating model to calibrate the tiers (more on this later)
  • gpt-oss:20b — a strong open-weights mixture-of-experts
  • Qwen3 8B — a small open-weights dense model

The last two run on my own machine through Ollama for free; the rest go through vendor APIs or OpenRouter.

Workspace MCP tools​

Workspace MCP is the server that turns OpenBB Workspace into something an agent can operate. It exposes the workspace (dashboards, widgets, data, apps, skills) as a set of MCP tools and resources, so a model can list what exists, read a widget's schema, create and update widgets, lay them out, and assemble whole apps.

What an LLM can access over MCP is a subset of what a person can do in the workspace, since a person can click around the interface.

As a matter of fact, a coding agent like Codex can use the in-app browser, and Claude Code can use the /chrome tool to navigate the browser, but I am ignoring this. So whatever the scores are in this post - treat them as a lower bound of what an agent can really do navigating the workspace. I'm trying to evaluate the raw model (without including any harness).

The benchmark can't call the real hosted MCP server on every run - that wouldn't scale well. So I shared the Workspace MCP code (which we actually had open source here) with my coding agent so it could build a simulator. A deterministic state machine that has the exact same tool contract - tool names, argument shapes and error responses. And to make sure the simulator doesn't quietly drift from the real thing, a smoke runner can replay any scenario through a real local workspace-mcp sidecar and diff the advertised tool surface, so a change to the server shows up as a failure instead of the benchmark grading a workspace that no longer exists.

The Workspace MCP has:

  • 2 prompts - the server's own usage guidance and session context, fetched with get_workspace_prompt
  • MCP resources - the app-builder index and the workspace skills, read with read_workspace_resource
  • 20 tools - the full read-and-write surface: discovery (get_workspace_snapshot, list_available_widgets, get_widget_schema), widget CRUD (create_widget, update_widget, update_widget_layout, delete_widget), navigation (navigate_workspace, manage_dashboard), the higher-level surface (manage_apps, get_skill_content, assign_tasks_to_agents), and a few more you can see in the chart below

Every bar below counts how many of the 300 scenarios require that part of the surface. Nothing goes untested - every tool is needed by at least 20 scenarios - but obviously some are more popular than others.

Resources and prompts are the newest and least familiar part of that surface, and the easiest for an agent to ignore. Here is one using them: reading the app-builder index over MCP resources to find an existing app template, then opening it into the workspace in a single step instead of hunting for it by hand:

IMPROVE HERE - WE SHOULD ACTUALLY BE ABLE TO BUILD AN APP THAT RELIES

Fifteen families of workspace work​

Every scenario is anchored on one MCP capability — its family. There are 15 of them, 20 scenarios each, and grouped they cover five kinds of workspace work:

  • Widget CRUD — create, update, delete, layout: place, change, remove, and arrange widgets on the grid.
  • Reading & recording — read, inspect, note: pull the data behind widgets, reason about a workspace the agent didn't build, leave durable notes behind.
  • Navigation & precision — navigate, params: move across dashboards and tabs; parameter surgery on existing widgets.
  • Platform leverage — apps, skills, delegate, backends: instantiate app templates, run workspace skills, delegate to other agents, manage data backends.
  • Protocol surface — resources, prompts: the MCP resources and prompts most agents never think to read.

Five tiers of difficulty​

Families define which capability a task exercises, on the other hand tiers tries to convey its difficulty via what the task is made of - i.e. how many tool calls it requires, whether the starting state is seeded with something wrong, how much discovery is required, and how tight the turn budget is.

Here they are:

  • t0 — single action. One tool call does the job.
  • t1 — one action and some reasoning. E.g. discover the catalog, fetch the schema, then act.
  • t2 — composed artifacts. Several widgets, tabs, and a note have to land together as one coherent dashboard.
  • t3 — repair & preserve. The episode starts on a populated workspace: sometimes something is seeded broken to fix, always something healthy that must survive untouched.
  • t4 — multi-intent composition. Several of the above in a single episode, under a specific turn budget.

Here is what repair & preserve looks like in practice — an agent (not a raw model in this case!) correcting a wrong ticker while leaving the healthy widgets untouched:

How the 300 scenarios are generated​

Every scenario is AI-generated including the early ones I drafted myself.

I got inspired by the approach taken by TMax which sampled 14,600 terminal environments compositionally from structured axes.

So the suite is generated compositionally: 15 families, each anchored on one MCP capability, each carrying a complete t0–t4 ladder - 4 scenarios per family, 300 in total.

The 4 per cell gives each scenario variety (different widgets, tickers, backends, prompts) and lets the boundary tiers split difficulty evenly (t1 is 2 easy + 2 medium, t3 is 2 medium + 2 hard). The same tool is tested at every tier: delete_widget at t0 means "remove the only widget"; at t4 it means "remove one duplicate, fix the survivor, document the repair" in a single episode.

I still didn't like that I didn't have a good view of what tools were required by each of the tasks (apart from looking at the code of course), so I built a tool required viewer that gave me that visibility. Also, click on one of the rows to dig into that specific task!

Here's an overview of the number of tool calls per task.

One more way to read the 300: ignore which tools a scenario calls and ask what the analyst actually asked for. Every scenario is one of five jobs - the workflow levels (L0–L4) below — from "read this workspace and tell me what's there" to "something here is broken, fix it without wrecking the rest".

A level isn't a third axis I designed; it's read straight off the family × tier cell.

Take the delete family: at t0–t2 the job is single-widget housekeeping (L1), but at t3 the dashboard has two identical widgets and the ask is "remove exactly one duplicate, then document what you did". Same anchor tool, but the job has become repair (L4). The mix is deliberate: building dashboards (L2) is the most product-relevant job, so it gets a quarter of the suite. And repair earns its slot because agents, and users, make mistakes and need to recover.

The one collision is L4 repair vs t3 repair & preserve. L4 says the job is repair — the analyst asked for something broken to be fixed — and all 32 L4 tasks sit at t3 or t4, because there is no structurally trivial repair. t3 says the structure involves pre-existing state under a preservation constraint, and only 20 of its 60 scenarios are actual repairs; the rest are tasks like "add this widget next to the one already there, without disturbing it".

How it runs​

The whole benchmark diagram can be seen below (click on each of the steps).

Follow one scenario through it:

  1. The scenario is a JSON file — the prompt, the fixture backends to seed, the tools the agent may call, and the exact checks that count as success. One file per task, under workspace_bench_v1/.
  2. The harness reads it, seeds a deterministic workspace, writes a task envelope to disk, sets four environment variables, and runs whatever command you named.
  3. Your agent — a Python adapter, a CLI tool, a model behind an API — reads the envelope and emits its actions as JSON-lines tool calls. It never imports the benchmark.
  4. The simulator (simulated_workspace.py) applies each call and hands back a result, turn by turn.
  5. Everything is saved — each run directory keeps the full conversation.json, model_responses.jsonl, and tool_calls.jsonl, and the whole run is exported as portable rollout JSONL in runs/exports/ (1,800 episodes committed there already).
  6. The grader (graders.py) reads the final workspace state first, then the trace, and emits pass/fail, a score, and stable issue codes.

Every edge in that loop is a plain, boring shape — a scenario file in, tool-call JSONL out, a graded scorecard — which is exactly why any agent, in any language, plugs in without importing a thing. Click any stage in the diagram above to see the file it lives in and an example of what flows through it.

Running the whole thing is then one command, and re-certifying it is another:

# run any adapter file against the full 300-scenario release
uv run --extra dev workspace-bench compare-models \
  --models-file models.json --pack all

# re-certify the pack: oracle passes, no-op fails, quotas + fingerprints hold
uv run --extra dev workspace-bench validate --pack all --min-scenarios 300

# export a finished run as portable rollout JSONL
uv run --extra dev workspace-bench export-rollouts \
  --comparison-dir runs/comparison/<run> --output rollouts.jsonl

The simulator is intentional: it is not a pixel or browser simulator, it is a deterministic state machine for the Workspace MCP contract, which makes runs fast, reproducible, and cheap enough for CI. A live smoke runner (live_mcp.py) drives the same scenario contract through a real local workspace-mcp sidecar, so drift from the real server shows up as a failure instead of going unnoticed.

Certifying the benchmark​

A tier system only makes sense if, well, there are tiers. So in this case I wanted to check whether with each difficulty tier, the models were struggling more and more. This would allow me to calibrate the difficulty axis empirically.

So I picked a cheap mid-strength model (gpt-4.1-mini) and run it over all 300 scenarios, to check whether the pass rate would fall monotonically from t0 to t4. And they did, very nicely as a matter of fact!

Synthetic test selection​

When you generate tasks instead of hand-writing them, you cannot trust any single one by default given that the generator will produce broken tasks, unsolvable tasks, and tasks that are ridiculously trivial. This is the problem TMax ran into at 14,600 environments and solves with a pipeline: generate, then put every candidate through machine checks before it is allowed into the suite.

WorkspaceBench borrows that idea.

Each stage does something specific, this is basically "synthetic selection":

  • Generate — build a candidate scenario from a family template: which tool it anchors on, which tier, which pathologies to seed.
  • Certify — two mechanical checks every scenario has to pass.
    • Oracle passes: the reference solution - i.e. the known-good sequence of tool calls - is replayed through the grader and must pass, which proves the task is actually solvable.
    • No-op fails: an agent that does nothing is graded and must fail, which proves the task isn't already satisfied by the starting workspace (no free passes).
  • Gate — run the calibration model and check its pass rate lands in the band the tier claims: high at t0, low at t4.
  • Fix the generator — if a whole tier or family comes out wrong, the bug is in the template, not in one task. Fix it there and re-run the loop over all 300.

Let me add some color on how this did indeed work:

The first gating run came back suspicious: the two families anchored on the benchmark's two newest tools scored exactly 0/20. Pulling on that thread surfaced three real defects - all in the benchmark, none in the models:

  1. The harness never documented the two new tools. get_workspace_prompt and read_workspace_resource had no schema reference, so every model invented argument shapes and got rejected call after call. That alone zeroed both families.
  2. Some prompts asked for less than the grader checked. One inspect task graded the note for mentioning the series id DGS10, but the prompt never asked for it. The model did what it was told and still failed.
  3. Schema discipline was specified inconsistently. Some prompts said "fetch the schema before creating", others silently expected it and failed the model for skipping it.

All three got fixed at the generator level, not by patching individual tasks, then the pack was re-certified and the sweep re-run from scratch. The gating curve went from a broken 83–67–48–47–30 to the clean 98–92–77–52–33 above.

One last thing about the gating model, since it is the only one I ran more than once: I ran gpt-4.1-mini three times over all 300 to check the numbers were stable.

Repeatability — gpt-4.1-mini × 3
attempt spread71.7% · 70.0% · 70.7% strict across three independent attempts. A 1.7-point spread, so one attempt is good to about ±1 point overall
tier stabilitypooled per-tier pass rates (98 · 93 · 77 · 55 · 31) match the single-attempt curve within ±3 points, and the curve is monotonic in all three attempts
flaky scenarios19 of 300 pass sometimes and fail sometimes for the same model - 1 · 0 · 3 · 5 · 10 across t0→t4. The flakiness concentrates exactly where the ladder puts difficulty
capability vs reliabilitypass@3 73.7% vs pass^3 67.3%. The 6.4-point gap between "passed at least once" and "passed every time" is what reliability costs

Every other model in this post is a single attempt, then: a point estimate good to about a point, assuming its variance looks anything like mini's.

Results​

These are real runs from the interactive comparison harness (model_compare.py): each model chooses one tool call, receives the simulated workspace result, then chooses the next call — all six against the same 300 certified scenarios, same grader, same logs.

The board​

The table below is the whole sweep in one place. Models are columns, ordered best to worst. The first two rows are the two scoring modes - Overall is the strict pass rate (a scenario only counts if every grader check passes) and mean check score is the same runs with partial credit. Below that, the pass rates sliced by tier, and then by all 15 families, easiest to hardest, with the best model per row highlighted. Read down a column and you get one model's profile; read across a row and you see where the suite is hard for everyone.

Two things I read off it before anything else.

The first is the spread. 94.0% to 49.7% strict is roughly two generations of capability, and the ordering is stable: across all 300 scenarios there is exactly one where the bottom model beats the top one (a t0 delegation task GPT-5.5 fumbles and Qwen3 8B happens to pass). Everything here is single-attempt pass@1 from one run (GPT-5.5 samples at temperature 1 because its API rejects 0), so any single cell carries some sampling noise. The repeatability check in the last section put that at about ±1 point, and the harness supports --repeats and pass@k if you want error bars.

The second is the gap between the two scoring rows. GPT-5.5 earns 99.2% of all grader checks but passes 94.0% of scenarios - partial credit flatters everyone, and that is why the headline metric is all-checks-or-nothing. There is also an inversion hiding in there: GLM-5.2's mean check score (90.1%) sits below gpt-4.1-mini's (95.7%) and even Qwen3 8B's (90.8%) despite a much higher strict pass rate. GLM fails messily - when it blows a scenario it takes several checks down with it - while the weaker models fail narrowly.

The family block at the bottom is where I spend most of my time. Read down a column and you get a model's failure fingerprint; read across the inspect row and you see the suite's open problem.

And one thing I learned while getting these runs fair: adapter plumbing can cost a frontier model double-digit points. OpenRouter can route Anthropic models through Bedrock, and requesting json_schema response_format on that route degraded Sonnet 5 to schema-minimal outputs. A controlled A/B caught it, and the fix was in my harness - disable response_format on that route - not in the model.

For anyone who wants to reproduce the table, this is the test of record:

  • Release: one release, 300 scenarios, certified before any model ran (oracle 300/300, no-op 0/300).
  • Models: gpt-5.5 & gpt-4.1-mini (OpenAI API) · anthropic/claude-sonnet-5 & z-ai/glm-5.2 (OpenRouter) · gpt-oss:20b & qwen3:8b (local Ollama).
  • Protocol: pass@1 - one attempt per scenario, interactive tool loop, temperature 0 (GPT-5.5: 1, API-enforced), same turn budget and grader for every model.
  • Artifacts: 1,800 full episode transcripts as rollout JSONL in runs/exports/, plus the compiled report in runs/reports/calibration.json.
  • Cost: ≈ $60 of API spend across the calibration loop and the final sweep, run July 2026.

Model by model​

Each pill above is one model: its strict pass rate, its tier curve, and a note on anything specific to its run. Clicking through them, each model has its own personality. GPT-5.5's misses are precision and ambiguity, not composition. Sonnet 5 is the clean one - monotonic from 100% at t0 down to 80% at t4. GLM-5.2 has the strangest curve: it dips at t0 because it treats simple tasks as beneath deliberation - invalid calls, skipped prompt fetches - then aces t1. gpt-4.1-mini is the gating curve you already know from the last section. gpt-oss:20b reproduces the same staircase with open weights, and Qwen3 8B shows where a small model's ceiling is.

Grouping the same numbers by tier shows three shapes. GPT-5.5 barely bends: 95% at t4, essentially its t0 level, so multi-intent composition is not what slows it down. Sonnet 5, gpt-4.1-mini, and gpt-oss:20b step down cleanly, each the same staircase at a different altitude - the tier axis measures the same thing at every capability level. And Qwen3 8B is the third shape: level with the mid-field at t0–t1, then a cliff - 47% at t2, 27% at t3, 17% at t4. Small models don't degrade here; they break, exactly where episodes turn multi-step. The contested band - the 192 scenarios where the models disagree - sits where the ladder says difficulty should live (24, 26, 43, 45, 54 across t0→t4); the rest are either passed by everyone or by no one.

Who passed what​

The rows nobody passes are the most valuable output of the whole sweep. 15 of 300 scenarios defeat all six models, and they are not scattered: 8 are inspect-family tasks (deduplicate or reconcile a workspace the agent did not build — ambient-state reasoning), 4 are companion-widget parameter tasks, 2 are precise layout splits, and 1 is a full skills-driven tearsheet. Even GPT-5.5 passes only 11/20 of the inspect family. That list is where a future t5 comes from, and it doubles as a work queue for the tool surface itself. It is also the clearest evidence that the remaining headroom is about reading state, not composing calls. At the opposite end, 93 scenarios are passed by everyone — the sanity floor, 70 of them in t0–t1, which is what "t0 nearly free" should look like.

Failures are also model-specific in ways a single number hides. Qwen3 8B scores 4/20 on the apps family — long instantiation chains — where every other model scores 17 or better. GLM-5.2 is the only model that struggles with the prompts family (8/20), because it skips the prompt fetch it considers unnecessary. gpt-oss:20b bottoms out on documentation notes (5/20). Six models, six different failure fingerprints on identical tasks — which is why the chart below breaks every model out by failure type, instead of averaging the differences away.

The honest read: the benchmark separates the mid-field sharply, but the frontier is already near the ceiling, and what it still fails is precision and ambient state, not composition. I would treat that as the roadmap for t5.


Failure analysis​

Every failed check maps to a stable issue code, so "model X is worse" becomes something specific. That specificity is the whole point — the difference between a useless verdict and a useful one is the difference between "the model did not complete the task" and missing_widget: expected one Bench Equities/price_performance widget with data_args {"symbol": "AAPL"}, found zero. Below is every model against the six most common failures — how many of the 300 scenarios each tripped on. Fewer is better. The shapes are the story: gpt-4.1-mini and Qwen3 8B pile up on schema_not_called_before_create (they skip discovery), GLM-5.2 and gpt-oss:20b burn too_many_invalid_calls (malformed calls), and layout_mismatch touches even the frontier — precise geometry is hard for everyone.

Conclusion​

I set out to answer one question: if agents are going to drive OpenBB Workspace for real users, how well do they actually do it? WorkspaceBench turns that from a vibe into a number. Six agents, one grader, 300 certified tasks that span single actions up to multi-intent repairs — and a result specific enough to act on. The frontier has essentially solved composition (GPT-5.5 is as strong at t4 as at t0) but not precision or ambient-state reasoning; the mid-field separates cleanly; and 15 tasks that nobody passes mark exactly where the ceiling is.

The point of a number is that it moves. Now every change — a clearer tool schema, a newer model, a fine-tune — has somewhere to show up, and I can hill-climb it instead of shipping a demo and hoping. That is the thing I was missing: a way to measure workspace-agent performance, and then tell both the product and our users what to do about it.

Future work​

Every failure here points at something to fix, in two directions, and both reuse this same benchmark unchanged.

One is the tool surface. Look at the failure analysis again: GLM-5.2 tripped the invalid-call budget on dozens of scenarios, and schema_not_called_before_create dominates the mid-field. Those are not only verdicts on the models; they are verdicts on the MCP tools — the schemas, the names, the error messages the workspace hands to agents. The same graded run that ranks the models is a to-do list for the server: make the surface clearer and every agent gets better at once.

The other is the agents themselves. Every episode leaves a full, graded trace — 1,800 of them are already in the repo. A grader that emits a clean pass/fail signal is exactly what a reinforcement-learning loop needs for reward, and the same deterministic simulator can serve as the environment. Turning these traces into training signal for smaller or cheaper agents is the obvious next step.

Both are open work. For now the point stands on its own: if agents are going to drive a financial workspace, you have to be able to measure how well they do it — and then the measurement tells you what to improve. Incentives always win. The least I can do is measure the ones I am creating.