4 posts tagged with "llms"

A few weeks ago at NY Tech Week I gave a talk on centralizing dashboards and agentic workflows in OpenBB Workspace. At the end I talked about how I'm seeing more analysts and quants using Claude Code and Codex to drive their work - and so we need to adapt the workspace to make it agent first.

And this is why we had just announced Workspace MCP, which enables agents to work with the workspace by reading data, building dashboards, and assembling whole apps.

If agents are going to drive this workspace for real analysts, we (OpenBB) need to find a way to measure how well they can navigate our software. And by navigate I mean being able to inspect data, call right tools, change widget parameters, build a dashboard, add new widgets on the dashboard, leverage skills, etc..

This has lead me to build WorkspaceBench.

It includes the harness, a deterministic workspace simulator, 300 different synthethically generated scenarios, graders, and everything else. The Stark Industries mock enterprise apps, which I used for the demo above, is where the benchmark's scenarios come from — the same widgets and apps, turned into tasks.

This experiment started after reading Prime Intellect's Systematic Reward Hacking and Prime Sprints post. Their setup made reward hacking feel small enough to test directly: give a model a visible task, add a hidden reward component the model is never told about, and watch whether RL learns the proxy instead of the intended behavior.

I wanted to try the same style of experiment with a continuous (length-based) hack instead of a binary keyword hack. But the question was not simply whether reward hacking would happen. I was deliberately creating a conflict between the prompt and the reward.

I made the full experiment public here: DidierRLopes/reward-hacking. It includes the environment, hosted training configs, cached run data, generated figures, and the notebook used for this post.

The more interesting question was whether better prompting could protect against it:

What if the user asks for a direct answer, but the training reward quietly pays the model for being longer?

The term 'vibe coding' undermines the strategic work of delegating tasks to AI. This post argues for a shift in perspective towards 'outcome-driven development' as a more accurate description of the future of software engineering.

This blog post provides a comprehensive guide on how to perform target market analysis for your company using LLMs. It includes a detailed explanation of the BCG Matrix and the GE McKinsey Matrix, and how these frameworks can be used to determine market attractiveness and competitive advantage.

The open source code is available here.