Skip to main content

77 posts tagged with "ai"

View All Tags

Introducing WorkspaceBench to evaluate OpenBB Workspace for agentic workflows

· 63 min read

A few weeks ago at NY Tech Week I gave a talk on centralizing dashboards and agentic workflows in OpenBB Workspace. At the end I talked about how I'm seeing more analysts and quants using Claude Code and Codex to drive their work - and so we need to adapt the workspace to make it (even more) agent first.

That is why we announced Workspace MCP: it exposes the workspace (dashboards, widgets, data, apps, skills) as a set of MCP tools, so an agent can read data, build dashboards, and assemble whole apps.

If agents are going to drive this workspace for real analysts, I need a way to measure how well they navigate it. That led me to build WorkspaceBench: a vanilla harness, a simulator of OpenBB Workspace, realistic tasks, and graders that evaluate the agent's output.

In order to build this, I got inspired by:

I wanted to bring some of these concepts to an actual financial workspace.

I can't recall last side project that took me as much time as this one. A lot of iteration and killing it and starting from scratch.

Initially I gave the reasoning and strategy to Fable, and let it come up with the how - i.e. how tasks are defined, what tools are used, initial dashboard setup, etc.. after a few iterations I actually liked the staircase of results I was seeing from the models on the tasks, and so thought we were almost done...

Except that when I went to dig into how the tasks were set up, expecting that my mental model for how it should look had been implemented 1:1 - it was actually not there at all. Not only was there a lot of fluff (like variables that didn't add anything to what we wanted to test), but I felt that many tasks were not super representative of what a user does in the workspace. E.g. some tasks were phrased as "use add_generative_widget markdown_note widget with text 'The revenue of AAPL is X'" - which is really stupid when you think about it. Instead I was expecting something like "Add a note to the dashboard with the revenue of the company in the dashboard". This means that the model has to reason about which company we are looking at, which widget contains the revenue data, and then the format on how that is output.

During this time period OpenAI released a post criticizing SWE Bench pro which really hit home for me. Reminds me of the acronym LATFD ("look at the f*cking data"). And the longer I avoided doing it, the more I felt like I was building on a sand foundation.

Don't get me wrong, Fable was still a major driver of what I've done here. But I almost scratched everything and started from start with a stronger foudnation and only gave the model certain degrees of freedom - and always checked the underlying tasks to ensure that it was on the right track.

This post will walk you through everything that was done, including: the workspace simulator, the datasets we used, what a task file is made of, what are we evaluating, the final results and what's next.

I hope you enjoy this more technical deep-dive into creating a financial workspace benchmark from scratch. I'm sure there are better ways to create benchmarks, but in this post I'll walk you through my thinking process and how I build it from scratch.

The Model Did What I Rewarded, Not What I Wanted

· 23 min read

This experiment started after reading Prime Intellect's Systematic Reward Hacking and Prime Sprints post. Their setup made reward hacking feel small enough to test directly: give a model a visible task, add a hidden reward component the model is never told about, and watch whether RL learns the proxy instead of the intended behavior.

I wanted to try the same style of experiment with a continuous (length-based) hack instead of a binary keyword hack. But the question was not simply whether reward hacking would happen. I was deliberately creating a conflict between the prompt and the reward.

I made the full experiment public here: DidierRLopes/reward-hacking. It includes the environment, hosted training configs, cached run data, generated figures, and the notebook used for this post.

The more interesting question was whether better prompting could protect against it:

What if the user asks for a direct answer, but the training reward quietly pays the model for being longer?

Why generative AI isn't uncovering alpha yet

· 8 min read

In the world of hedge funds, everyone's talking about AI. But recently, Ken Griffin, one of the industry's most respected voices, made a bold claim: generative AI isn't delivering alpha for hedge funds. Instead, it's mainly boosting workflows and productivity.

He's not entirely wrong but he's not telling the whole story either.

Here's the thing: the problem isn't that AI lacks the capability to find alpha. The problem is that most firms aren't giving AI the environment it needs to succeed.

How I built an MCP Server for developers building OpenBB apps

· 7 min read

The hardest part of building an OpenBB app isn't writing Python, it's getting the widgets.json spec right, since this spec was invented by us.

Every app requires developers to define widget types, inputs, and configuration options in exactly the right format. The documentation is strong, but it is long-form, which forces you to dig around for the parts you need. That slows down iteration, especially when you just want to get an idea live in OpenBB Workspace.

Long live long context with Gemini

· 13 min read

A practical exploration of using Gemini's long context window capabilities to analyze multi-page documents, featuring a Streamlit app for testing and iterating prompts.

Learn how to move beyond traditional RAG approaches for document analysis and leverage the power of large context windows for more accurate information retrieval.

The open source code is available here.

There's a zero percent chance that open source doesn't win

· 4 min read

AI is transforming software development at a groundbreaking pace, creating a reinforcing cycle that makes open source unstoppable.

As development costs plummet and AI tools make codebases more accessible than ever, the traditional moats of proprietary software are evaporating.

Here's why I believe that the future of software is inevitably open.

Why chat-only AI financial assistants are not the future

· 7 min read

Financial assistants structured like ChatGPT are great for quick searches but fall short for comprehensive investment research. They are limited by their one-dimensional approach, which hinders efficient data retrieval and long-term usability. Read on to discover how OpenBB Terminal Pro addresses these issues with a three-dimensional solution.

rabbit r1, there is hope

· 21 min read

I can see a future where people use rabbit r1 for very particular use cases where phone is suboptimal. For instance, when multiple people want to interact with said phone (e.g. selecting music at a party without having to give phone away) and that is not ideal due to personal information on phone, or when the phone isn't ideal because it has too many distractions and user wants to focus on doing something (e.g. practicing a presentation using recording session and then asking for feedback).

Creating an AI-powered financial analyst

· 12 min read


Our Platform aims to empower the OpenBB Copilot, an AI-powered financial analyst, to perform tasks ranging from knowledge retrieval to fully autonomous analysis. The architecture involves task decomposition, tool retrieval, and subtask agents, showcasing impressive results in both deterministic and non-deterministic workflows. Read on to explore its capabilities and don't forget to watch the demos.

The open source code is available here.