Operator Brain Evaluation

A public meta-skill that runs the whole Operator Brain architecture (case study 02) for a stranger: a staged interview writes real files from real answers, and a scope boundary, build the memory layer, never plan the day, is tested and measured, not just claimed.

DETAIL LEVEL

Full technical. Skill code in skills/operator-brain/, benchmark data in skills/operator-brain-workspace/.

STACK

Claude · prompt and system design · eval harness

Table of Contents

The problem

The private version of this architecture (case study 02) is real and working, but it’s yours: folder names, routing rules, and content built around one specific business, redacted before publishing. Useful as an architecture sample, but nobody else can install it and have it do anything for them.

The underlying pattern, though, doesn’t depend on being about you: a small set of durable files an assistant reads every session, an index instead of one blob, a routing rule for where new information goes, history that only appends. It generalizes.

The approach

Build a meta-skill: it runs a staged interview, then writes real files from someone else’s real answers, not a template dressed up to look personal.

Draw an explicit boundary around it too. Picture a fence marking a property line: the skill tends its own yard, the context files, and never steps into the neighbor’s yard, deciding what the user should actually do with their day.

That line exists on purpose, so this skill and a private day-planning skill (case study 05) never end up doing each other’s job.

How it works

  1. States the privacy exposure model once, up front, before asking a single question: these files load every future conversation, so anything written here can surface in a screenshot or a shared chat.
  2. Runs a staged interview and writes what the user actually says into four numbered, plain-language folders, holding none of the builder’s own content.
  3. Refuses a live credential outright, no warning-label workaround, and says where the secret actually belongs instead.
  4. Reads a file immediately before writing to it, so a second writer, the user editing by hand, another open conversation, never gets silently overwritten.
  5. Declines to plan the user’s day, even when asked directly, holding the one boundary the whole design depends on.

Design decisions worth noting

  • Numbered, plain-language folders. 1-about-me/, 2-current-work/, 3-what-i-know/, 4-history/ instead of developer-jargon names. A file browser sorts numerically, so reading order and sort order become the same thing at no cost.
  • The fence line is the single most important behavior in the design. It exists so this skill and a private day-planning skill never collapse into fighting over the same job, and it’s tested directly, not stated.

The benchmark

Four of nine planned cases in EVALS.md have been run so far, chosen by a priority order set before any results existed. Each ran with the skill loaded and once on a plain Claude conversation with no skill, self-graded against the same checklist, three runs per condition.

CaseWith skillWithout skill
Bootstrapping from nothing100%28%
Day-planning boundary100%25%
Contradiction detection100%100%
Credential refusal100%93%
Case-level average100%61.5%

The real differentiation is in the first two cases: without the skill, Claude invented a file structure for a made-up persona, and produced a fully sequenced day plan, a competent answer to the one question this skill exists to decline.

Contradiction detection and credential refusal are not the skill’s differentiation. Both predictions expected a partial result, not a clean pass, and both were wrong the same way: baseline did the fuller, correct thing on its own. Reported as a baseline strength, not a win. Repeat runs found one real gap, a missed credential-rotation warning, fixed via the privacy rule and filing-routes reference, re-verified at five of five.

Checks are unweighted, self-graded counts, and the highest-stakes case, a concurrent-edit data-loss test, hasn’t been run yet. Full run data in the case study.

What this demonstrates

Building a tool that does the work instead of describing it: a meta-skill that runs a real interview and writes real files for someone else, not a template staged to look like one. Naming a fence line and holding it under pressure to step over into the neighbor’s job anyway.

And a testing method built to be wrong in public: predictions written before a single run existed, and results reported including the cases where a plain model already does the job.

Using this, you can build the same discipline into any tool that has to hold a hard line next to a capable neighbor: an agent that drafts but never sends, a system that reads records but never diagnoses, anywhere “it won’t do that” has to survive being asked nicely.

Let's Talk