Running many agents on 2.16: the output-economy

The 2.16 line is the first one where the interesting question stopped being "can it run four agents" and became "can I run forty and still read the bill, trust the summaries, and reproduce the run." This is the operator walkthrough of the pieces that answer that. Same posture as every recap: not a roadmap, not a refactor, just the things that started to hurt once the orchestrator was doing real work at real fan-out.

If you are running one agent, most of this is invisible. If you are running many, each item below removes a specific class of "I do not know what just happened" that shows up only at scale: spend you cannot attribute, summaries you cannot trust, completion payloads you cannot parse, a fleet you cannot pin, a lifecycle you cannot pause.

the output-economy suite

The bill was never the problem. The problem was that the bill arrived at the end, as one number, with no line you could point at and no lever you could pull before the fact. Four related surfaces landed to turn output spend from a post-hoc surprise into something you shape up front and read back per task.

per-role response-style profiles

Every role now carries a response-style profile that governs how verbose its output is meant to be. A docs role that writes prose and a qa role that should emit a terse pass/fail verdict were, until this cycle, spending output tokens the same way. They should not.

A profile is a small declaration in bernstein.yaml:

roles:
  qa:
    response_style: terse        # verdict + minimal rationale, no restated context
  security:
    response_style: standard     # findings with enough context to action them
  docs:
    response_style: expansive    # full prose, examples allowed

The profile is passed through to the agent as a system-level instruction and is recorded in the run's audit entry, so the style a role ran under is part of the reproducible record, not a prompt you have to reconstruct later. The point is not to clamp quality; it is to stop paying expansive-prose output rates for a role whose whole job is to say "green" or "here are three findings."

per-profile cost attribution, straight off the ledger

The cost ledger was already per-task, per-step, per-adapter, per-model. This cycle adds the response-style profile as an attribution dimension, so bernstein costs can answer "what did my verbose roles cost me versus my terse ones" without a spreadsheet.

bernstein costs --group-by profile

profile      tasks   output_tokens   usd
expansive       12         418,220   4.91
standard        31         192,540   2.16
terse           58          61,110   0.72

Every charge is written against the profile that incurred it. The line-item breakdown is the whole point: "the run cost more than I planned" becomes "my expansive roles were 63 percent of output spend and half of them are docs tasks that could be standard." That is a decision you can act on. The unattributed total was not.

a three-arm cost-and-quality A/B harness

Cheaper output is only a win if the work still passes. The harness runs the same task across three arms and reports cost and a quality verdict side by side, so the trade-off is measured rather than asserted.

bernstein ab run --plan plan.yaml \
  --arm cheap:haiku \
  --arm balanced:sonnet \
  --arm thorough:opus

Each arm executes the plan; each arm's output is scored by the same gate battery the normal run uses (tests green, lints clean, gate report). The harness emits a table of arm -> cost -> quality verdict -> gate pass rate, so "the cheap arm is 6x cheaper" sits next to "and it failed the security gate on 2 of 8 tasks." You pick the arm knowing both numbers. There is no automatic downgrade hidden inside this: the harness measures, the operator chooses.

operator-gated role-template compression

The role templates - the standing instructions each role carries into every task - accreted over dozens of releases. Compression rewrites a role template to a shorter form that preserves its behaviour, which cuts the fixed input cost every task in that role pays.

The load-bearing word is operator-gated. Compression never rewrites a template silently. bernstein roles compress produces a proposed diff and a rationale; nothing changes until you approve it.

bernstein roles compress --role backend --dry-run   # show the proposed rewrite + token delta
bernstein roles compress --role backend --apply     # writes only after you have seen the diff

The gate exists because a role template is behaviour, and a "helpful" automatic rewrite of behaviour is exactly the kind of change you want a human to have signed for. The token saving is real; it is just not worth surprising you with.

proactive context compaction, with a receipt

Long-running agents fill their context window. The old behaviour was reactive: compact when you are about to overflow, mid-task, under pressure, which is the worst possible moment to be summarising your own working memory. The new behaviour is proactive - compact on a threshold, before the window is tight, when there is headroom to do it carefully.

Two properties make this safe to trust rather than merely convenient:

Mechanical validation. After a compaction, a deterministic check confirms the compacted context still contains the invariants the task depends on (the active plan step, the task id, the declared acceptance criteria). If the mechanical check fails, the compaction is rejected and the un-compacted context is kept. A summary that drops the thing you were doing is worse than a full window.
A signed receipt. Every accepted compaction writes an HMAC-chained entry recording what was compacted, the before/after token counts, and the hash of the retained invariant set. The receipt lands in the same audit chain as everything else, so "the agent forgot the requirement halfway through" becomes a question you can answer by reading the record instead of guessing.

You do not invoke this; it runs inside long tasks. What you get is the receipt, and the guarantee that a compaction that would have dropped a load-bearing fact never happened silently.

a credential-shaped-content gate on model-bound summaries

When an agent summarises its own working context to send upstream to a model, that summary can accidentally carry a secret it read along the way: an API key in a config file, a token in an error message, a PEM banner in a log.

The credential-shaped-content gate runs over any summary before it is sent to a model. It scans for the shapes of secrets - provider key prefixes, high-entropy hex and base64 runs, private-key banners - and redacts a match in place, replacing it with a typed placeholder and recording the redaction in the audit chain. The gate is fail-closed: a summary that trips the detector is redacted before it leaves, not after.

This is deliberately a shape gate, not a semantic one. It does not claim to understand whether a string is a live credential; it claims that content shaped like a credential does not get forwarded to a model unredacted. For an operator running many agents over repositories they did not personally vet, that is the property that matters: the blast radius of "an agent read a .env it should not have" stops at the redaction line.

schema-enforced worker completion payloads, with typed refusals

A worker that finishes is supposed to hand back a completion payload the orchestrator can act on. When that payload was free-form, a worker could return prose the orchestrator then had to guess at, and a refusal - "I could not do this" - looked syntactically identical to a success.

Completion payloads are now schema-enforced. A worker returns a typed object the orchestrator validates against a schema at the boundary; a payload that does not validate is rejected at ingestion rather than misread downstream. Critically, refusal is a first-class, typed outcome:

{
  "status": "refused",
  "reason": "missing_precondition",
  "detail": "acceptance criteria reference an endpoint that does not exist on this branch",
  "artifacts": []
}

A typed refusal routes differently from a failure and completely differently from a success. The orchestrator can surface "this worker refused because the precondition was wrong" as its own state, label the ticket accordingly, and stop pretending an honest "I will not do this because X" was a crash. At fan-out, the difference between a refusal and a failure is the difference between a five-minute fix and a wild-goose chase.

sha256-pinned team manifests, with drift detection

A team manifest declares the fleet: which roles, which adapters, which models, which policies make up the set of agents a plan runs against. Until this cycle a manifest named its parts; it did not pin them. Two operators running "the same team" could be running subtly different fleets, and nobody would know until the outputs disagreed.

Manifests are now sha256-pinned. Each referenced component is anchored by content hash, so a manifest identifies an exact fleet, not an approximate one. Drift detection runs at load: if a pinned component's content no longer matches its recorded hash, the load surfaces the drift instead of running against a fleet that quietly changed underneath you.

bernstein team pin       # write the sha256 pins for the current manifest
bernstein team verify    # fail if any pinned component has drifted

bernstein team verify is the check you put in CI. Two operators who both pass it are provably running the byte-identical fleet, which is the precondition for comparing their runs at all. A team that drifts is a team whose results you cannot trust to be about the change you made rather than about the fleet that moved.

hold/release spawner lifecycle

At fan-out you sometimes need to freeze the fleet: a cost spike, an upstream outage, a bad plan you want to stop before it spends more. The spawner now has an explicit hold/release lifecycle so you can pause new agent creation without tearing down the run.

bernstein spawner hold      # stop spawning new agents; in-flight agents keep running
bernstein spawner status    # held | active, plus the in-flight count
bernstein spawner release   # resume spawning

hold is graceful: it stops new spawns and lets in-flight agents finish, rather than killing work midway. This is the control you reach for when you notice something is wrong and want the fleet to stop growing while you look, without losing the agents that are already doing correct work. Release picks up exactly where hold left off.

per-agent run instrumentation

Aggregate fleet metrics tell you the run is slow or expensive; they do not tell you which agent. This cycle adds per-agent run instrumentation: each agent's run carries its own structured record - tokens in and out, wall-clock, retries, gate outcomes, the response-style profile it ran under - attributed to that agent, not just rolled up into the fleet total.

bernstein agents stats --run <run-name>

agent            role       out_tokens   retries   wall_s   gates
brisk-sparrow    backend         41,220         0      182   pass
lucid-otter      qa               6,110         0       44   pass
calm-heron       docs            88,540         1      301   pass
wary-finch       security        12,900         2      158   fail

The value at scale is triage. "The run was expensive" is not actionable; "calm-heron on docs emitted 88k output tokens and wary-finch retried twice into a gate failure" is. The instrumentation ties each number to the agent that produced it, so the outlier is a row you can read instead of an average you have to decompose.

the Windows adapter path fix

The unglamorous one, and the most important line here for anyone on Windows.

The standard install path - pipx install bernstein, point it at a wrapped agent, run a plan - was failing on Windows in a way that produced no useful signal. The adapter's binary discovery resolved the agent executable through a cascade that was implicitly POSIX-shaped: it did not account for the .exe / .cmd / .bat extension resolution Windows applies through PATHEXT, so a binary that was genuinely present and on PATH read as absent. Worse, the failure surfaced as a silent exit rather than an error, so the operator got no traceback, no hint, no next step.

The fix does two things:

Extension-aware discovery. The resolver now walks the platform's executable-extension rules, so a Windows agent binary on PATH resolves the same way its POSIX sibling does. The discovery cascade - operator override wins, then the preferred binary, then the fallback - is unchanged; only the platform-correct lookup underneath it was wrong.
A structured error instead of a silent exit. When discovery genuinely fails, the adapter raises BinaryNotInstalledError with the binary name it looked for and where it looked, routed through the same first-run categorisation guard the rest of the CLI uses. The operator gets a hint panel and a sysexits.h exit code, not a blank prompt.

Why this matters more than a normal bug: the platform install path most people follow is the one that has to work without a support thread, and on Windows it silently did not. A regression that only surfaces through user reports is exactly the class that costs the most to run down - by the time it reaches you it has already cost someone else an afternoon. The adapter conformance suite now exercises the Windows discovery path so it stays closed.

why these fit together

Read as a list, this is a grab-bag. Read as one arc, it is a single answer to the fan-out question: at forty agents, the things that break first are the things you cannot see, cannot attribute, and cannot pin.

The output-economy suite makes spend attributable and shapeable before the fact.
Proactive compaction and the credential gate make an agent's own summaries trustworthy - they will not silently drop a requirement or forward a secret.
Schema-enforced payloads with typed refusals make worker completions legible - a refusal is a refusal, not a mystery.
Pinned manifests with drift detection make the fleet reproducible - two operators can prove they ran the same team.
Hold/release and per-agent instrumentation make the running fleet controllable and readable - you can pause it, and you can see which agent did what.

None of these is a headline on its own. Together they are the difference between an orchestrator you can demo and one you can operate.

upgrading

pipx install --upgrade bernstein     # or: pip install --upgrade bernstein

That is the whole upgrade. 2.x configs and plans run unchanged - bernstein run, bernstein conduct, bernstein gui serve all behave as before. The new surfaces are opt-in: response-style profiles default to the previous behaviour until you set them, the A/B harness and role compression are commands you invoke deliberately, and proactive compaction plus the credential gate run inside tasks without changing their interface.

# read the bill by response-style profile
bernstein costs --group-by profile
 
# measure cost against quality before you commit to an arm
bernstein ab run --plan plan.yaml \
  --arm cheap:haiku --arm balanced:sonnet --arm thorough:opus
 
# pin the fleet and fail CI if it drifts
bernstein team pin
bernstein team verify
 
# pause spawning without tearing down the run
bernstein spawner hold
bernstein spawner release
 
# see which agent spent what
bernstein agents stats --run <run-name>

bernstein 2.x recap: the cycle this one picks up from - lineage, ten trackers, A2A cards, the standalone verifier.
cost-aware routing: the bandit that decides which arm gets the work, which the output-economy suite now attributes and bounds.
v2.0 release notes: when the web UI landed in the wheel.