Field note: treat your agent’s token spend like a CI cost gate
Running multi-agent pipelines, the failure mode that keeps biting me isn’t a crash — it’s a silent
cost blowup. A run finishes, produces something plausible, and only later do I notice one agent
looped, or a context assembly step ballooned, and that run cost several times what it should have.
The output looked fine, so nothing flagged it.
What’s helped, none of it tied to a specific framework:
-
Budget the prompt before the run, not after. Measure the tokens going into the window before
you fire the task. Most of my blowups were an upstream step quietly stuffing the context. -
Attribute spend per step/agent. When a pipeline gets expensive, “which agent ate the budget”
is the whole game. Aggregate cost tells you nothing actionable. -
Gate the cost in CI the way you’d gate build time or test duration. We already fail a build
when it gets too slow. An agent change that silently 10x’s token spend should trip the same kind
of check — a budget number that’s allowed to fail the PR. Cost regression is a regression.
The mental shift that made it click: the context window is a cost gate you can enforce in CI, not a
bill you reconcile afterward.
How does everyone here handle this? When an agent change quietly multiplies token spend, what
catches it for you — a gate, an alert, or nothing yet?