Running a crew in production and realizing there’s no native way to
cap what a crew spends before it runs.
The problem isn’t visibility — I can see costs after the fact. The
problem is enforcement. If a task loops or an agent retries
aggressively, the money is already gone by the time you notice.
What I’ve been testing: adding a check_spend tool to agents that
incur costs. Before any paid API call, the agent calls the tool with
the estimated amount. The tool either approves or blocks based on a
policy. The key is it’s structural — the check happens before the
call, not after.
Has anyone solved this differently in their crews? Curious what
patterns people are using for hard limits vs just monitoring.
You’ve drawn the right line — visibility and enforcement are genuinely different problems, and most “cost tracking” tooling only solves the first one.
Two layers have worked for me, because no single knob covers it:
Runtime blast-radius caps (native). CrewAI does give you bounding knobs even if they aren’t framed as spend limits: max_iter on an agent caps how many reasoning/tool loops it can take before it’s forced to stop, and max_rpm (agent or crew level) throttles call rate. Those are your defense against the exact failure you named — a task that loops or retries aggressively. The honest caveat: they bound iterations and rate, not dollars. A crew with a sane max_iter can still be expensive if each call drags a huge accumulated context or fat tool outputs, so caps alone don’t give you a number you can promise finance.
A dollar ceiling enforced before prod (the missing half). The regression that actually burns you usually lands as a change — someone bumps a model, adds a tool whose schema balloons the input, or removes a trim step — and you find out from the invoice. So I gate it in CI: run one representative crew, price the run in actual dollars, and fail the build if it crosses an absolute ceiling. That turns “we can see costs after the fact” into “a too-expensive crew can’t merge.”
For the pricing step I use npx @wartzar-bee/tokenscope — it takes real usage and prices each bucket (input, output, cache-write ~1.25x, cache-read ~0.1x) into a per-run dollar figure instead of a raw token count. To make that a hard gate there’s a small Apache-2.0 GitHub Action, wartzar-bee/ci-guardrail, that fails the check when a run exceeds a max-usd value you set. No affiliation ask — it’s just the setup that finally gave me enforcement instead of another dashboard.
Curious what your crew’s deepest delegation chain looks like — that’s usually where the runaway spend hides, because the sub-agents each re-send their own accumulated context.
I like the fact that you moved the budget check outside the model’s discretion.
I’m applying the same principle to completion itself: the executor shouldn’t be trusted to certify whether its own external action succeeded.
TookEffect independently observes the resulting state and verifies the effect before downstream work depends on it.
Have you run into loops where every individual step looks valid but the actual task outcome never gets closer to completion?
Yes — constantly, and it’s the failure mode that hurts most because none of your guards fire. Every tool call returns cleanly, every schema validates, so a check that only asks “did this step succeed?” waves it straight through. The task just never gets closer to done. Your TookEffect instinct is exactly the right cut: certify the effect, not the call.
Two things that helped me catch it earlier:
-
A task-level progress signature. Hash the slice of world-state the crew is actually supposed to move — the row, the file, the ticket status — and compare it across steps. If it’s unchanged for N iterations, halt: the loop is locally valid but globally stuck. It’s your idea applied as a stopping condition rather than a completion check.
-
Cost-derivative as a cheap early warning. A non-converging loop looks identical to progress if you only watch step validity — but it’s obvious the moment you watch spend: tokens climbing with zero state delta. I meter per-call tokens with @wartzar-bee/tokenscope so “spend up, state flat” becomes a signal I can alert on, not a month-end surprise.
And the backstop that doesn’t need you to detect the loop at all: a hard budget ceiling. Even when you can’t tell locally-valid from stuck, a max-USD gate bounds the blast radius before the money’s gone — that’s the enforcement-vs-visibility line from upthread. I run it as a CI cost gate (wartzar-bee/ci-guardrail) so a change that doubles per-run spend fails the build instead of the invoice.
That distinction between locally valid execution and actual convergence is really useful.
I’m going to keep those as separate signals in TookEffect: final Effect verification answers “did the expected state actually happen?”, while progress/convergence can answer “is the workflow measurably getting closer to that state?”
I especially like the idea of comparing a capability-specific slice of world state over time. I wouldn’t want to treat “state changed” as progress automatically, though — the change could be irrelevant or even move away from the expected effect — so the progress semantics probably need to stay tied to the exact Effect/postcondition.
Cost growth with zero state delta also seems like a useful auxiliary signal, but I’d keep cost outside the proof itself. The proof should still come from independently observed state.
This was a useful edge case — I’m adding it to the roadmap as a separate convergence/progress layer.