A cost lesson from putting LLM agents into production that I wish I’d measured on day one: once you give a model tools/functions and retrieved context, your token bill stops tracking “how much the user said” and starts tracking “how much scaffolding you re-send every single turn.” The model is stateless, so all of that gets re-billed on every call. I finally sat down and measured where the input tokens actually go across a run of real multi-turn sessions, and the split was lopsided enough that I want to share it.
What I measured on my own runs (one agent, tool/function definitions + retrieved context in the prompt):
- Fixed per-turn overhead dominated. The tool/function definitions and system scaffolding are re-sent on every turn — for my setup that was roughly +3,200 tokens/turn just for the definitions of 4 integrations, before the user says a word. Tools are convenient, but each one you attach is a per-turn tax, not a one-time cost.
- Retrieved / tool-result context was 34% of input tokens. A third of what I paid for on input was material I fetched and stuffed into the prompt (API results, retrieved docs), not the conversation itself.
- Input:output ran ~1.9:1. I paid nearly twice as much to feed the model context as to get its answer back out. For short-output agent loops that ratio skews even harder toward input, which is the opposite of the intuition most people (me included) start with.
Three things that moved the needle once I could see the numbers:
- Trim tool/function definitions to what’s reachable right now, not the whole catalog every turn. Fewer attached tools ≈ directly fewer tokens/turn.
- Cache the static prefix (definitions + system prompt) where your provider supports prompt caching — the re-billing above is exactly the cost caching is designed to erase.
- Treat retrieved context as a budget, not a bucket. Cap it and measure whether it’s actually used — 34% is a lot to pay for context that might not change the answer.
Two honest questions for anyone here running LLM apps/agents beyond a notebook:
- Have you measured your own fixed-overhead-per-turn vs. user-content ratio? Mine felt high and I genuinely don’t know if that’s typical — I’d love to compare.
- If you use prompt caching on the tool/system prefix, what cache hit-rate are you actually seeing across a real conversation?
Full disclosure: I work at a small AI studio where we run a lot of agents, so token cost is a live line item — this is a field note from our own measurements, not a pitch (no link, nothing to sell). Happy to share the methodology if useful.