AI Agent Cost Control: Model Routing, Tool Budgets, Caching, and Retry Limits

At 3 a.m., a background report agent returns an empty response. The HTTP status is 200, but the body is empty. The retry logic only checks the status code, so it keeps going. Each request sends 500 input tokens. After 1,500 retries, the job has burned 750,000 tokens. The bill the next morning is not a pleasant way to start the day.

The root cause is not “the model was too expensive.” Three controls were missing: a circuit breaker, a budget check, and error classification. Agent cost can get out of control in three common ways:

Unlimited retries. The system does not classify failure modes, so an empty response is treated as retryable. There is no circuit breaker, so 1,500 consecutive failures still do not stop the run. Each retry resends the full context, multiplying cost by 2-5x.

Context bloat. A long-running task runs for 6 hours and grows the conversation history to 80K tokens. Without a checkpoint, a failure restarts the task from the beginning, and every step is paid for again.

Model overuse. Every task uses a frontier model because there is no routing policy. Even simple classification tasks use the most expensive path, wasting 70% of the tokens.

AI agent cost control is not one optimization. It is a layered design: budget objects, routing policy, cache hits, retry circuit breakers, cost logs, and alert thresholds. The practical move is to turn those six engineering objects into decision tables and executable checks.

Budget Object Design: What to Record, Where to Record It, and When to Break

Cost control starts with a budget object, not a single total. If everything is aggregated, a bill spike cannot tell you which user, task, or tool burned the budget.

Seven budget layers

Budget objects can be layered from coarse to fine:

Budget layer	Budget object	Suggested cap	Alert trigger
Layer 1	user	Daily/monthly cap per user	Alert when remaining budget < 20%
Layer 2	tenant	Separate budget pool per tenant	Alert when remaining budget < 30%
Layer 3	workflow	Separate budget per workflow type	Alert when remaining budget < 40%
Layer 4	task	Separate budget per task type	Alert when remaining budget < 50%
Layer 5	tool	Budget per tool call	Skip the tool when over the cap
Layer 6	retry	Retry limit + circuit breaker	Disable the tool after N consecutive failures
Layer 7	cache	Cache hit-rate monitoring	Alert when the hit rate is below expectation

The exact caps depend on the business model and are mutable configuration. The layer structure is more stable. The remaining-budget field should be written into cost logs so alerts and circuit breakers can use it.

Fields each layer should record

Every budget layer should record these fields:

Field	Purpose	Type	Why it matters
model	Locate the model	string	Shows whether model routing is reasonable
inputTokens	Input token count	integer	Calculates input cost
outputTokens	Output token count	integer	Output cost often differs from input cost and should be tracked separately
cachedTokens	Cached token count	integer	Measures cache savings
costEstimate	Estimated cost of this call	float	Supports realtime cost accumulation
budgetRemaining	Remaining budget	float	Drives circuit-breaker decisions

The point of the budget object is dimensional accounting, not “just record total_cost.” In a multi-tenant system, cost needs to be allocated by tenantId. In a multi-tool system, toolName is how you find the black box.

Circuit-breaker logic

When the remaining budget falls below a threshold, trigger a circuit breaker:

def check_budget_before_retry(budget_remaining, retry_cost_estimate):
    if budget_remaining < retry_cost_estimate:
        return "skip_retry"  # Skip the retry when it would exceed budget
    if budget_remaining < threshold:  # threshold, for example 20%
        return "wait_approval"  # Budget is low, wait for approval
    return "continue"

The circuit breaker checks the remaining budget before the retry, not after the money is already gone. Estimate the cost before each retry and stop if the run would exceed the budget. That is how you avoid the “empty response retried 1,500 times” failure.

The companion article on context engineering will cover which context belongs in a stable prefix and which content should stay as runtime variables.

Model Routing Strategy: Not Every Task Needs the Most Expensive Model

A ticket triage agent often has a distribution like this: 70% of tasks are simple classification, 20% need a drafted reply, and only 10% should upgrade to a frontier model before sending a customer-facing email. A routing policy can save 40-85% of cost.

Model-level routing

Route by task complexity:

Task tier	Typical tasks	Recommended model tier	Share	Cost profile
70% - S tier	Classification, extraction, filtering, simple Q&A	nano/flash (cheapest)	70%	Short output, few turns, few tool calls
20% - M tier	Drafting, summarization, code generation, medium reasoning	mid-tier (moderate price)	20%	Medium output length, may call tools
10% - L tier	Review, architecture design, complex reasoning, multi-tool coordination	frontier (most expensive)	10%	Long output, many turns, frequent tool calls

A routing policy has three steps:

Step 1: Classify the task. Define S/M/L complexity criteria for each workflow, including output length, tool-call count, reasoning depth, and risk level.

Step 2: Default to S tier. Upgrade to M or L only when the task matches more complex features.

Step 3: Cascade routing. If the S-tier model fails, upgrade to M. If M fails, upgrade to L. If L fails, go to human intervention. Check the remaining budget before each upgrade, and skip the upgrade when it would exceed budget.

Service-tier routing

The same model can also be split by latency priority. Discounts and completion windows are mutable facts, so recheck official pricing before publishing.

Service tier	Cost discount	Completion time	Good fit
Realtime API	No discount	Immediate response	Interactive agent chat, high-priority tasks
Batch API	50% cost discount (recheck before publishing)	24-hour turnaround (recheck before publishing)	Batch evals, classification, embeddings, content-repository processing
Flex Processing	Lower cost (recheck before publishing)	Slower response, occasional unavailability	Low-priority async tasks, model evaluations, data enrichment

Offline routing checklist:

• Does the task need an immediate response? Yes -> Realtime API, with model routing.
• Can it accept a 24-hour delay? Yes -> Batch API.
• Is it low priority and tolerant of occasional failure? Yes -> Flex Processing.
• Is it batch work such as eval, classification, or embedding? Yes -> Batch API.

Risk tiers

Route tasks by risk level as well:

Risk level	Typical operation	Routing policy	Budget branch
Low risk	Classification, extraction, internal summary	S-tier model + automatic path	No approval, relaxed budget cap
Medium risk	Drafting a customer reply, suggesting a code change	M-tier model + optional approval	Over-budget branch can request approval
High risk	Sending customer email, charging money, architecture change	L-tier model + required approval	Approval wait, rejection, and timeout all become budget branches

The Human-in-the-Loop article in the same series covers how approval waits, rejections, and timeouts affect the budget branch.

Key constraints

Model routing needs a few hard constraints:

• Do not hard-code price numbers: pricing is mutable. Record model + pricingVersion instead of freezing a cost formula in business logic.
• Check remaining budget: before upgrading, check budgetRemaining. If the upgrade would exceed budget, skip it or request approval.
• Classify errors: routing failures should distinguish recoverable model-capability misses from unrecoverable parameter errors or permission denials.

Tool-Call Budgets: per-tool Budget, Timeout, and Retry Limits

Tool calls have schema and API cost. Each call sends the schema, arguments, and response parsing context, and the external API can still rate-limit or time out. Those costs are separate from the model call itself.

Tool budget controls

Give every tool its own budget controls:

Control	Suggested config	Monitoring field	Triggered action
Per-tool budget	Cap per call	tool_cost_estimate	Skip the tool or degrade when over the cap
Tool timeout	Timeout for the external API	tool_duration	Mark timeout as a retryable error
Retry limit per tool	Retry cap per tool	tool_retry_count	Give up on the tool instead of entering a retry loop

External-API tools, including search, databases, and third-party services, need their own accounting. Otherwise, tool calls become the cost black box.

Tool failure classification

Tool failures split into recoverable and unrecoverable cases.

Failure type	Typical errors	Handling strategy	Cost impact
Recoverable failure	Network timeout, 503 Service Unavailable, 429 Rate Limit	Retry automatically with exponential backoff and retry-after	Each retry sends full context again
Unrecoverable failure	403 Permission Denied, 400 Bad Request, missing tool	Do not retry; inject the error so the model can decide	No retry, so no repeated waste

The rule is simple: retry only temporary failures caused by external conditions. Do not retry internal configuration errors.

Circuit breaker

Disable the tool after N consecutive failures so an “empty response retried 1,500 times” cannot happen:

def circuit_breaker_tool(tool_name, consecutive_failures, threshold=5):
    if consecutive_failures >= threshold:
        return "disable_tool"  # Disable the tool
    return "continue"

Write the circuit-breaker state into logs so you can explain why the tool was disabled. After the breaker trips, wait for human intervention or an automatic recovery check instead of repeatedly calling an unstable tool.

The basics of tool calling are covered in Tool Calling. This article extends that foundation with per-tool budgets, timeouts, and retry limits.

Prompt Caching Design: Stable Prefix, Variable Placement, and the 1024-Token Threshold

Prompt Caching optimizes input-token cost for a stable prompt prefix. It is not a business result cache. It routes requests with the same prompt prefix to a server that recently processed the same prefix, which can reduce latency and input-token cost.

Prompt Caching is not a result cache

Prompt Caching stores a stable prompt prefix, not a business result. The difference matters:

• Prompt Caching: caches a stable prompt prefix, such as the system prompt or tool schema. A cache hit saves input tokens, but the model still performs inference.
• Business result cache: stores a complete output, such as a tool result or database query. A hit returns directly and does not call the model.

The goals are different. Prompt Caching reduces input-token cost. A business result cache reduces the cost of the full call. You can use both: stable prefixes go through Prompt Caching, while high-frequency tool results go through a business cache.

Structure requirements

The key is separating stable prefixes from runtime variables:

Content type	Position	Cache-hit likelihood	Typical content
Stable prefix (goes into cache)	Front of the prompt	High	System prompt, tool schema, policy docs, few-shot examples
Runtime variables (do not go into cache)	Later in the prompt	Low	User input, file fragments, runtime state such as the current turn and temporary variables

Design steps:

1. Put the system prompt, tool schema, and policy text first: these parts are stable across calls and are easier to cache.
2. Put user input, file fragments, and runtime state later: these parts change on every call and should not be part of the stable prefix.
3. Monitor cache hit rate: record cachedTokens and total input tokens, then calculate the hit rate. A cache hit rate above 40% is healthy. Below 20%, inspect the prompt structure.

Threshold and effect

The automatic threshold and effect of Prompt Caching are mutable facts, so recheck the official docs before publishing:

• Threshold: automatically enabled at 1024 tokens and above (recheck before publishing).
• Effect: cache hits can reduce cost and latency (recheck the exact ratios).
• How to check hits: the usage.prompt_tokens_details.cached_tokens field.

Supported models, thresholds, and discount ratios for Prompt Caching can change, so verify them against the official pricing page before publishing. The durable design principle is stable: put static content first and variable content later.

Retry Circuit Breakers: Idempotency, Checkpoints, and Remaining-Budget Checks

Retries are one of the biggest sources of runaway cost. A background report agent can get stuck on an unstable tool, resend the full context on every failure, and after 1,500 retries drift far away from the normal cost path.

Check the budget before retrying

Check the remaining budget before a retry, not after the retry has already spent it:

def should_retry(error_type, budget_remaining, retry_cost_estimate):
    # Error classification
    if error_type in ["403", "400", "tool_not_exist"]:
        return False  # Unrecoverable error, do not retry

    # Budget check
    if budget_remaining < retry_cost_estimate:
        return False  # Over budget, do not retry

    return True  # Retryable

Put the budget check before retry logic. That is how you avoid spending an entire daily budget on an empty response retried 1,500 times.

Idempotency

Retries must not repeat side effects such as sending email or charging money:

• Use an idempotency ID, such as requestId, for tool calls. If the external API receives the same ID again, it should return the cached result instead of processing the operation twice.
• Write the idempotency ID into cost logs so repeated calls can be diagnosed.

The point of idempotency is “the same operation should not spend twice.” Without it, retries amplify both cost and side effects.

State saving (checkpoint)

Long-running tasks should recover without rerunning the entire workflow:

• Save a checkpoint at key execution nodes, including completed steps, current state, and a context summary.
• Resume from the checkpoint after a failure instead of starting from the beginning.
• Persist the checkpoint. Do not keep it only in memory.

Checkpoint and thread-state design are covered in LangGraph Agent Architecture.

Retry policy

Different error types need different retry policies:

Error type	Typical error	Retry policy	Cost impact
Network timeout	No response after 10 seconds	Exponential backoff + retry-after, max 3 retries	Each retry sends full context
503/429	Service Unavailable, Rate Limit	Wait for the rate-limit window + retry-after, max 3 retries	Waiting does not spend tokens, but retries do
403/400	Permission Denied, Bad Request	Do not retry; inject the error so the model can decide	No retry, avoiding invalid spend

The rule is “retry recoverable errors only.” Do not send invalid requests back to the model again and again.

Circuit breaker

After N consecutive failures, stop retrying and wait for intervention:

def circuit_breaker(consecutive_failures, threshold=5):
    if consecutive_failures >= threshold:
        return "stop_retry"  # Stop retrying
    return "continue"

Write the circuit-breaker decision into logs so you can explain why retries stopped. After the breaker trips, wait for human intervention or budget recovery instead of repeatedly calling an unstable tool or model.

Cost Logs and Alerts: Which Fields to Record and Which Thresholds to Set

Cost observability is the prerequisite for cost control. If the log fields are incomplete, you cannot locate the problem.

OpenTelemetry trace span attributes

Design cost logs around three span levels:

Agent run span (top level):

Field	Purpose	Type	Why it matters
runId	Locate a specific execution	string	Distinguishes repeated runs of the same workflow
tenantId	Locate the tenant	string	Allocates cost in multi-tenant systems
userId	Locate the user	string	Tracks cost trends per user
workflowName	Locate the workflow	string	Tracks cost by workflow type
totalCost	Estimated total cost	float	Supports realtime cost accumulation
budgetRemaining	Remaining budget	float	Drives circuit-breaker decisions
totalRetries	Total retries	integer	Shows retry amplification

Model call span (child span):

Field	Purpose	Type	Why it matters
model	Locate the model	string	Shows whether model routing is reasonable
pricingVersion	Pricing version	string	Avoids hard-coding the cost formula
inputTokens	Input token count	integer	Calculates input cost
outputTokens	Output token count	integer	Tracks output cost separately
cachedTokens	Cached token count	integer	Measures cache savings
costEstimate	Estimated cost for this call	float	Supports realtime cost accumulation
latencyMs	Call latency	integer	Helps decide whether Batch/Flex fits

Tool call span (child span):

Field	Purpose	Type	Why it matters
toolName	Locate the tool	string	Identifies tool-call overhead
toolBudget	Tool budget cap	float	Drives circuit-breaker decisions
toolTimeout	Tool timeout	integer	Classifies timeout failures
retryCount	Retry count	integer	Shows retry amplification
errorType	Error type	string	Distinguishes recoverable from unrecoverable failures

Do not only record total_cost. Split cost by dimension. Without these fields, a bill spike only says “over budget”; it does not tell you which user, tool, or retry path caused it.

The full design for logs, alerts, and failure recovery is covered in Agent Monitoring and Recovery. This article adds the cost fields and budget objects.

Alert thresholds

Set alert thresholds by dimension:

Alert dimension	Alert threshold	Alert channel	Triggered action
Global budget consumption	70%, 90%, 100%	Slack/email alert	70% notify, 90% degrade, 100% break
Single user/tenant consumption	More than 3x the average	Slack/email alert	Check for abnormal calls
Single-model failure rate	> 5%	Dashboard alert	Inspect model routing or service status
Single-tool retry count	> threshold	Dashboard alert	Inspect tool stability
Cache hit rate	< expected value	Dashboard alert	Inspect prompt structure

Alert thresholds should be written into the cost logic so they can trigger alerts and circuit breakers automatically.

Degradation strategy

After an alert fires, the degradation path can look like this:

Degradation path	How it degrades	Good fit	Cost impact
Model degradation	Large model -> small model	A single model has a high failure rate	Lower cost, possible quality drop
Path degradation	Realtime API -> Batch API -> Flex Processing	Global budget is being consumed too quickly	Higher latency, lower cost
Feature degradation	Disable non-core tool calls	A single tool has too many retries	Reduces tool-call overhead
User degradation	Rate limit, queue, or tell the user to retry later	A single user is consuming abnormally	Prevents one user from burning the budget

Degradation belongs in the budget logic. When an alert fires, the system should degrade automatically instead of waiting for manual intervention.

Next Steps

Agent cost control depends on monitoring, tool calling, and context engineering:

• Published: Agent Monitoring and Recovery: log fields, alert settings, and failure recovery. This article adds cost fields and budget objects.
• Published: LangGraph Agent Architecture: checkpoints, thread state, and failure recovery for long-running tasks.
• Published: Tool Calling: tool-calling basics. This article extends them with per-tool budgets, timeouts, and retry limits.
• Same series: context engineering: stable prefixes, cache hits, and which context belongs in the stable prefix versus runtime variables.
• Same series: Human-in-the-Loop: approval waits, rejection, timeout branches, and how approval changes cost and retry paths.

Start with the session-level guardrail: set a per-session cost limit and terminate a session automatically when it exceeds budget. That is the fastest first protection against one runaway task burning the whole day’s budget. Then expand the design into layered budget objects, model routing, tool-call budgets, Prompt Caching, retry circuit breakers, and cost logs.