Token Economics in Enterprise AI: Designing for Scale and Cost Control

Moving an AI agent from a local demo into production changes the question.

In a demo, you ask: Does it work?

In production, you also ask:

• How many tokens does one successful task consume?
• What happens after ten conversation turns?
• How much tool metadata is attached to every request?
• Which model handles each step?
• How often does the agent retry?
• What does one completed business outcome cost—not merely one API call?

This is the practical meaning of token economics in enterprise AI: understanding how architectural choices translate into token consumption, latency, throughput, quality, and ultimately cost.

The token price on a model card matters, but it is only one variable. A cheaper model can become expensive if it needs repeated retries. A larger model can be economical if it solves a difficult task on the first attempt. A well-designed retrieval pipeline can save more than prompt trimming because it prevents irrelevant context from entering the request in the first place.

The central idea is simple:

Start with the Right Unit Economics

A token is a unit produced by a model-specific tokenizer. It may represent a word, part of a word, punctuation, or whitespace. Token counts therefore do not map cleanly to word counts, and the same text can tokenize differently across model families.

For a pay-as-you-go deployment, a simplified inference-cost model is:

request cost
  = input tokens × input rate
  + cached input tokens × cached-input rate
  + output tokens × output rate
  + any model- or tool-specific charges

Azure OpenAI Standard deployments are billed on input and output tokens, while Provisioned Throughput uses allocated capacity rather than a simple per-request token charge. Batch processing can also have different economics. Published prices are estimates rather than customer-specific quotes, so production planning should use the Azure pricing calculator and the terms of the applicable agreement.

Do not assume a universal ratio between input and output prices. Output tokens are often more expensive for many models, but the ratio varies by model, deployment type, region, and date. Treat the current model price sheet—not a rule such as “output always costs five times more”—as the source of truth.

Cost per request is useful, but cost per successful outcome is better:

cost per successful task
  = total inference, retrieval, and tool cost
    ÷ number of tasks meeting the quality threshold

This denominator matters. If a smaller model costs half as much per attempt but succeeds only after several retries, it may have worse unit economics than a more capable model.

Think in Terms of a Token Supply Chain

A production request is rarely just a user message. Its token payload can include:

1. System and developer instructions
2. Conversation history or a state summary
3. Retrieved documents
4. Tool and function definitions
5. Tool results
6. The current user request
7. The generated answer
8. Hidden reasoning tokens for models that use them
9. Retry, repair, and evaluator calls around the main request

That is the token supply chain. Optimization means removing waste at every stage without destroying the information the model needs to succeed.

A useful mental model is packing a suitcase with a weight limit. The context window is the suitcase. Every instruction, message, document chunk, tool schema, and output token competes for capacity. A larger suitcase helps, but it does not make poor packing efficient.

Context Growth: The Quiet Cost Multiplier

The model itself does not magically remember earlier Chat Completions calls. With a stateless API, the application must manage and resend the state needed for the next turn. However, Microsoft Foundry also offers stateful experiences: the Responses API supports multi-turn state, and optional stateful entities can persist message history. The correct design therefore depends on the API and storage mode you choose.

When a client resends the entire transcript on each turn, the prompt grows with the conversation. The per-turn input generally grows linearly with accumulated history; the cumulative tokens consumed across the session can grow roughly quadratically when every earlier message is repeatedly transmitted. Calling this “exponential growth” is usually inaccurate.

For example, if each turn adds approximately 500 tokens and every request includes the full transcript, ten turns do not consume only 5,000 input tokens. Earlier turns are paid for repeatedly:

500 + 1,000 + 1,500 + ... + 5,000 = 27,500 input tokens

This is why apparently harmless chat history becomes expensive at scale.

Use a tiered memory strategy

Do not treat memory as a single transcript. Separate it by purpose:

• Working context: the few recent turns required for conversational coherence.
• Rolling summary: a compact representation of older decisions, constraints, and unresolved items.
• Durable state: structured facts stored outside the prompt, such as customer ID, workflow stage, approvals, or task status.
• Retrievable history: older messages or artifacts fetched only when relevant.

The design principle is:

A database such as PostgreSQL can hold durable state, but the technology is secondary. The important architectural move is to stop equating “stored” with “sent to the model.” Retrieve only the state required for the current decision.

Summarize without discarding truth

Rolling summaries are useful, but summarization is lossy. A summary can omit a constraint, blur who approved what, or turn uncertainty into a false fact. For important workflows:

• preserve authoritative records outside the summary;
• structure the summary into facts, decisions, open questions, and constraints;
• include references to source events or document IDs;
• refresh the summary at deliberate checkpoints rather than after an arbitrary number of turns;
• evaluate whether the summary still supports the tasks the agent must perform.

The goal is not the smallest possible context. It is the smallest sufficient context.

In the Mechanics example, replacing two raw conversation pairs with a summary reduced the third-turn request to 118 tokens. The number is specific to that prompt, but the design lesson is general: summarize repeated narrative state and retrieve durable facts only when they are needed.

Prompt Hygiene Without Prompt Folklore

System prompts should be clear, compact, and testable. Remove duplication, stale examples, conflicting instructions, and prose that does not change behavior.

But do not ask a deployed model to reveal its hidden or base instructions as an optimization technique. That is unreliable, may conflict with platform protections, and does not tell you which parts of your own prompt are behaviorally redundant.

A safer prompt-audit process is empirical:

1. Establish a representative evaluation set.
2. Record quality, latency, and token usage for the current prompt.
3. Remove or consolidate one instruction group.
4. Re-run the same evaluation.
5. Keep the change only if quality remains within the accepted threshold.

Token reduction without evaluation is merely prompt shortening. Token reduction with stable task performance is optimization.

The source demonstration asked a model to reproduce what it described as its base instructions and estimated that copying similar language into an application prompt would add roughly 300 tokens per turn. That is useful as a warning about duplicated instructions, but not as a dependable audit method. Model-generated descriptions of hidden instructions can be incomplete or fabricated. Audit the instructions you control through ablation testing instead.

Bound Output Deliberately

Output control protects cost, latency, and user experience. The model-specific API may expose parameters such as max_output_tokens, max_completion_tokens, or max_tokens; support varies across models and API surfaces, so verify the parameter for the deployed model rather than hard-coding one assumption across every deployment.

A hard ceiling alone is not enough. If the model reaches it unexpectedly, the response may stop mid-sentence. Combine two constraints:

• Hard constraint: set the appropriate API output limit.
• Behavioral constraint: tell the model what shape of answer should fit—for example, “Return at most five bullets” or “Respond with one JSON object matching this schema.”

Prefer constraints tied to structure rather than vague requests such as “be concise.” A sentence count can work for prose, but schemas, bullet limits, or field-level maximums are often easier to validate programmatically.

Also distinguish visible output from reasoning consumption. Some reasoning-capable models may account for internal reasoning tokens within completion usage, so a short visible answer does not necessarily imply a tiny completion bill. Test representative workloads and inspect actual usage telemetry for the model you deploy.

Prompt Caching: Reuse the Prefix, Not the Myth

Prompt caching can reduce latency and input-token cost when requests share a sufficiently long, identical prefix. In Azure OpenAI, supported requests need at least 1,024 input tokens, and the first 1,024 tokens must be identical to qualify. Cached tokens receive discounted input pricing for supported Standard deployments, while Provisioned deployments can receive a larger discount. Retention behavior depends on the selected cache policy and model support.

This leads to a concrete prompt-layout pattern:

[stable system instructions]
[stable tool definitions]
[stable reference material]
[variable conversation and user input]

Put stable content first and frequently changing content later. Small differences near the beginning of the prompt can prevent reuse of the cached prefix.

The original article referred to an ephemeral cache-control flag and a universal “10% of normal input cost.” Those are not safe generalizations for Azure OpenAI in Microsoft Foundry. Azure’s documented behavior uses automatic prefix matching and, where supported, the prompt_cache_retention option; the actual discount depends on the deployment and current pricing.

The source demonstration used content blocks marked with an ephemeral cache-control setting and described follow-up cache hits at roughly 10% of the normal input-token price. That pattern is provider- and API-specific. In Azure OpenAI, supported prompt caching is based on matching request prefixes; requests require at least 1,024 tokens and an identical initial 1,024-token prefix. Current Azure documentation describes discounted cached input for Standard deployments and potentially larger discounts for Provisioned deployments. Always verify the API, model, deployment type, retention policy, and current price sheet.

Repeated RAG content can also be cacheable when the same retrieved material remains stable across requests. However, retrieval results often vary in order or content, so measure whether they actually preserve the common prefix needed for a hit.

Measure caching rather than assuming it works. Relevant indicators include:

• cached-token count or cache match rate;
• effective input cost;
• first-token and end-to-end latency;
• the percentage of requests sharing stable prefixes;
• whether load balancing or prompt variation fragments reuse.

Microsoft’s performance guidance explicitly identifies cache match rate, alongside input length, output length, and call rate, as part of workload shape for provisioned throughput.

Prompt caching is not semantic caching

These solve different problems:

Semantic caching can avoid a model call entirely, but it introduces correctness risks. Similar wording does not guarantee the same intent, authorization context, freshness requirement, or answer. If you use semantic caching, include tenant, identity, policy, data version, and freshness boundaries in the cache design—not just vector similarity.

Model Selection Is the Largest Macro-Lever

Using the strongest model for every request is simple, but often economically inefficient. Many agent workloads mix trivial classification, extraction, retrieval, tool orchestration, and difficult reasoning. These steps do not necessarily need the same model.

Microsoft Foundry’s model router is a trained routing model that evaluates each request and selects an eligible underlying model. It supports Balanced, Cost, and Quality modes, can use configured model subsets, and exposes the selected model in the response. The router makes the decision per request rather than once per conversation.

A practical routing strategy has three layers:

1. Deterministic routing for tasks with obvious requirements, such as a fixed extraction model.
2. Dynamic model routing for mixed or unpredictable requests.
3. Escalation when confidence, validation, or tool execution indicates that the first model was insufficient.

Model router is not a free pass to ignore compatibility. Its effective context window is constrained by the smallest eligible underlying model unless you define an appropriate subset, and model-router versions can change the pool of available models. Test quality, tool calling, latency, context limits, and cost against your own workload.

Avoid selecting models by price alone. Compare:

• task success rate;
• retry and repair rate;
• tokens per successful task;
• latency percentiles;
• structured-output validity;
• tool-call accuracy;
• safety and grounding performance.

A cheap failed call is still waste.

The Mechanics comparison used GPT-5.4 and GPT-5.4 mini and displayed a workload-specific quality score of 0.81 versus 0.67, together with an estimated mini-model cost that was 72% lower. Those values belong to that demonstration and its evaluation setup; they are not permanent model characteristics or a reusable price quote. The important lesson is to compare quality, safety, throughput, latency, and effective cost on the same task dataset.

The Tool-Schema Tax in Agentic Systems

Tools make agents useful, but their definitions consume context. If an application attaches dozens of verbose JSON schemas to every request, the model must repeatedly process metadata for tools it may never use.

Reduce this tax by:

• making tool names and descriptions precise but concise;
• removing redundant examples from schemas;
• grouping tools by domain or task stage;
• exposing only tools allowed and relevant for the current user and workflow;
• retrieving tool definitions on demand;
• measuring tokens before and after orchestration changes.

Microsoft Foundry Toolbox addresses the lifecycle and integration problem by packaging curated tools behind a centrally managed, MCP-compatible endpoint. An agent connects to one endpoint and can discover the available tools at runtime; Toolbox also centralizes authentication and versioning. Microsoft documentation describes Build and Consume as available, while some related quickstart items remain preview, so verify the status of each feature used in your architecture.

The Mechanics demo compared two agents answering the same product-catalog request. The agent with all tool definitions attached consumed almost 4,700 input tokens; the Toolbox-enabled agent consumed 467—about a 90% reduction in that test—with nearly identical answers. This is valuable evidence for the pattern, but it is a measured example, not a universal product guarantee. Actual savings depend on the number and size of tools, discovery behavior, selected tools, and orchestration design.

Be precise about what this proves. Toolbox can simplify discovery, reuse, credentials, and centralized lifecycle management. Measure tool-definition tokens and task quality in your own traces rather than assuming every workload will reproduce the demo’s reduction.

MCP also changes the trust boundary. Remote tools can send data to external services, invoke side effects, and introduce new authentication and approval requirements. Foundry supports public and private MCP endpoints, but private connectivity has specific setup requirements.

Retrieval Quality Is Token Quality

A RAG pipeline is not efficient merely because it uses vector search. If retrieval returns irrelevant or redundant chunks, you pay to send noise into the model—and may reduce answer quality at the same time.

Think of retrieval as a context compiler. Its job is not to return the most text. Its job is to assemble the smallest evidence package that supports a correct answer.

Useful levers include:

• query rewriting and decomposition;
• source selection before retrieval;
• hybrid keyword and vector search;
• metadata and permission filters;
• deduplication;
• reranking;
• chunk-size and overlap tuning;
• confidence thresholds;
• context compression with source references;
• stopping when enough evidence has been found.

Foundry IQ provides multi-source knowledge bases backed by Azure AI Search. Its agentic retrieval layer can plan queries, select sources, run searches, and return grounded results with citations. Some Foundry IQ capabilities are generally available while others remain in preview, depending on the API version and feature, so implementation guidance must be tied to the specific API version being used.

The source described Foundry IQ as a single developer endpoint that plans and decomposes a query, directs subqueries to appropriate knowledge sources, and returns retrieved context to the agent. That mental model is useful, but the current product description is broader: Foundry IQ provides multi-source, permission-aware knowledge bases backed by Azure AI Search agentic retrieval. Query-planning model usage is optional at minimal retrieval effort and used for more thorough low or medium effort. It should therefore not be reduced to “a small language model that routes intent.”

Optimize the Whole Agent, Not One Prompt

Token optimization is empirical. Every reduction can affect quality, and every quality improvement can alter token use, latency, or tool behavior.

The minimum production loop is:

1. Build a representative task dataset.
2. Define pass/fail and graded quality criteria.
3. Capture input, cached input, output, latency, model, tool calls, retries, and outcome.
4. Change one architectural variable.
5. Re-run the same evaluation.
6. Compare cost per successful task, not only average tokens.
7. Roll back changes that save tokens but harm the target outcome.

Microsoft Foundry Agent Optimizer automates a closed-loop evaluation process for hosted agents. It can optimize instructions, skills, tool definitions, and model selection, and can be started with azd ai agent optimize. As of July 12, 2026, Agent Optimizer is in preview, requires the relevant CLI extension and access prerequisites, and is not recommended by Microsoft for production workloads without appropriate preview evaluation.

In the Mechanics demonstration, Agent Optimizer produced a candidate that improved the configured evaluation pass rate from 70% to 90% and created a new agent version for testing before production. This preserves the key result from the source, but it should be read as one experiment—not a guaranteed uplift.

The optimizer evaluates candidates against a dataset; it does not guarantee a specific improvement such as “70% to 90%.” Results depend on the agent, dataset, evaluators, candidate count, and optimization target. Also remember that evaluation can repeatedly invoke external tools, potentially creating charges, mutations, or rate-limit pressure. Use test endpoints or mocks where side effects matter.

Build a Token Observability Model

You cannot optimize what you cannot attribute. At minimum, capture these dimensions per request or trace:

Azure Monitor exposes processed prompt-token and generated completion-token metrics that can be analyzed across time to estimate workload throughput. Microsoft recommends using a multi-week view rather than relying on a handful of test calls.

For shared model endpoints, Azure API Management can apply per-key token rate limits and quotas to supported LLM APIs. That is a guardrail against usage spikes, not a substitute for architectural optimization.

Tokens Are Only One Part of the AI Cost Stack

Token efficiency should not become tunnel vision. A production AI application also pays for the infrastructure around inference: containers and application hosting, VMs or managed compute, databases used for state, storage accounts and search indexes, private endpoints, networking and data transfer, observability, and the execution of external tools.

Optimizing tokens while leaving an oversized retrieval cluster, noisy telemetry pipeline, or overprovisioned application tier untouched does not produce an economically efficient system. Build one cost model that connects model inference with retrieval, tool execution, data storage, networking, and hosting. Then attribute those costs to a successful task or business outcome.

A Practical Optimization Order

When an agent is too expensive, optimize in this order:

1. Measure the baseline. Break down input, cached input, output, retries, tools, retrieval, latency, and success rate.
2. Remove obvious context waste. Eliminate duplicated instructions, repeated documents, irrelevant history, and unused tool schemas.
3. Fix retrieval. Better evidence usually improves both quality and token efficiency.
4. Bound output. Use model-appropriate API limits plus structural instructions.
5. Stabilize cacheable prefixes. Reorder prompts and measure actual cache hits.
6. Route by task difficulty. Test smaller models and dynamic routing against success criteria.
7. Externalize state. Keep durable facts outside the conversational transcript and inject them selectively.
8. Evaluate continuously. Every prompt, model, retrieval, and tool change can alter the economics.
9. Add quotas and alerts. Protect the platform from anomalous or runaway workloads.

This order avoids a common mistake: spending weeks shaving a few lines from a prompt while the agent retrieves ten irrelevant documents and advertises fifty tools on every turn.

The Architectural Rules of Thumb

If you remember only a few principles, make them these:

• Optimize cost per successful outcome, not cost per call.
• Store broadly, inject selectively.
• Aim for the smallest sufficient context—not the smallest context.
• Stable prefixes are cacheable; early prompt variance is expensive.
• A short answer can still carry a large input and reasoning bill.
• Tool descriptions are part of your prompt budget.
• Retrieval quality is token quality.
• Use the cheapest model that reliably passes the task—not simply the cheapest model.
• Treat previews, benchmarks, and optimization claims as hypotheses until your workload confirms them.

Final Perspective

Sustainable enterprise AI is not achieved by choosing one cheap model or imposing one token limit. It emerges from the interaction of context design, state management, retrieval, caching, model routing, tool orchestration, evaluation, and infrastructure.

The most important shift is conceptual: stop treating tokens as an incidental API meter. Treat them as an architectural resource.

Once you do that, the right questions become clearer:

• Which information deserves to enter the context?
• Which information can remain external until needed?
• Which parts of the prompt can be reused?
• Which model is sufficient for this specific decision?
• Which tool definitions are relevant now?
• Did the request produce a successful outcome?

That is how an agent moves from an impressive prototype to an economically sustainable production system.

Further Reading

• Microsoft Mechanics: Tokenomics—the new AI currency and your options explained
• Prompt caching with Azure OpenAI in Microsoft Foundry Models
• Model router for Microsoft Foundry
• Create, test, and deploy a Toolbox in Microsoft Foundry
• Agent Optimizer in Microsoft Foundry
• What is Foundry IQ?
• Azure OpenAI monitoring data reference