The Economics of AI Operations: Tokens, GPUs, Latency and Energy

All courses / AI Operations / Module 3 / Chapter 15

Learning objectives

After the lesson the participant can:

• Calculate the drivers of cost for a metered call and explain why an agent's unit of cost is the run
• Compare the four forms of AI capacity by what is fixed, what varies and which failure shows first
• Evaluate a cost-saving lever by its effect on latency and quality and the evidence needed before promotion
• Draft a cost-and-latency budget record with layered spend limits, a fallback ladder and a stop condition

Key concepts

Cost per outcome as a distribution

A cost per request is a means to an end; the chapter measures the use case, for example cost per successful outcome, and states it as a distribution, with the 50th, 95th and 99th percentile of cost per run, since the tail is where a runaway sits.

The run as unit of cost

For an agent a run consists of several model calls interleaved with tool calls, each resending the context accumulated so far, so cost grows with the number of steps and a loop multiplies it.

Fallback rung

A second path for a request when the primary is too slow, refused, wrong or absent. Each rung is declared in the service record with its own objective, data tier, owner, cost and date of last use, is triggered by an independent probe, and counts as unavailable unless exercised.

Energy claim

A measurement with a boundary, a period and a count. Published per-request figures use different boundaries and do not combine, and one provider's figure is not the energy of the operator's own use.

Layered spend limits

Caps per call, per run, per day and per month, enforced at the gateway on a credential the agent cannot widen, each with a stop held outside the agent. Every cost limit is paired with an outcome measure so that the agent cannot meet it by doing less.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. Prices for a fixed level of capability have fallen fast, yet bills rise. What in the way your services use models explains the difference?
2. For a critical service the stop degrades and does not halt. What is the degraded mode of your most important AI service, and when was it last used?
3. The Region's record states that it cannot yet claim energy per conversation. What would it need to measure, over what boundary, before it could?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. Which failure does the chapter's capacity table say shows first for on-demand, metered capacity?

   1. Hardware failure in the organisation's own halls
   2. A rate limit or quota refusal at a peak
   3. Slow growth of the pool through procurement
   4. Timeouts after depreciation of the accelerators
   Show the answer

   B. A rate limit or quota refusal at a peak
   With on-demand capacity nothing is fixed but the price per token; the bill and the granted quota vary, so the first failure is a refusal at a peak. The evidence to ask for is the quota in the contract and a test of the limit.

2. What is the quality risk the lever table gives for a cascade of a cheap model with escalation to a stronger one?

   1. Wrong answers accepted by the cheap stage
   2. Truncated answers from a shorter output limit
   3. Higher latency at the tail from larger batches
   4. A different data position in a cheaper region
   Show the answer

   A. Wrong answers accepted by the cheap stage
   A cascade saves cost only when the cheap stage answers, so its risk is wrong answers that are accepted without escalation. The evidence required is the escalation rate, the pass rate and a review of accepted answers.

3. Where must an agent's spend limits be enforced, according to the chapter?

   1. In the agent's instructions, as a budget it is asked to respect
   2. In the monthly invoice review by the FinOps lead
   3. At the gateway, on a credential the agent cannot widen, with the stop held outside the agent
   4. In the model provider's dashboard, under the fair-use clause of the licence that the vendor manager holds
   Show the answer

   C. At the gateway, on a credential the agent cannot widen, with the stop held outside the agent
   A spending limit is a control and must not depend on the agent's compliance. The gateway enforces the caps, and revoking the credential at the gateway's identity service stops the agent from outside.