Cost, Capacity and Sustainability of AI

All courses / AI Platform and Security / Module 5 / Chapter 22

Learning objectives

After the lesson the participant can:

• Explain each term of the inference cost formula as an architectural decision.
• Compare the levers of caching, routing and batching by the cost driver each acts on.
• Draft a cost policy with budget alerts, a hard cap with fallback, and a unit-economics target.
• Calculate a Software Carbon Intensity score for a use case and state its boundary and limitations.

Key concepts

Token economics

The cost of a request is roughly input tokens times the input rate plus output tokens times the output rate, multiplied by request volume. Input tokens come from the system prompt, retrieved context, history and the user's message.

Routing policy

A gateway configuration that classifies each request and sends it to the cheapest model that meets the quality objective from the evaluation suite. Each route names its evidence, and a fallback makes behaviour under budget exhaustion explicit.

Chargeback and showback

Attribution of tokens and GPU hours through the gateway to use case and team. Whether the consuming team pays or is merely informed is a management decision; the architecture is the same.

Unit economics

Cost set against a unit of work, such as a resolved enquiry or a processed claim. It relates spending to value and is the figure a steering committee can act on, unlike cost per request.

Software Carbon Intensity (SCI)

The specification published as ISO/IEC 21031:2024: energy consumed times the carbon intensity of the electricity, plus embodied emissions of allocated hardware, divided by the functional unit. It rewards reduction, so purchased offsets do not lower the score.

Interactive exercises

Do them here and get feedback at once. Choose a card and then where it belongs, or drag it.

Teaching plan

Discussion

1. A two-thousand-token system prompt sent with every request is called a fixed cost nobody chose deliberately. Which costs in your AI services did nobody choose?
2. The fallback on budget exhaustion is the difference between a cost policy and a cost surprise. What should your most used AI service do when its budget is exhausted?
3. Which functional unit would make cost and carbon comparable across your AI use cases, and who would have to agree to it?

Written exercise

Self-test

Choose an answer to see whether it is right, or open the answer below each question.

1. What did the weekly showback report at Nordvik Region reveal about the eligibility-explanation route?

   1. It had the most requests and the lowest cost per resolved enquiry of all routes in the routing policy
   2. It was served entirely from the exact-match cache at no cost
   3. It exceeded the hard cap of the monthly budget in every week
   4. With the fewest requests it carried 40 per cent of spend for 13 per cent of resolved enquiries
   Show the answer

   D. With the fewest requests it carried 40 per cent of spend for 13 per cent of resolved enquiries
   The finding only becomes visible when cost is set against a unit of work, which is why the chapter calls unit economics the most useful line in a cost policy.

2. What does the chapter call the most common mistake in GPU capacity planning?

   1. Buying on-demand capacity for the interactive peak
   2. Sizing for the interactive peak and leaving capacity idle at night
   3. Scheduling batch work into the troughs of interactive load
   4. Measuring quality after adopting quantisation
   Show the answer

   B. Sizing for the interactive peak and leaving capacity idle at night
   Scheduling batch work into the troughs raises utilisation, and utilisation is the largest determinant of cost per request for self-hosted models.

3. How can a Software Carbon Intensity score be improved, according to the chapter?

   1. Only by less energy per unit, cleaner energy or less allocated hardware; offsets do not lower it
   2. By purchasing offsets equal to the operational emissions of the period and recording them in the calculation
   3. By excluding the hosted model routes from the boundary of the calculation
   4. By changing the functional unit from resolved enquiries to requests
   Show the answer

   A. Only by less energy per unit, cleaner energy or less allocated hardware; offsets do not lower it
   SCI rewards reduction, and each way of improving it maps to a lever: routing to a smaller model, a low-carbon grid, and batching.