Building with AI Coding Agents · Course book
Chapter 4: From a Model to an Application
Coding agents and chat assistants are finished products. Underneath each one is a model reached through an API, wrapped in a program that manages the conversation, runs tools and enforces limits. Once you understand those layers you can build AI features of your own, switch between providers without starting again and predict what a feature will cost before the bill arrives. This chapter explains how to choose a model, what an API call contains, how cost is counted, and what a harness adds to turn a model into an agent.
Choosing a model
Providers usually offer a family of models and not a single one, and the members differ along three lines that pull against each other: capability, speed and price. The largest model reasons best and costs the most for each token. The smallest answers quickly and cheaply and manages simpler work perfectly well.
There is a further distinction in how a model is supplied. A hosted model runs on the provider's computers, and you pay for each use. An open-weights model can be downloaded and run on your own hardware; the weights are the learned numbers that make up the model. Running your own gives you control and privacy, in exchange for the cost and effort of operating the machines.
| Kind of model | Good at | Trade-off | Typical use |
|---|---|---|---|
| Large hosted model | Hard reasoning, long tasks, writing complex code | Highest price and slowest replies | Planning, difficult debugging, agent work |
| Small hosted model | Quick, simple tasks in great numbers | Less capable on difficult problems | Sorting, extracting fields, short replies |
| Open-weights model run yourself | Privacy, control, no charge per token | You pay for and maintain the hardware | Sensitive data, offline use, experiments |
Published benchmarks, which are standard test sets with scores, help to compare models but deserve some caution. A benchmark measures particular tasks that may not resemble yours, and models can be tuned to do well on famous ones. The dependable method is to collect twenty or thirty examples of your own task, run each candidate on them and compare the results. A sound default is to start with the smallest model that passes your examples and move up only where it falls short.
What an API call contains
An API is the doorway through which your program talks to the model. A request is a short structured message, and it has three essential parts: which model to use, the conversation so far and some settings.
{
"model": "example-model",
"max_tokens": 500,
"messages": [
{ "role": "system", "content": "You are a helpful assistant for a bakery." },
{ "role": "user", "content": "What time do you open on Saturday?" }
]
}
The conversation is a list of messages, each with a role. The system message sets the model's standing instructions. User messages are what the person said. Assistant messages are the model's earlier replies. The response carries the new assistant message and a count of the tokens used.
Two facts about this arrangement surprise newcomers. First, the model is stateless: it remembers nothing between calls. The appearance of memory in a chat is produced by your program sending the whole conversation again every time. Second, the request format has become something of a common language. Many providers accept requests in the same shape as the most widely used client libraries, so moving to a different provider can be as small a change as pointing the library at a new address and changing the model's name. Writing your code against that common shape keeps your options open.
Counting the cost
Hosted models are billed by the token, with separate prices for input (what you send) and output (what the model writes), each quoted per million tokens. Output usually costs several times more than input. Prices vary between models and change over time, so the figures below are round illustrative numbers: 3 units of currency per million input tokens and 15 per million output tokens.
Take one call with 2,000 input tokens and 500 output tokens.
- Input: 2,000 ÷ 1,000,000 × 3 = 0.006
- Output: 500 ÷ 1,000,000 × 15 = 0.0075
- Total: 0.0135 for the call
That looks negligible, and for one call it is. A feature used 10,000 times a day costs 10,000 × 0.0135 = 135 a day. Notice also that the 500 output tokens cost more than the 2,000 input tokens.
Statelessness makes long conversations dearer than they seem. Suppose the system message is 1,000 tokens, each turn brings 500 new input tokens, and each reply is 300 tokens. Because everything is sent again, the input for turn t is 700 + 800 × t tokens.
- Turn 1: 1,500 input tokens
- Turn 10: 8,700 input tokens
- Turn 20: 16,700 input tokens
Across all 20 turns the input adds up to 182,000 tokens. Had every turn been as short as the first, it would have been 20 × 1,500 = 30,000. Resending the history multiplied the input roughly sixfold.
This is why long-running tools use context compaction. When the history grows large, the older turns are replaced by a brief summary. In the figure, compaction after turn 10 swaps 8,000 tokens of history for a 1,500-token summary, so turn 11 sends 3,000 tokens instead of 9,500. Detail is lost, which is the price, so anything that must survive, such as a decision already made, should be written somewhere permanent.
From a model to an agent
A model on its own can only produce text. It becomes an agent through tool calling. Along with the messages, your program sends a list of tools, each with a name, a description and the inputs it accepts. The model cannot run anything itself. What it can do is reply with a structured request: call this tool with these inputs. Your program runs the tool and sends the result back as a new message, and the model carries on.
Repeat that exchange in a loop and you have an agent. The program that runs the loop is called a harness. It holds the conversation, offers the tools, carries out the calls, applies the rules and decides when to stop. A coding agent in a terminal is a harness whose tools read files, edit files and run commands. You can write a harness yourself in a few dozen lines, or use an agent framework, a library that supplies the loop and common tools so that you write only what is particular to your task.
Harnesses share a set of controls that are worth recognising, whatever they are called in a given product.
- A project instruction file read at the start of every session, holding the conventions of the project.
- Skills, packaged instructions loaded only when a task needs them.
- Task breakdown, in which a large goal is split into a checklist of smaller steps that are worked through and ticked off.
- Reasoning effort, a setting for how much the model thinks before replying. More effort improves hard answers and costs more time and tokens, so it is wasted on easy ones.
Keeping an agent within bounds
A loop that runs commands needs limits that do not depend on the model behaving well. Four are standard.
Sandboxing runs the agent's commands in a restricted environment. The agent sees only the project folder, and access to the rest of the machine and to the network is blocked unless granted. If the agent is misled or mistaken, the damage is confined.
Approval rules state which actions run freely and which wait for a person. Reading is normally free; deleting, installing and sending are normally not.
Budgets cap the number of steps, the tokens or the time a task may use. Without one, an agent stuck repeating a failing command will continue until someone notices. With one, it stops and reports.
Connection standards. The Model Context Protocol gives agents a uniform way to reach outside systems such as databases and issue trackers. Each server adds tools, and each tool's description takes up room in the context window, so connecting many servers "in case they are useful" makes an agent both slower and less focused. Connect what the task needs.
Check your understanding
- Why is it sensible to begin with the smallest model that passes your own test examples?
- What does it mean that a model is stateless, and how does a chat appear to remember earlier turns?
- At 3 per million input tokens and 15 per million output tokens, what does a call with 10,000 input and 1,000 output tokens cost? (Answer: 0.03 + 0.015 = 0.045.)
- Using the formula 700 + 800 × t, how many input tokens does turn 15 send? (Answer: 700 + 12,000 = 12,700.)
- In tool calling, who actually runs the tool: the model or your program?
- What does context compaction gain, and what does it lose?
- Name two limits a harness can enforce that do not rely on the model's good behaviour.
This course book is available to read online only at tech-academy.chrysolites.org.