Inside AI Hardware: Chips and Data Centers · Course book
Chapter 1: From Sand to Silicon Brain
Every answer from an AI system is, in the end, arithmetic carried out by billions of microscopic switches on a slab of silicon. Making those chips is among the most difficult things people do, and it depends on a chain of specialised companies spread across the world. This chapter follows a chip from the physics of a single switch, through design and manufacture, to the finished package, and shows why a handful of links in that chain matter so much.
The transistor: a switch with no moving parts
A transistor is an electrical switch. It has a channel through which current can flow and a gate that controls the channel. Put a voltage on the gate and the channel conducts, which is "on"; remove it and the current stops, which is "off". On and off stand for the 1s and 0s of digital computing, and a modern AI chip holds tens of billions of these switches.
For decades the route to faster, cheaper chips was simply to make transistors smaller. But as the channel shrinks, the gate's grip on it weakens and current leaks through even when the switch is meant to be off. Engineers answered by changing the transistor's shape so that the gate touches the channel on more sides, as Figure 1.1 shows.
In the original planar design the channel lies flat and the gate sits on top. In a FinFET the channel stands up like a fin and the gate drapes over three sides. In the newest gate-all-around design the channel is a stack of thin sheets, each one completely surrounded by the gate.
Why AI chips favour many slower cores
The power a chip uses for switching follows a simple rule: it is proportional to the capacitance being charged, to the square of the voltage and to the clock frequency, the number of switching cycles a second.
power = capacitance x voltage x voltage x frequency
Because voltage is squared, small reductions pay well. Cutting the voltage by 10 percent multiplies power by 0.9 × 0.9 = 0.81, a saving of 19 percent. Halving the frequency halves the power.
For many years a happy pattern held, known as Dennard scaling: each time transistors shrank, their voltage could be lowered in step, so a chip could hold more of them and run them faster without getting any hotter. Around the mid-2000s this broke down. Voltages could not go much lower without leakage becoming severe, so packing in more transistors and raising the clock speed now produced more heat than could be removed. Clock speeds stopped climbing.
The industry's response was to go wide instead of fast: many processing units working side by side. This suits AI perfectly, because the arithmetic of a neural network consists of enormous numbers of similar multiplications that can be done at the same moment. A graphics processing unit, or GPU, contains thousands of simple units running at a modest clock rate, where a general-purpose processor has a few complex ones running as fast as possible. More work gets done for each watt.
From a description to a set of masks
Nobody draws billions of transistors by hand. Engineers describe what the chip should do in a hardware description language, at a level called RTL, which reads rather like program code. Specialised software then does the translation in stages. Synthesis converts the description into a network of logic gates. Place and route decides where every gate sits and how the wires between them run. Verification checks, over and over, that the result behaves as intended and can actually be manufactured.
This software is called electronic design automation, or EDA, and nearly all of it comes from three suppliers: Synopsys, Cadence and Siemens EDA. The final output is a mask set, a series of patterns, one for each layer of the chip. A mistake found after this stage means repeating months of work, which is why so much effort goes into checking beforehand.
Building the chip, layer by layer
Chips are made on wafers, thin discs of extremely pure silicon, typically 300 millimetres across. The silicon is grown as a single crystal from molten material held in crucibles of very pure quartz, much of which comes from one small mining area in North Carolina. Manufacture takes place in a fab, inside cleanrooms where the air is filtered continuously, because a single speck of dust is a boulder at this scale.
A chip is built up in dozens of layers, each made by repeating the same cycle: deposit a thin film of material, coat it with a light-sensitive layer, project a pattern on to it, and etch away the exposed parts. The projection step is lithography, and it sets the limit on how small features can be, since finer detail needs light of a shorter wavelength. Earlier machines use deep ultraviolet light at 193 nanometres. The most advanced use extreme ultraviolet light, or EUV, at 13.5 nanometres, which is 193 ÷ 13.5 = about 14 times shorter. EUV is absorbed by air and by glass, so the whole light path sits in a vacuum and the beam is steered by extraordinarily smooth mirrors in place of lenses. One company, ASML in the Netherlands, makes these machines.
Not every chip on a wafer works. The fraction that does is the yield, and a simple model shows why big chips are costly. If defects fall at random with an average density D per square centimetre, the chance that a chip of area A has none is:
yield = e ^ (-A x D)
Take D = 0.1 defects per square centimetre. A small chip of 1 square centimetre has a yield of e-0.1 = 0.905, so about 90 percent are good. A large AI chip of 8 square centimetres has a yield of e-0.8 = 0.449, so fewer than half are good.
The effect compounds. A 300 millimetre wafer has an area of about 3.14 × 15 × 15 = 707 square centimetres. Ignoring the wasted edge, that gives room for 707 small chips but only about 88 of the large ones. At the yields above, the wafer delivers roughly 707 × 0.905 = 640 good small chips, or 88 × 0.449 = about 40 good large ones. The large chip is eight times the size yet costs about sixteen times as much per working unit.
Memory and advanced packaging
A processor is of little use without fast memory beside it, and AI workloads move vast amounts of data. Ordinary memory cannot deliver it quickly enough, so AI chips are paired with high-bandwidth memory. In this design several memory chips are stacked on top of one another and connected by through-silicon vias, vertical wires drilled straight through the silicon. Stacking shortens the connections and multiplies their number, so far more data can move each second.
The processor and its memory stacks must then sit extremely close together. In advanced packaging they are mounted side by side on a shared base layer, called an interposer, that carries thousands of fine connections between them. The finished assembly behaves almost like one large chip. This step has become as much of a constraint as making the chips themselves: there have been periods when completed processors waited because packaging capacity was full.
Packaged chips are finally built into servers, and servers into racks in which dozens of GPUs are linked so tightly that they work as one machine. A single such rack can draw more electrical power than a hundred homes, which is the subject of the next chapter.
A chain with narrow points
Put together, the chain looks like Table 1.1. Its striking feature is how few suppliers exist at several stages.
| Stage | What happens | Why it is a narrow point |
|---|---|---|
| Design software | A description of the chip is turned into mask patterns | Three companies supply almost all the tools |
| Materials | Ultra-pure silicon wafers, quartz, gases and chemicals are produced | Some come from very few sources worldwide |
| Lithography machines | Patterns are projected on to wafers | EUV machines come from a single company |
| Fabrication | Wafers are processed into chips over many weeks | Only a few firms can make the most advanced chips, and one makes most of them |
| Memory | High-bandwidth memory stacks are made | Three main producers |
| Packaging | Processor and memory are joined on an interposer | Capacity is limited and concentrated |
A point in a chain where one failure stops everything is called a choke point. A natural disaster, a factory fire or a political decision at any of these stages affects the whole industry. That is why governments now treat chip manufacturing as a strategic matter: some restrict the export of advanced chips and of the machines that make them, and many are paying to build fabs within their own borders. A new fab costs tens of billions and takes years to complete, so the map changes slowly.
Check your understanding
- What problem does wrapping the gate around more sides of the channel solve?
- If a chip's voltage is reduced by 20 percent and nothing else changes, what happens to its switching power? (Answer: it is multiplied by 0.8 × 0.8 = 0.64, a saving of 36 percent.)
- Why did the end of Dennard scaling push chip designers towards many parallel units?
- Why must the light path in an EUV machine be kept in a vacuum and use mirrors?
- With 0.1 defects per square centimetre, what is the yield of a chip of 2 square centimetres? (Answer: e-0.2 = 0.819, about 82 percent.)
- Name two stages of the chain that depend on three or fewer suppliers.
This course book is available to read online only at tech-academy.chrysolites.org.