By Gustavo TorresOn 11 October 1999, NVIDIA, a graphics card maker, launched the GeForce 256 and billed it as "the world's first GPU". By the definition NVIDIA wrote for the occasion, a GPU (graphics processing unit) was a chip that could draw 3D scenes on its own. What set the GeForce 256 apart was that it took over a calculation the computer's processor had always handled until then: multiplying every vertex in the scene by a 4×4 matrix to put it in front of the camera.
Twenty-five years later, Meta, the company behind Facebook, trained Llama 3, a language model with 405 billion parameters, on 16,384 NVIDIA H100 GPUs. The H100 is a direct descendant of that card, yet it can barely draw anything any more: it has no video output, and only 4 of its 132 compute blocks can run the programs a video game uses to paint the screen.
How does a gaming chip end up training language models? Why not use the processor every computer already has? And if artificial intelligence (AI) is worth so much money, why hasn't anyone built a chip just for it?
At heart, a video game and a language model do the same job: they multiply matrices, millions of times over, against the clock. The processor is built for almost the opposite job, which is why video games had to invent a chip of their own. Language models inherited that chip and, just as video games did before them, are now building their own.
The story runs twice, first for video games and then for language models, with the same steps each time: what gets computed, why it has to be fast, why the processor isn't enough and which chip was built to fill the gap. First, though, three properties of the operation that links the two stories, matrix multiplication. Every figure comes from the manufacturers' published specs or from the papers linked, and any figure derived from them comes with its working alongside.
The operation this is all about
Multiplying an matrix by a one takes about operations, half of them multiplications and half additions. These are floating-point operations (floating point is the format computers use to store numbers with decimals), which is why a chip's power is measured in FLOPS (floating-point operations per second). A TFLOPS is a trillion () operations per second, and a PFLOPS is a thousand trillion.
Three properties of this operation explain almost everything that follows.
Every number in the result is independent of the others. Each dot product can be computed at the same time as all the rest, on a different part of the chip and in any order.
There are no decisions. The recipe is always the same, multiply and add, with no "if this happens, do that" that depends on the data.
Every number is used many times. When you multiply two matrices, each number in the first is multiplied by numbers from the second. That's operations on input numbers, or operations per number: with thousand-by-thousand matrices, every number read from memory can feed a thousand operations, as long as it stays close at hand in the meantime.
All three hold for a video game just as much as for a language model. Only the size of the matrices changes.
Video games
What a video game computes
To a computer, a 3D world is a collection of triangles. A character, a wall or a tree is a mesh of thousands of triangles, and each triangle is defined by its three vertices, each with three coordinates: , and . To draw a frame (one of the still images that make up an animation when shown in quick succession), the game has two big jobs to do.
First, it places every vertex. A character's coordinates are stored relative to its own centre. To draw it, the game has to move it to its place in the world, rotate it, view it from where the camera is and project it onto the flat screen. Each of those steps is a multiplication by a matrix, and since multiplying by several matrices in a row is the same as multiplying by their product, the game combines them into one and applies it to every vertex of the object:
The matrix is 4×4 rather than 3×3 because of a trick. Add a fourth number to each vertex, always a 1, and moving an object becomes a multiplication too: the matrix's fourth column adds a fixed amount to each coordinate. Perspective, which makes distant things look smaller, works the same way: the fourth coordinate of the result, , grows with distance, and dividing and by it shrinks whatever is far away. That comes to 16 multiplications and 12 additions per vertex, with the same matrix for every vertex of the object.
Then, it colours every pixel. Once projected, each triangle covers a handful of pixels, the dots of colour that make up the image, and working out which ones is called rasterisation. Each of those pixels then needs a colour, and that's algebra too. The brightness of a matte surface, for example, depends on the angle between the direction it faces (its normal, a vector perpendicular to it) and the direction the light comes from, and that angle comes from a dot product. Textures (images stuck onto the triangles), shadows and reflections go on top, but the pattern doesn't change: the same calculation, on different data, for every pixel.
One frame, simplified. The two big loops, one over vertices and one over pixels, repeat the same small piece of algebra on data where nothing depends on anything else.
Both jobs have the shape of matrix multiplication: a small calculation, always the same, repeated over millions of elements that don't depend on one another. A pixel's colour doesn't wait for its neighbour's.
Why it has to be fast
A game doesn't draw one image: it draws sixty a second, or more. At 60 frames per second, each frame has 16.7 milliseconds to be ready, and if it's late the player sees a stutter. A Full HD screen (1920 × 1080) has 2,073,600 pixels, and a 4K one (3840 × 2160) has 8,294,400. At 4K and 60 frames per second, that's almost 500 million pixels a second, each with its own colour calculation, and that's without counting the ones that get drawn and then hidden behind something closer.
Nor can any of the work be done in advance, because nobody knows what the next frame will hold until the player moves the camera. Animated films can afford to wait: each frame of Toy Story (1995) took Pixar between 45 minutes and 30 hours of computing on a cluster of 117 Sun workstations. A game has to do something similar in 16 milliseconds, and keep doing it.
Notice what actually has the time limit. No pixel is in a hurry on its own: the deadline applies to the frame as a whole, and it doesn't matter if one pixel takes a long time as long as all two million finish in time. What a game needs is throughput, how much work gets finished per second, rather than low latency, how long each individual task takes. That distinction decides which chip is up to the job.
Why the CPU couldn't keep up
The CPU (central processing unit) is the computer's general-purpose processor: it runs the operating system, the browser and the game's logic. It's designed to run a single thread of instructions (a sequence in which each step usually depends on the one before) as fast as possible, and to do that it devotes a good part of the chip to avoiding waits.
Caches. Reading a piece of data from main memory takes on the order of a hundred nanoseconds, time in which a core could have done hundreds of operations. Caches are small, fast memories on the chip that keep recently used data in case it's needed again.
Branch prediction. An ordinary program is full of decisions: if the player presses the button, the character jumps. The CPU bets on which way each decision will go and carries on working before it knows. If it guessed wrong, it throws that work away.
Out-of-order execution. If an instruction is waiting for data, the CPU looks further ahead for others that don't depend on it and runs those first.
All of this makes a program full of decisions and unpredictable data run fast, which describes nearly all software. None of it helps with painting pixels: there are no decisions to guess, the data needed is known in advance, and millions of independent calculations are waiting their turn. A core is the part of the CPU that runs one thread, and the CPU of 1996 had just one, which spent most of its area making that thread fast rather than adding and multiplying.
Games really were drawn by the CPU for years. Quake, id Software's shooter from June 1996, did everything on it. The box asked for a Pentium, Intel's processor of the day, running at 75 MHz or more, and a 120 MHz Pentium managed somewhere around 30 frames per second. Every extra frame, or extra pixel, had to be paid for with a more expensive processor.
Intel's response, in 1997, was MMX, its first set of SIMD instructions (single instruction, multiple data), aimed at multimedia and games: a single instruction works on several numbers at once, four additions in one go, say. SSE (Streaming SIMD Extensions) followed in 1999 and did the same with decimal numbers, four at a time. It helped, but a core with SIMD still did only a handful of operations per clock cycle, and the pixels ran into the millions. And the CPU had other jobs to do as well: physics, sound, the enemies' behaviour.
What was needed was a chip that made the opposite bet: give up on making any single task fast in exchange for doing a huge number of them at once.
A chip of its own: the GPU
The first answer was the 3D accelerator card, a board added to the computer purely for drawing. Voodoo Graphics, from a company called 3dfx, arrived at the end of 1996 and did half the job: it rasterised the triangles and applied their textures, but the CPU still placed the vertices. In January 1997 id Software released GLQuake, a version of Quake that drew with these cards, and the difference was obvious at a glance: higher resolution and smoother textures.
In 1999, the GeForce 256 took over the other half as well. NVIDIA described it as a single chip with "integrated transform, lighting, triangle setup/clipping, and rendering engines", able to process at least 10 million polygons a second. Transform means the multiplication by the 4×4 matrix: in 1999, the operation this post is about left the CPU for a chip of its own.
Those chips were fixed-function: each stage did exactly what its designers had hard-wired it to do. In 2001 the GeForce 3 introduced programmable shaders, small programs written by the game itself that the GPU runs once for every vertex or every pixel. From then on, the GPU would run any calculation it was given, on one condition: that it be the same for millions of elements. In 2006 the GeForce 8800 merged the two kinds of shader into a single kind of core, and it had 128 of them, serving vertices and pixels alike.
That's the origin of the design GPUs still have today, and it's the opposite of a CPU's.
Many simple cores. A desktop CPU has one or two dozen big cores, and the largest server CPUs a few hundred. The GeForce RTX 5090, NVIDIA's most powerful gaming card, from 2025, has 21,760 small ones, which together do 104.8 TFLOPS. At 4K and 60 frames per second, that's about 210,000 operations per pixel per frame.
One instruction for many threads. The cores are grouped into multiprocessors (the compute blocks from the opening), and on NVIDIA GPUs every group of 32 threads, a warp, runs the same instruction at the same moment, each thread on its own data. NVIDIA calls this SIMT (single instruction, multiple threads). Fetching an instruction and deciding what to do with it costs the same for one thread as for 32, so almost the whole chip can be devoted to calculation.
Waiting without stalling. A GPU doesn't try to dodge memory waits with huge caches and clever predictions. Instead it keeps thousands of threads in flight, and when one warp is waiting for data, the scheduler switches to another whose data has already arrived, at no cost. That works because graphics always has millions of threads to offer.
Very wide memory. The RTX 5090 reads 1,792 GB per second from its memory. A desktop with two DDR5-5600 modules, the kind of memory most computers use today, manages 89.6 GB/s at most.
Schematic, not to scale. The CPU spends its chip on keeping one thread from waiting; the GPU spends it on arithmetic for thousands of threads and on a memory that can keep it fed.
The price is that a GPU is bad at what a CPU does well. A single thread crawls along, and if the threads in a warp go different ways at a decision, the warp runs both paths one after the other. That's why a computer has both: the CPU for the program, the GPU for the millions of identical calculations.
From drawing to computing
As soon as shaders became programmable, some researchers realised that a GPU was, underneath, a machine for doing the same operation on millions of numbers, faster than any CPU. All they had to do was dress the problem up as drawing. In 2001, E. Scott Larsen and David McAllister, at the University of North Carolina, published Fast matrix multiplies using graphics hardware: they stored the matrices as textures and let the stage that combines textures do the multiplications and additions. In the words of their abstract, they got the result by literally visualising the computation.
It worked, but every calculation had to be disguised as a picture. In 2007 NVIDIA released CUDA (Compute Unified Device Architecture), which let programmers write for its GPUs in a variant of C, with no textures or pixels involved: you write a function and ask for it to run on a million threads at once. The first GPU to support it was the GeForce 8800, a gaming card.
Neural networks, programs that learn from examples by adjusting millions of numbers called parameters, were among the first to benefit, because under the hood they are mostly matrix multiplications. In 2009, Rajat Raina, Anand Madhavan and Andrew Ng, at Stanford, trained networks on GPUs up to 70 times faster than on a dual-core processor: a model with 100 million parameters went from several weeks of training to a day. And in 2012 AlexNet, the network built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, won ImageNet, an image recognition competition, by a wide margin. They trained it for five or six days on two GeForce GTX 580s, gaming cards with 3 GB of memory each, and had to split the network between them because it didn't fit on one. That's where modern deep learning, with its many-layered networks, took off, and it took off on hardware bought at a games shop.
Language models
What a language model computes
A language model is a neural network that, given a piece of text, works out what is likely to come next. An LLM (large language model) is one with billions of parameters, trained on a huge share of the public text on the internet. The text goes in chopped into tokens, which are pieces of words (a short word is usually a single token, a long one several), and the answer comes out one token at a time: the model works out the next token, appends it to the text and starts over.
Inside the model, each token is a vector: in Llama 3 405B, the model from the opening, a vector of 16,384 numbers. Almost everything the model does with those vectors is multiply them by matrices. Nearly every LLM today is a transformer, the architecture published in 2017, and a transformer is a stack of identical layers, 126 in Llama 3 405B, each with two parts.
Attention lets each token look at the ones before it. Each token's vector is multiplied by three matrices to produce three new vectors: a query (what it's looking for), a key (what it offers) and a value (what it contributes if chosen). Comparing each token's query with the keys of all the earlier tokens is another matrix multiplication; mixing their values according to that comparison is one more; and a fourth matrix brings the result back to 16,384 numbers. In Llama 3 the key and value matrices are narrower, only 1,024 columns wide, thanks to an optimisation that saves memory.
The feed-forward network handles each token separately: it multiplies its vector by a matrix that widens it from 16,384 to 53,248 numbers, applies a simple function to each number and multiplies the result by another matrix that brings it back to 16,384. Llama 3 uses a variant with three matrices instead of two.
At the top of the stack, one last matrix turns the vector into a score for each of the 128,000 tokens in the vocabulary, and the next token is chosen from those scores.
One layer of Llama 3 405B. Whatever isn't matrix multiplication (normalising, turning the comparisons into weights, the simple function) comes to only a few operations per number.
The matrices are the parameters. Those in one layer of Llama 3 405B add up to about 3.2 billion numbers, and the 126 layers, plus the vocabulary matrices, make the 405 billion in the name. Every parameter takes part in one multiplication and one addition for each token that goes through the model, so processing a token costs about operations, where is the number of parameters: about 810 billion for Llama 3 405B.
Training costs three times as much per token. During training, the model reads a text, predicts each token from the ones before it, measures how wrong it was and adjusts its parameters to do better next time. Besides the forward pass, which makes the prediction, there's a backward pass, which works out which way to nudge each parameter and costs twice as much as the first. The total is about operations per token. Llama 3 405B was trained on 15.6 trillion tokens:
That's a 38 followed by 24 zeros, matching the figure Meta publishes.
It's the same operation as in a video game, just on a different scale in two ways. The matrices aren't 4×4 but thousands by thousands. And they aren't multiplied by one vector at a time but by many: each of Llama 3's training steps processed 16 million tokens at once, so the tokens' vectors are stacked into a matrix and every multiplication is matrix by matrix.
Why it has to be fast
Meta trained Llama 3 405B on 16,384 H100 GPUs, and each one sustained about 400 TFLOPS, 41% of its theoretical peak. At that rate, a single H100 would have needed about 3,000 years to do the operations. All 16,384 together needed about 67 days.
With that many components, something is always breaking. Meta reports 466 interruptions during a 54-day stretch of training, 419 of them unplanned, with more than half of those caused by failures of the GPUs themselves: an interruption every three hours or so, on average. Each one sends the run back to the last saved checkpoint.
And a model is trained once but used millions of times. Using it is called inference: the trained model generating answers. Every generated token costs operations, and there's a deadline here too, just like the frame's: a chat that writes at 20 tokens per second has 50 milliseconds for each token. And that's not for one person, but for everyone using it at the same time.
The parallel with video games is exact: the same calculation over millions of pieces of data, against a deadline that comes round every few milliseconds.
Why the GPU fits, and the CPU doesn't
The three properties from the start explain why the GPU is such a good fit.
The results are independent, like pixels, so thousands of cores can each work out their own piece at the same time.
There are no decisions, again like pixels, so a CPU's branch prediction and out-of-order execution have nothing to guess and nothing to reorder.
Every number is reused many times. When a 16,384 × 16,384 matrix is multiplied by another of the same size, every number read from memory feeds 16,384 operations. If the chip can keep those numbers close to the arithmetic units while it uses them, its speed depends on how much it can compute rather than how much it can read. This is measured by the arithmetic intensity of a calculation: how many operations it performs for each byte it reads from memory. It will come up again shortly.
And what about the CPU? Today's isn't the one from 1996. Just as Intel added MMX for games, its Xeon server processors have shipped since 2023 with AMX (Advanced Matrix Extensions), a matrix multiplication unit inside every core. Intel credits it with a peak of 1,024 operations per cycle per core on 16-bit numbers. The Xeon 6980P, from 2024, has 128 cores at 2 GHz, so on paper it reaches about 262 TFLOPS.
Comparing it with GPUs requires one more ingredient: precision. BF16 is a 16-bit format for decimal numbers, half the usual 32 bits; FP8 and FP4 use 8 and 4 bits; and INT8 means 8-bit integers, whose speed is measured in TOPS, trillions of integer operations per second. Fewer bits mean less precision, but more operations per second and fewer bytes to move. Neural networks tolerate that loss well, which is why AI chips compete at these precisions. Here's the Xeon next to NVIDIA's data-centre GPUs, the H100 and the B200:
| Xeon 6980P (CPU, 2024) | H100 (GPU, 2022) | B200 (GPU, 2024) | |
|---|---|---|---|
| Matrices in BF16 | ≈ 262 TFLOPS | 989 TFLOPS | 2,250 TFLOPS |
| At its lowest precision | INT8: ≈ 524 TOPS | FP8: 1,979 TFLOPS | FP4: 9,000 TFLOPS |
| Memory bandwidth | 845 GB/s | 3,350 GB/s | 8,000 GB/s |
| Memory | up to 3 TB | 80 GB | 180 GB |
| Power | 500 W | 700 W | ≈ 1,000 W |
All three columns are theoretical peaks that nobody reaches in practice, but they're good enough for a comparison.
The comparison makes three points. First, the gap is far smaller than it used to be. In 2009, Raina and his co-authors measured a GPU up to 70 times faster than a dual-core CPU; today, on paper, an H100 multiplies BF16 matrices about four times as fast as a 128-core Xeon, and a B200 about nine times as fast. At each chip's lowest precision the gap widens again: a B200 in FP4 does 17 times as many operations as the Xeon in INT8.
Second, the Xeon's memory lags by a similar margin, with four to ten times less bandwidth. And per watt, the H100 computes almost three times as much as the Xeon.
The third point isn't in the table: how many of them you can link up. A server board takes two Xeon 6980Ps at most. The servers Meta trained Llama 3 on had eight GPUs and two CPUs each, and the eight GPUs talked to each other over NVLink, a direct link between NVIDIA GPUs that moves 900 GB/s on the H100. The CPUs are still there, but with a different job: preparing the data, coordinating and handing out work to the GPUs.
There's a fourth reason, and it's less technical. For more than a decade, AI software, from the matrix libraries up to PyTorch, the Python library almost every model is trained with, has been written for NVIDIA GPUs first. By the time CPUs added AMX, the rest of the ecosystem had long since moved elsewhere.
There's one thing the CPU does win at, though: capacity. A Xeon can take up to 3 TB of memory, nearly forty times as much as an H100. That matters more than it seems, because training a model and using it aren't the same problem.
Training is not the same as inference
Training makes three demands at once.
Compute. With batches of millions of tokens, every multiplication is matrix by matrix and the arithmetic intensity is very high: the chip spends nearly all its time computing. It's exactly what GPUs are designed for.
Memory. Training needs more than the parameters. It also needs each one's gradient (which way to move it) and the state of the optimiser, the algorithm that decides how far to move it; the most widely used one, Adam, keeps two averages per parameter. With the usual recipe, which computes in 16 bits but keeps a copy of the parameters in 32, that's 16 bytes per parameter: 6.5 TB for Llama 3 405B, enough to fill 81 H100s all by itself, before counting each layer's intermediate results.
Network. Meta split the model three ways at once: each matrix sliced across the 8 GPUs in a server, the layers divided into 16 groups that pass the work along like an assembly line, and 128 copies of all that processing different data (). At every step the GPUs exchange results, over NVLink inside a server and over the network between servers. A large training run is as much a communications problem as a computing one, and Meta notes that its complexity exceeds that of the much larger CPU clusters it had already run.
Inference has two phases, and they behave very differently.
Prefill reads the question. All its tokens are known from the start, so the model processes them together, matrix by matrix, as in training, and compute is the limit.
Decoding writes the answer, and here everything changes. Each new token depends on the previous one, so tokens are generated one at a time, and each one has to go through every matrix in the model. It's matrix by vector: each parameter is read from memory for a single multiplication and a single addition. With 16-bit parameters, that's two operations for every two bytes, or one operation per byte.
An H100 performs 989 trillion operations per second and reads 3.35 trillion bytes. To keep its arithmetic busy it needs about 295 operations for every byte it reads. Decoding for a single person manages one, using 0.3% of the chip's compute. Memory sets the speed: Llama 3 8B, the small version, takes up 16 GB in 16 bits, and an H100 can read that only about 200 times a second, so it can't generate more than about 200 tokens per second for one person, whatever else it does. Llama 3 405B, at 810 GB in 16 bits, doesn't even fit in a server with eight H100s, so Meta spreads it over the 16 GPUs of two servers.
BF16, from published specs. Left of the knee, memory sets the speed; right of it, arithmetic does. Decoding for one person sits far to the left.
That chart is the roofline model, named for its shape, and it's the most useful tool there is for reasoning about a chip's speed: a calculation to the left of the knee is limited by memory, and one to the right by arithmetic. The standard fixes for decoding all come down to moving to the right or reading less.
Serving many users at once (batching). If the server generates the next token for 64 conversations together, every parameter it reads serves 64 multiplications, so the intensity rises to 64. It takes about 300 to saturate an H100.
The KV cache. To avoid recomputing the keys and values of all the previous tokens at every step, the server keeps them in memory (KV stands for key and value). That saves compute but takes up memory for every conversation, and it's what limits how many fit at once.
Quantisation: storing the parameters in 8 or 4 bits instead of 16. That leaves half or a quarter as many bytes to read, so decoding runs nearly twice or four times as fast, at some cost in precision.
And this is where the CPU comes back into the picture. Because decoding is limited by memory rather than compute, a machine with fast memory does reasonably well at it. The Xeon 6980P, with its 845 GB/s, can read Llama 3 8B in 16 bits about 50 times a second. A laptop with two DDR5-5600 modules, running the model quantised to 4 bits (about 4 GB), reaches about 20 tokens per second on paper. That's why programs like llama.cpp, which runs language models on just about any computer, work on laptops with no GPU. Training a model that way is another story.
Training and inference need different things: training wants compute, memory for all its state and a network linking thousands of chips; inference wants, above all, bandwidth and a low cost per token. That difference explains much of what has been built since.
A chip of its own, again
Video games had to invent a chip because the CPU wasn't up to the job. Language models found one ready-made, and since then history has repeated itself twice over: the GPU has gradually turned into an AI chip, and chips designed purely for AI have appeared.
The GPU specialises. In 2017, NVIDIA's V100 introduced Tensor Cores, units that multiply not two numbers but two small matrices at once: each one computes on 4×4 matrices in a single step, 64 multiply-adds per cycle. The V100's 640 Tensor Cores delivered 125 TFLOPS, eight times what the same card's ordinary cores could manage. This is the third property of matrix multiplication built into the circuit itself: if every number is going to be used many times, let the hardware reuse it without reading it again.
Lower precisions followed: FP8 on the H100 (2022) and FP4 on the B200 (2024), each doubling the operations of the one before. Alongside them came HBM (high bandwidth memory), stacked right next to the chip so it can be read faster, and NVLink, so that many GPUs behave a little more like one. Along the way, the data-centre GPU left graphics behind: according to the H100 whitepaper, only 4 of its 132 multiprocessors can run the shaders that draw a video game. It's a matrix multiplication chip that kept its old name.
Chips of their own. The first to reach production was Google's. In 2013, an internal forecast showed that if every user spoke to voice search, which ran on neural networks, for just three minutes a day, Google would have to double its data centres. So it set about designing its own chip for inference (for training, it bought GPUs) and had it running within 15 months. The TPU (tensor processing unit) is an ASIC (application-specific integrated circuit), a chip designed for a single task, and it has been in Google's data centres since 2015.
Its heart is a systolic array, an idea dating from the late 1970s: a grid of 256 × 256 cells (65,536 in all) that multiply and add, with data flowing from cell to cell. The parameters are loaded into the cells and the inputs enter from the left. Each cell multiplies the input arriving at it by its parameter, adds the product to the partial result handed down from the cell above and passes the sum to the cell below, while the input carries on to the right. A number read from memory is used 256 times without going back to memory. According to the paper in which Google described it, the TPU was on average 15 to 30 times faster than the CPU and GPU of its day, and delivered 30 to 80 times as many operations per watt.
Three ways to do the same operation. From left to right, each number read from memory is reused more times inside the circuit.
Google now trains its Gemini models on TPUs, and the seventh generation, Ironwood (2025), delivers 4,614 TFLOPS in FP8 with 192 GB of HBM at 7.4 TB/s, linked in groups of up to 9,216 chips. Google presented it as its first TPU designed mainly for inference. And Google isn't the only one using them: in October 2025, Anthropic, the company that makes the Claude models, announced plans to use up to a million.
The other tech giants have their own: Amazon has Trainium for training and Inferentia for inference; Meta has MTIA (Meta Training and Inference Accelerator); Microsoft has Maia. And several start-ups have made riskier bets.
- Cerebras makes a chip the size of an entire silicon wafer, with 900,000 cores and 44 GB of SRAM (the fast kind of memory that caches are made of) on the chip itself.
- Groq designed an inference chip that keeps the parameters in SRAM instead of HBM, so decoding never has to wait on memory. In December 2025, NVIDIA took a non-exclusive licence to its technology and hired its founder, in a deal the press valued at around $20 billion.
- Etched announced a chip in 2024 that can only run transformers, trading away everything else to do that one thing faster.
In phones and laptops, NPUs (neural processing units), small matrix units inside the processor, run models without draining the battery on the GPU. Many of these chips started with inference, and not by chance: it's the problem from the previous section, the one that depends on bandwidth and on the cost of each token, and where a general-purpose GPU leaves the most room for improvement.
Why the GPU is still in charge. Despite all these specialised chips, NVIDIA's GPUs still account for more than 60% of all the AI computing capacity sold since 2022, according to estimates by Epoch AI, an independent research group that tracks trends in the field, with Google and Amazon making up much of the rest. There are three reasons.
Software. CUDA has been around since 2007, and everything written on top of it (matrix libraries, PyTorch, each lab's own tricks) works there first. A new chip needs its own compiler, its own libraries and programmers who know how to use it, and Google had to build all of that for the TPU.
Time. Designing a chip and getting it into production takes years, and models change faster than that. The first TPU was designed for the networks Google was running at the time, three kinds that made up 95% of its inference workload, and none of them was a transformer, which arrived two years later. The TPU survived the change because it had bet on matrix multiplication rather than on any particular network. A chip that runs only transformers, like Etched's, is the opposite bet: if the architecture changes, the chip is left behind. The GPU, being programmable, can adapt to whatever comes next.
Scale. For twenty years video games paid for the GPU's development; now data centres pay for it. And what sells by the million gets cheaper and better faster than what sells by the thousand.
That's the big difference from 1999. Back then the CPU couldn't do the job; today the GPU does it well. A new chip isn't up against a processor that can't cope, but one that improves every two years and absorbs its rivals' ideas along the way: tensor cores are, in miniature, the same idea as the TPU's systolic array, and NVIDIA already holds a licence to Groq's technology.
Not to scale. The same operation moves from the CPU to a gaming chip and, from there, to chips that do nothing else.
What to take away
If you remember only one thing, make it this: the GPU didn't end up in AI by accident. A video game and a language model do the same kind of work: the same small operation on millions of independent pieces of data, with no decisions and every number reused many times. The CPU is designed for the opposite: a single fast thread of instructions, full of decisions. Video games had to invent the GPU, and AI inherited it.
The same steps, side by side:
| Video games | Language models | |
|---|---|---|
| What is computed | Every vertex times a 4×4 matrix, and the colour of every pixel | Every token times matrices thousands of rows by thousands of columns, layer after layer |
| How much | Almost 500 million pixels a second, at 4K and 60 frames per second | operations to train Llama 3 405B; per generated token |
| Time limit | 16.7 ms per frame | Tens of milliseconds per token |
| Why not the CPU | One core, built for a thread full of decisions | Four to ten times less compute and bandwidth, and only two per server |
| A chip of its own | The GPU (1999) | Tensor cores, the TPU and the rest (from 2015) |
And if you have room for three more, make them practical ones.
Always ask what's holding you back: compute or memory. Count how many operations a calculation performs for each byte it reads, and compare that with how much the chip can compute for each byte its memory delivers. If the calculation falls short of that ratio, more arithmetic won't make it any faster.
When generating text, count bytes, not operations. Decoding speed for one person is at most the memory bandwidth divided by the size of the model. That's why quantisation speeds things up, and why a laptop with good memory can run small models.
The name "GPU" is a historical leftover. What matters in an AI chip is how many matrix multiplications it can do per second and at what precision, how many bytes per second it can be fed, and how many other chips it can be linked to. And behind all that, the software that knows how to use it.
In 1999, a card for playing games took a multiplication by a 4×4 matrix off the CPU's hands. Twenty-five years later, that same operation, on matrices thousands of rows tall, is what trains language models. The chips have changed in size, in name and even in maker; the operation is the same.