How a Pile of Random Numbers Learns to Write

When Jacob Coxon walked away from Anthropic, the news barely caused a ripple—but it really should have. He worked as a 'pretraining researcher,' a title that sounds like dull corporate jargon until you realize what that work actually involves. This single phase is where an AI transforms from a jumble of raw numbers into a system capable of holding a real conversation. It’s the invisible foundation for everything currently dominating the debate in Congress, and it’s where our story begins.

From Bits to Intelligence

A computer only ever works with 0s and 1s, called bits. String enough of them together and you can represent any number, and once you can represent numbers, you can represent anything: letters, images, sound. That part is not mysterious. It is the same principle as counting, just in a different base.

Understanding Neural Networks

The leap from "numbers" to "something that writes like a person" happens through a structure called a neural network. Picture a massive grid of simple switches, called nodes, arranged in layers. Every connection between two nodes carries a number called a weight, essentially a measure of how much that input should matter. Feed data in one end, and it passes through layer after layer, getting multiplied and added up according to those weights, until something comes out the other side. A modern language model has billions of these weights. That pile of numbers is the entire model. There is no code sitting inside it that says "if someone asks a question about history, answer this way." There is only the weights, and what they do to whatever data passes through them.

The Mechanics of Pretraining

So where do the weights come from? At the very start, they are essentially random, which means the model's output is nonsense. Pretraining is the process that fixes that, and it works through nothing more sophisticated than repeated trial and error. Show the model the start of a sentence and ask it to guess the next word. It guesses wrong. Calculate exactly how wrong, then nudge every single weight a tiny amount in the direction that would have made the guess less wrong. That nudging process is called backpropagation. Do it once, and nothing changes. Do it trillions of times, across enormous amounts of text scraped from books, articles, and websites, and the pile of random numbers slowly reorganizes itself into something that can predict language well enough to write like a person.

Key Components: Embeddings and Attention

Two more pieces complete the picture. Words get broken into chunks called tokens, which are then converted into something richer called embeddings: each word gets placed as a point in a huge mathematical space, positioned so that related words end up near each other and unrelated words end up far apart. And the technique that actually separates modern models from older, clumsier AI is called attention, part of what is known as the transformer architecture. Attention lets the model weigh which earlier words in a passage matter most to understanding the word it is working on right now, which is how a model can track that "it" three sentences later still refers to the thing you mentioned at the start.

Why Pretraining Matters for Policy

Here is why this stage matters more than people realize. Anthropic's own researchers recently found that some of the more concerning behaviors they caught in testing traced back to pretraining itself, not to anything that went wrong in the safety training layered on afterward. In their own words, the safety training that came later "did not include any agentic tool use," and proved insufficient on its own to prevent problems that had already taken root earlier in the process. In plain terms: what a model absorbs during the trillions of trial-and-error adjustments of pretraining shapes its behavior more than anyone fully appreciated, which is exactly why the job Coxon held is not a footnote at all.

Demystifying the Black Box

None of this is exotic or hidden knowledge. Author Brian Christian walks through this same training process, and where it quietly goes wrong, in his book "The Alignment Problem," which remains one of the clearest plain-language accounts of how machine learning actually works. This is worth saying because a lot of the fear driving the current debate in Washington treats AI training like a black box. It is not a black box. It is bits, weights, and a staggering amount of repetition, and now you know how it works.

That still leaves the actual scary phrase unexplained. Pretraining explains how a model learns to write a sentence. It does not explain "recursive self-improvement," the idea of an AI redesigning itself to get smarter without a person steering the process, which is the term doing the heaviest lifting in every speech coming out of Sanders' upcoming briefing and Lieu's kill switch bill. That is Part 3.

Sources:

Anthropic — Teaching Claude Why

Brian Christian, "The Alignment Problem: Machine Learning and Human Values" (W. W. Norton, 2020)

WECU News — "Congress Is Reacting to AI Doom. Do They Know What It Means?"

WECU News — The Money Behind the Viral Anthropic AI Doom Post