Train a transformer on a Tuesday

Take the laptop you bought five years ago. Open a tab. Train a transformer on it — not fine-tune: train, from randomly initialized weights to text that mostly spells, on the machine's own GPU, with the Wi-Fi off for most of the run. A character-level model, six layers, about 4.8 million parameters. Let's walk the whole evening, mistakes left in, because the mistakes are where a platform shows its character.

If you have ever set up a training environment — the driver roulette, the dependency pinning, the “works on my machine” — the point of this walk is that none of it happens. You open a tab; the engine talks to the GPU through WebGPU, which the browser already ships. That is the setup.

Sketching the stack#

A model node on the canvas opens into Model View when you double-click it: a full-screen editor with three synchronized projections of the same model — a Graph you draw, a Form you type into, and a Code view in SML, a declarative, Pythonic rendering of the same structure. Edit any one and the other two follow. Draw in the Graph — token embedding, learned positions, six pre-norm blocks, a tied output head — and nudge the widths in the Form. The Code view rewrites itself underneath, which turns out to be oddly good company at nine in the evening.

# Code view of the stack (abridged)
model CharTransformer(context=256, vocab=96):
  d = 256; heads = 8; layers = 6

  x = token_embed(vocab, d)(tokens) + learned_pos(context, d)
  repeat layers:
    x = x + attention(norm(x), heads=heads, causal=true, dropout=0.1)
    x = x + mlp(norm(x), hidden=4 * d, activation="gelu", dropout=0.1)
  logits = head(norm(x), tied=token_embed)

That context=256 is a survivor's number — the first draft says 512. We'll get there.

The wire turns red before you let go#

The first real mistake is a classic, so let's make it. Type the MLP's hidden width, 1024, into its output projection too — the block now wants to hand [B, T, 1024] back to a residual add that expects [B, T, 256]. In a script, that is a stack trace on the first run, twenty minutes later. Here the wire turns red before you have let go of the mouse. Validation in Model View is live; the diagnostic names both ports and both shapes, and offers the repair as a one-click quick-fix: set the projection back to d_model. You click it — slightly offended, mostly grateful.

That is the quiet thesis of the whole surface: a model here is a structured document the editor understands, not a text file that fails at runtime. Shape errors surface at authoring time because shapes are part of the contract — not a surprise inside the forward pass.

Napkin arithmetic: 4.8 million parameters#

Transformer parameter counts fit on a napkin. Each block carries attention (4d² for Q, K, V and the output projection) plus the MLP (8d², for d→4d and back): twelve d² per layer. With d = 256 that is 786,432 parameters a layer; six layers make 4,718,592. The token embedding adds 96 × 256 ≈ 25k — tied with the output head, so it is counted once — learned positions another 256 × 256 = 65,536, norms and biases a few thousand more. Call it 4.8 million.

At f32 that is about 19 MB of weights. The weights were never going to be the problem. The activations are.

The plan says no#

Ask for context 512 and batch 64, because ambition is free until you compile. Before anything runs, the engine plans the run and prices its memory — parameters, gradients, optimizer state, activations, working memory — and refuses outright if it does not fit. The plan comes back red. The breakdown is blunt: activations dominate, and inside them the attention buffers, which grow with the square of the context. At batch 64 and context 512, the attention matrices alone are 64 × 8 × 512 × 512 floats per layer — half a gigabyte, times six layers, before anything is even saved for the backward pass. Peak: north of 6 GB. The budget on a machine like this: about 4.

The refusal comes with its own remedies: shorten the context, shrink the batch, change precision. Take all three. Context 256 — the square is what kills you, so halving it quarters exactly those buffers. Batch 32. And mixed precision: activations and matmuls in f16, master weights kept in f32, a stateful GradScaler doing loss scaling because f16 gradients underflow otherwise. Replan: green, peak just under 2 GB. Had it stayed red, activation checkpointing was still in reserve — recompute instead of store. The part worth keeping: memory is never a mystery. You know where every large buffer will come from before a single byte moves.

Watching the loss fall#

The training configuration lives in Model View too, next to the architecture instead of scattered across scripts: AdamW at 3e-4, WarmupCosineLR, CrossEntropy, gradient clipping at 1.0, dropout 0.1, seed 1337, determinism tier prod — reproducible within a declared tolerance on this device, which is the honest default; debug would pin it bitwise, at a throughput cost.

A character model's first guess is uniform over the vocabulary: with 96 characters, cross-entropy starts at ln 96 ≈ 4.56 nats. From there the curve falls fast through the warmup, and the sampled text walks through the stages anyone who has trained one of these will recognize: noise with the right character frequencies, then word-shaped things, then whole clauses that spell almost everything and mean almost nothing. Somewhere in the middle you notice the Wi-Fi has been off since dinner. Nothing cared. The model, the data, the compiler, the run — all of it local-first, on this side of the network.

Now the honest part. 12,000 steps at batch 32 and context 256 is roughly 98 million tokens — that number is arithmetic. What that costs in minutes on a five-year-old GPU is an estimate, and it deserves to be named as one: an evening, not a coffee break. We will publish measured timings when there is a release to measure, and not before. What can be stated precisely today is the shape of the cost: training runs on the eager autograd tape, so the gradient is correct but not fused (kernel fusion currently serves the inference forward path). Nobody is pretending a tab on an old laptop outruns a rented datacenter GPU. The claim is a different one: there is no environment to build, no upload, no meter running — and the machine you already own does the whole thing while you make tea. The run panel even offers a wake lock so the laptop does not doze off mid-epoch.

Checkpoint, commit, cite#

The checkpoint at the end is not a weights file; it is a bundle — weights, optimizer state, RNG state, epoch, step, metrics — content-addressed as one unit. Resumability falls out of that structure: if the tab dies at step 9,000, the run reconciles to “interrupted” with a one-action resume from the last checkpoint, and because the RNG state is inside, the shuffle and the dropout pick up exactly where they left off.

Then you commit the run. A committed version is content-addressed — the hash covers the workflow, the model, the dataset version, the training configuration, the checkpoint, down to the engine version — and the snapshot URL re-resolves those exact bytes, for anyone, login-free. The precise reproducibility claim is worth stating carefully, because it is the one this platform refuses to inflate: the cited bytes are exact, forever; a re-execution on different hardware agrees within the declared tolerances of the prod tier, not bitwise — floats on heterogeneous GPUs do not do bit-for-bit, and the platform says so instead of rounding up.

That is the evening. One shape mistake, one red plan, one green one, a falling loss curve — and at the end, a URL that is the experiment rather than a picture of it. The same walk, step by step and with fewer of these mistakes, is in the guide's training in the browser. It does not require a Tuesday.