rustlab_llm — Lesson Site

A self-contained tutorial series for building a GPT-style language model from first principles, using rustlab as the scripting and visualisation environment. Each lesson is a single executable Markdown notebook — derivations, code, and plots in one file.

What you see here is the rendered output of the source notebooks: prose, executed rustlab blocks, captured print() output, and inline SVG plots. GitHub displays it directly.

The source notebooks live at ../notebooks/<slug>.md. This book/ tree is regenerated by make notebooks from the repo root; only book/README.md (this file) is hand-written.

Reading order

Each lesson depends on the ones before it, and each closes with an Engineering Lenses section (Signals · Systems · Information) whose claims are labelled Exact, Model, or Analogy and backed by executed code. See ../PLAN.md for phase status and ../docs/proposal-2026-09-27-ee-controls-it-revision.md for the design of the revision.

Lesson 00 — Orientation

#	Lesson	Focus
00	The LLM as a System	the signal-flow map, the three lenses, notation, your toolkit mapped to lessons

Phase 1 — Foundations

#	Lesson	Focus
01	Tokens & Text Encoding	character vocabulary, integer indices, one-hot encoding
02	Probability & Softmax	discrete distributions, softmax, temperature scaling
03	Cross-Entropy Loss	comparing predicted vs. true distributions; MLE connection
04	Embeddings & Similarity	dense vectors, cosine similarity, geometry of meaning

Phase 2 — First Working Model

#	Lesson	Focus
05	The Bigram Language Model	count-based next-token prediction; train/sample loop
06	Linear Layers & Gradient Descent	\(y = Wx + b\); loss landscape; gradient step

Phase 3 — Attention

#	Lesson	Focus
07	Context & Naive Averaging	bag-of-tokens averaging; the causal mixing matrix
08	Scaled Dot-Product Attention	\(Q, K, V\); attention scores; causal masking
09	Multi-Head Attention	parallel heads, concatenation, output projection

Phase 4 — Transformer Components

#	Lesson	Focus
10	Positional Encoding	sinusoidal encoding; why attention is order-blind
11	Feed-Forward Block	position-wise MLP; GELU
12	LayerNorm & Residuals	normalisation; the residual stream

Phase 5 — Full GPT Architecture

#	Lesson	Focus
13	The Transformer Block	Pre-LN block: MHA → residual → FFN → residual
14	Full GPT Architecture	embeddings + N blocks + LM head; parameter count

Phase 6 — Training

#	Lesson	Focus
15	Backpropagation	chain rule through attention and MLP; gradient flow
16	AdamW Optimizer	momentum, adaptive steps, decoupled weight decay
17	Learning-Rate Scheduling	warmup + cosine decay
18	The Training Loop	forward → loss → backward → step; diagnostics

Phase 7 — Tokenisation & Evaluation

#	Lesson	Focus
19	Byte Pair Encoding	iterative merges; subword vocabulary
20	Perplexity & Evaluation	PPL = exp(loss); train/val; overfitting

Phase 8 — Generation, Backprop, Capstone

#	Lesson	Focus
21	Sampling and Generation	autoregressive loop; greedy / temperature / top-K / top-P; KV cache
22	Full Backprop Through the Block	the complete analytical backward pass; gradient check; the shared library
23	Putting It All Together	char → BPE → train the full transformer → sample at checkpoints

Phases 9–10 — Extensions

#	Lesson	Focus
24	Modern Architectural Variants	RoPE, RMSNorm, SwiGLU, GQA
25	Fine-Tuning — SFT and DPO	loss-masked SFT and catastrophic forgetting; DPO with a frozen reference
26	Quantization and Fixed-Point Inference	b-bit weights, 6 dB per bit, perplexity vs bits, KV-cache memory and the bandwidth bound

Local interactive view

For Plotly hover/zoom/pan and KaTeX-rendered math, run make html from the repo root and open index.html (right next to this file) in a browser. The HTML build includes an auto-generated entry page with prev/next navigation between notebooks. The index.html and per-notebook *.html files are gitignored — regenerate them locally any time.

How this is organized

The pattern (sources flat in notebooks/, rendered output committed to a top-level book/, standalone .rlab scripts in lessons/<slug>/, shared code in lib/) is documented at ../rustlab/docs/lesson-site-pattern.md and shared across the rustlab course family (rustlab_em uses the same layout).