Lesson 00: The LLM as a System

A large language model is a discrete-time system that reads a stream of symbols, keeps a state, and emits a probability distribution over the next symbol. Built that way — as a signal-processing chain, as a dynamical system with feedback, and as a device that moves bits — every component of a GPT is something an electrical engineer, a control engineer, or an information theorist already knows how to reason about. This course derives the whole model from those three toolkits and implements every step in rustlab, so nothing is a black box.

This lesson is the map: what the system looks like, which three questions every lesson asks of each component, the notation the whole course uses, and where each tool you already own will be used.

Learning Objectives

• Read the full GPT pipeline as one signal-flow graph and name which lesson builds each box.
• State the three lenses — Signals, Systems, Information — as the fixed questions asked of every component, and the three honesty labels (Exact, Model, Analogy) attached to every answer.
• Adopt the course notation: tokens are rows (time), features are columns, temperature is \(\tau\), losses are nats in code and bits in prose.
• Locate, for each tool in your existing kit (FIR/IIR filters, eigenvalue stability, adjoint equations, entropy and coding), the lesson where it does the work.

Background

Linear algebra (matrix products, transposes, eigen- and singular values), signals and systems (LTI and time-varying filters, impulse and frequency response, sampling, correlation), state-space and discrete-time stability, and Shannon's entropy, cross-entropy, KL divergence and source coding. No deep-learning background is assumed, and no deep-learning vocabulary is used before it is derived.

The System in One Picture

Theory

A decoder-only transformer maps a token sequence \(x_{1..T}\) to a distribution over \(x_{T+1}\). Read left to right it is a chain of operators on a \(T \times d\) signal; read top to bottom it is a closed loop — the training loop feeds the loss back into the parameters, and the generation loop feeds the sampled token back into the input.

The residual stream — the \(T \times d\) matrix threading through the blocks — is the system's state along depth; the key/value cache (21-sampling-and-generation) is its state along time. Attention is the only operator that mixes information along the time axis (rows); everything else acts on each row independently along the feature axis (columns). Holding that picture is most of what it takes to read any transformer variant.

Three Lenses

Theory

Every lesson ends with an ## Engineering Lenses section that asks three fixed questions of the component just built:

Every answer carries one of three labels, written in bold at the start of the sentence:

• Exact. A formal equivalence: the two objects are the same mathematics under a change of names. (Backprop is the discrete adjoint recursion.)
• Model. A faithful simplification whose assumptions are stated. (LayerNorm as automatic gain control — instantaneous, not a loop.)
• Analogy. Intuition only, not an identity; useful for orientation, dangerous if pushed. (Warmup as a soft start.)

The labels are a promise: an Exact statement can be checked by algebra, and every lesson checks at least one numerically.

Notation

Theory

The whole course uses one convention so that mathematics and code read the same way.

Example — Bits, nats, and perplexity

The one conversion that recurs in every lesson: rustlab's log is natural, so losses come out in nats; dividing by \(\ln 2\) gives bits, and exponentiating either gives the same perplexity. The helpers in lib/info.rlab are used from Lesson 02 onward, after it derives entropy inline.

p = [0.5, 0.25, 0.125, 0.125];       % a four-symbol source
H_bits = entropy_bits(p);
H_nats = entropy_nats(p);
print("H =", H_bits, "bits =", H_nats, "nats");
print("perplexity 2^H_bits =", 2 ^ H_bits, "   e^H_nats =", exp(H_nats));
print("uniform bound log2(4) =", log2(4), "bits");

H = 1.75 bits = 1.2130075659799042 nats
perplexity 2^H_bits = 3.363585661014858    e^H_nats = 3.363585661014858
uniform bound log2(4) = 2 bits

The source has \(1.75\) bits of entropy — a perplexity of \(3.36\), i.e. it behaves like a fair choice among \(3.36\) symbols even though the alphabet has 4. Every model in this course is scored the same way: cross-entropy in nats, reported in bits, read as an effective alphabet size.

Your Toolkit, Mapped

Theory

Where each tool you already own does the work:

How to Read a Lesson

Theory

Each lesson is one executable notebook: prose and equations, then rustlab blocks whose printed output and figures are captured into the rendered book. Every concept H2 splits into ### Theory and ### Example — …; the ## Engineering Lenses H2 follows the last concept; then Key Takeaways, the table of parallel standalone scripts (lessons/<slug>/*.rlab, runnable from a shell with rustlab run or make lesson-NN), an Expected Numerical Outputs table you can check by hand, exercises, and a forward link. Shared code that more than one lesson needs — the transformer forward/backward pass, the optimiser, sampling, information helpers — lives in lib/ and is pulled in with run.

Read in order. Each lesson assumes only the ones before it.

Key Takeaways

• A GPT is a chain of operators on a \(T \times d\) signal inside two feedback loops (training and generation).
• Three questions are asked of every component — Signals, Systems, Information — and every answer is labelled Exact, Model, or Analogy.
• One notation: tokens are rows, features are columns, temperature is \(\tau\), losses are nats in code and bits in prose.
• The tools of EE, control, and information theory are not analogies bolted on afterwards; in most lessons they are the mathematics itself.

Standalone Scripts

None — this lesson is the map. The first runnable scripts are in 01-tokens-and-encoding.

Expected Numerical Outputs Summary

Exercises

1. Trace a token. Follow one token id through the diagram and write down the shape of the signal after every box for \(T = 8\), \(d = 64\), \(\lvert\mathcal{V}\rvert = 50\).
2. Name the loops. Which arrows in the diagram carry gradients, which carry sampled tokens, and which carry activations? What is the "plant", the "sensor", and the "controller" in the training loop?
3. Label practice. For each of these statements decide Exact, Model, or Analogy before you reach the lesson that settles it: "softmax is a Boltzmann distribution"; "LayerNorm is automatic gain control"; "attention is a matched filter"; "backprop is the adjoint method".

What's next

01-tokens-and-encoding starts at the alphabet: text becomes integers, integers become one-hot basis vectors, and the first information-theoretic floor — the entropy of the unigram source — is computed.