Deep learning
Language model training
pretrain a small model, adapt an open model to a domain, preference tuning
Train a language model from scratch, or adapt a pretrained one to a domain, format, or behaviour.
Probability & Bayes
Information theory
Linear algebra
Calculus & gradients
Optimisation
Foundations
Next-token prediction is cross-entropy over a vocabulary; attention is matrix multiplication; scale is an optimisation problem.
Frame the problem
The LLM ladderbackground
Frame
Pick the rung: continued pretraining for domain knowledge, supervised fine-tuning for format and task, preference tuning for behaviour. Decide the eval before the data.
Sourcing & signal
Labeling & data collection
Cleaning the data
Data & labels
Corpus quality beats corpus size. Deduplicate, filter, and decontaminate against your evals. For fine-tuning, a few thousand excellent examples beat millions of mediocre ones.
Tokenization
Embedding layersbackground
Represent
Tokenizer choice is permanent. Reuse the base model's for adaptation.
Split the data
Temporal splits & backtesting
Split
Held-out documents, and eval prompts that share nothing with training data. Temporal for anything with a knowledge cutoff.
Transformer internals
Attention & transformersbackground
MLP & backpropbackground
Model
The architecture is settled; the choices are size, context length, and which base model.
Training craft
Pretraining
Next-token at nano scale to understand it; at real scale only with a budget and a reason.
Next-token pretraining
Masked & contrastive pretraining
Supervised fine-tuning
LoRA first. Full fine-tuning when you have the data and the hardware.
Fine-tuning a pretrained model
Fine-tuning & LoRA
Preference tuning
DPO before RLHF. Both need a preference dataset and an eval that catches regressions.
RLHF & DPO
Distillation
A small student on the big model's outputs, for latency and cost.
Distillation & domain adaptation
Train
Evaluating generative output
Error analysis & significance
Evaluate offline
Perplexity for pretraining, task evals for fine-tuning, win rate for preference tuning. Always run the general benchmarks too; adaptation forgets.
A/B testing & interleaving
Evaluate online
Whatever product consumes the model measures it.
Inference optimisation
Serving & release
Monitor & retrain
Ship & monitor
Quantise, batch, cache the KV. Version prompts with the model.
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.