Building Large Language Models From Scratch — The Complete Engineer's Handbook
Building LLMs
from Scratch
A definitive, lecture-by-lecture engineering walkthrough of how a Large Language Model is conceived, tokenized, pretrained, finetuned, evaluated, and deployed — distilled from a 31-lecture master class and rebuilt as a single self-contained reference.
Abstract
This document is a deeply annotated expansion of a 43-lecture video series that walks an engineer from zero to a fully functional, fine-tuned Large Language Model. The original lectures emphasize that modern LLMs are not magic: every component — tokenization, embedding lookup, self-attention, layer normalization, the GELU non-linearity, residual connections, causal masking, cross-entropy loss, temperature scaling, top-k sampling, supervised classification finetuning, instruction finetuning, and weight loading from OpenAI's released checkpoints — is a small, inspectable, learnable idea. By the end of the series, a student has written a 124-million-parameter GPT-2-style model in pure PyTorch, trained it on a public corpus, measured its perplexity, generated text with deterministic and stochastic decoding, loaded OpenAI's official weights, and adapted the model to both classification and instruction-following tasks before exporting it to Ollama for local inference.
The pages that follow reproduce every lecture title, summarize the conceptual content, surface the canonical external references (original papers, library docs, foundational blog posts), and provide the mathematical and engineering detail required for a newcomer to internalize each topic. The reading experience is engineered to feel like a long-form magazine feature: serif type, editorial red accents, double horizontal rules, and a strict grid. Every claim that benefits from a deeper source is hyperlinked to the most authoritative public reference available.
The Mental Models You Need Before Writing a Single Line of Code
Before any tensor is allocated, a working LLM engineer must hold a clear picture of what a "language model" actually is, how it differs from older NLP systems, what a transformer is, and the three canonical stages of building one. Part I lays exactly that groundwork.
Series Introduction — Why Build an LLM from Scratch?
The opening lecture is deliberately non-technical. It explains the pedagogical philosophy of the series: every modern LLM — from the original Transformer (Vaswani et al., 2017) to GPT-3 (Brown et al., 2020) to today's open-weights models like Llama 2 and Mistral — is a composition of a small set of reusable building blocks. The series aims to teach those blocks by hand, in PyTorch, with no high-level abstractions hiding the math.
The lecturer frames three "reasons" a working engineer should build one from scratch: intuition (you can no longer be fooled by marketing claims about "emergence"), debuggability (when a fine-tune fails, you can finally see why), and fluency (you can read and write research papers in the field). He also previews the arc: pretraining, classification fine-tuning, instruction fine-tuning, and deployment via Ollama.
External reading: Attention Is All You Need — Wikipedia summary · The Illustrated Transformer (Jay Alammar) · Goodfellow, Bengio & Courville — Deep Learning (free).
LLM Basics — What Problem Are We Even Solving?
Before defining an LLM, the lecture defines a language model: a probability distribution over token sequences, P(t₁, t₂, …, tₙ). The factorization P(t₁,…,tₙ) = ∏ P(tᵢ | t₁,…,tᵢ₋₁) converts the joint distribution into a sequential next-token prediction problem, which is exactly the loss surface modern LLMs are trained against. A "Large" Language Model simply means the distribution is parameterized by a transformer with a very high parameter count trained on a very large corpus.
The lecture distinguishes LLMs from earlier NLP systems in three ways: scale (parameters and tokens), generality (a single model handles many tasks via prompting, instead of one model per task), and emergence (qualitatively new behaviors appear past a threshold). The lecturer also introduces the canonical dataset of the series: Project Gutenberg, the TinyShakespeare corpus, and the SMS Spam dataset for classification work.
External reading: Brown et al. 2020 — Language Models are Few-Shot Learners · Radford et al. 2018 — Improving Language Understanding by Generative Pre-Training (the original GPT) · Wikipedia: Large language model.
Pretraining LLMs vs. Finetuning LLMs
The crucial conceptual move of the modern LLM stack is the split between pretraining and finetuning. In pretraining, a transformer learns the general statistical structure of language by predicting the next token on a huge unlabeled corpus — a process measured in weeks of GPU time and billions of tokens. The output is a base model: a fluent text completer, but not a useful assistant.
Finetuning is what turns a base model into a product. There are two main flavors covered later in the series: classification finetuning (Lecture 35 — append a small classification head and train on labeled data, e.g. spam vs. ham) and instruction finetuning (Lecture 36 — train on prompt-response pairs in a chat format such as Alpaca). A third flavor, RLHF, is mentioned as the production-grade step used in InstructGPT (Ouyang et al., 2022).
External reading: Sebastian Ruder — Transfer Learning in NLP · Hugging Face — Trainer documentation · Howard & Ruder — Universal Language Model Fine-tuning (ULMFiT).
What Are Transformers?
The transformer is the single architecture that powers essentially every modern LLM. The lecture walks through its two halves: an encoder (bidirectional, used for classification and representation learning, e.g. BERT (Devlin et al., 2018)) and a decoder (autoregressive, used for generation, e.g. the GPT family). A "GPT-style" LLM is a stack of decoder-only transformer blocks.
Inside every block live the four ideas the rest of the series will dissect: self-attention (let every token look at every previous token), layer normalization (stabilize activations), feed-forward GELU network (per-token non-linearity), and residual / shortcut connections (let gradients flow). The lecture draws the canonical diagram and introduces the concept of the "context window" — the maximum number of tokens the model can attend to at once.
External reading: Vaswani et al. 2017 — Attention Is All You Need (original paper) · The Illustrated Transformer · Harvard NLP — Annotated Transformer · TensorFlow's annotated walkthrough.
How Does GPT-3 Really Work?
Lecture 5 zooms in on GPT-3 as a concrete instance of the abstract transformer. The 175-billion-parameter model was trained on roughly 300 billion tokens of Common Crawl, web text, books, and Wikipedia using the same next-token cross-entropy loss discussed in Lecture 1, scaled by three orders of magnitude. The lecture highlights three architectural details that GPT-3 inherited from GPT-2: pre-norm (layer-norm before attention, not after), causal masking (triangular attention mask), and context length 2048 tokens.
The lecturer also discusses the famous "in-context learning" phenomenon: at sufficient scale, a frozen language model can perform new tasks from a handful of examples placed inside the prompt, without any weight updates. This is what makes modern prompting possible and is the foundation of techniques like few-shot prompting and chain-of-thought reasoning (Wei et al., 2022).
External reading: Brown et al. 2020 — Language Models are Few-Shot Learners · GPT-3 — Wikipedia · Hugging Face — GPT-2 architecture summary.
Stages of Building an LLM from Scratch
This lecture is the road map for the rest of the series. The lecturer draws a five-stage pipeline on the whiteboard:
- Data preparation — collect a corpus, tokenize with BPE, build input–target pairs, and add positional embeddings (Lectures 7–12).
- Attention — implement the self-attention mechanism in five progressive variants: simplified, key/query/value, causal, multi-head, and the full block (Lectures 13–18).
- Architecture — assemble layer norm, GELU, residual connections, the transformer block, and the final 124M-parameter GPT-2 (Lectures 19–24).
- Pretraining — write the training loop, measure cross-entropy loss, evaluate on held-out data, generate text (Lectures 25–28).
- Finetuning & deployment — classification finetune, instruction finetune, load OpenAI's weights, and ship to Ollama (Lectures 29–31).
The lecture also previews the data sources used throughout: Project Gutenberg for pretraining, SMS Spam for classification, and the Alpaca instruction dataset for chat-style finetuning. The pipeline laid out here is exactly the same one used by production teams at OpenAI, Anthropic, and Meta — only the scale differs.
External reading: Andrej Karpathy's nanoGPT (the canonical minimal reference implementation) · Hugging Face Transformers — full library documentation · PyTorch — Sequence-to-Sequence Modeling with nn.Transformer.
Tokenization, Embeddings, and the Input Pipeline
A language model is, mechanically, a sequence-to-sequence function over integers. This part turns raw text into the integer tensor the network will actually consume — a process more subtle and more consequential than most beginners expect.
LLM Tokenizer from Scratch in Python
The lecture opens with a deceptively simple question: how do you turn the string "The quick brown fox" into a list of integers? The naive answer — one integer per word — fails because the model would have no way to represent out-of-vocabulary words. The character-level answer solves OOV but explodes the sequence length and makes learning long-range dependencies nearly impossible. The standard answer is sub-word tokenization: split text into pieces that are frequent enough to be in a fixed vocabulary (~50,000 entries) and short enough to keep sequence length manageable.
The lecturer builds a toy tokenizer from scratch in pure Python: it iterates over a corpus, finds the most common adjacent pair of tokens, and merges them into a single new token — repeating the process a fixed number of times. This is the Byte Pair Encoding (BPE) algorithm. The lecture ends by contrast with the production-grade tiktoken library used by OpenAI and with Hugging Face's tokenizers library.
External reading: Sennrich et al. 2016 — Neural Machine Translation of Rare Words with Subword Units · Hugging Face NLP Course — Chapter 6: Tokenizers · tiktoken — OpenAI's fast BPE implementation.
The GPT Tokenizer — Byte Pair Encoding in Detail
Lecture 8 implements a production-quality tiktoken-style BPE in Python. Key topics include: pre-tokenization with a regex that splits on whitespace and punctuation, byte-level encoding so that any UTF-8 string can be tokenized without an unknown token, and special tokens like <|endoftext|> used to mark document boundaries.
The lecture also exposes a famous practical issue: the GPT tokenizer (and most BPE tokenizers) allocates very different numbers of tokens to different languages and to different scripts — a sentence in English might be 10 tokens while the same sentence in Burmese is 60. This has measurable cost, fairness, and capability implications, as analyzed in Peters & Lecocq (2020) and Ahia et al. (2023).
External reading: tiktoken's _educational.py — readable reference implementation · Hugging Face — Training a new tokenizer · HF — Fast tokenizers (Rust-backed).
Creating Input–Target Pairs Using Python DataLoader
Once text is tokenized, the next step is to chop the resulting 1 × N stream of token ids into fixed-length input–target pairs. The convention is to take a window of size context_length tokens as the input and the same window shifted one position to the left as the target. The model then learns, for every position, to predict the next token from all previous tokens.
The lecture introduces PyTorch's Dataset and DataLoader abstractions. A custom GPTDataset class wraps the token stream, returns random windows, and the DataLoader batches them and shuffles them. A DropLast flag is set to drop the final incomplete batch (it would otherwise produce tensors of the wrong shape).
# A minimal PyTorch dataset for next-token prediction class GPTDataset(Dataset): def __init__(self, txt, tokenizer, max_length, stride): self.input_ids, self.target_ids = [], [] token_ids = tokenizer.encode(txt) for i in range(0, len(token_ids) - max_length, stride): self.input_ids.append(torch.tensor(token_ids[i : i+max_length])) self.target_ids.append(torch.tensor(token_ids[i+1 : i+max_length+1])) def __len__(self): return len(self.input_ids) def __getitem__(self, idx): return self.input_ids[idx], self.target_ids[idx]
External reading: PyTorch — Datasets & DataLoaders tutorial · torch.utils.data documentation · Hugging Face Datasets library.
What Are Token Embeddings?
An integer token id is just a label. Neural networks need dense vectors. A token embedding is a learnable lookup table of shape [vocab_size × embedding_dim]: row i is the dense vector that represents token i. The model starts with random vectors and learns the values during training so that semantically similar tokens end up close together in cosine distance. Embedding dimension in GPT-2 is 768; in GPT-3 it is 12,288.
The lecture builds the embedding layer with torch.nn.Embedding, indexes into it with token ids, and visualizes the resulting vectors with TensorBoard's Embedding Projector. The famous result: after training, the embedding for "king" minus the embedding for "man" plus "woman" is approximately the embedding for "queen" (Mikolov et al., 2013).
External reading: Mikolov et al. 2013 — Efficient Estimation of Word Representations in Vector Space (word2vec) · torch.nn.Embedding documentation · TensorFlow Embedding Projector.
The Importance of Positional Embeddings
A self-attention layer is permutation-equivariant: it processes the set of input vectors, not the sequence. Shuffle the tokens and the output is shuffled in exactly the same way. This means that without extra information, the model cannot tell the difference between "the dog bit the man" and "the man bit the dog". The fix is to add a second learnable lookup table — a positional embedding — of shape [context_length × embedding_dim] to the token embedding before it enters the first transformer block.
The lecture compares the two canonical schemes: absolute positional embeddings (used by GPT-2; one vector per position 0…1023) and relative / rotary positional embeddings (used by Llama, Mistral, and most modern models; see RoFormer / RoPE, Su et al. 2021). The lecturer also explains ALiBi (Press et al., 2022) as a parameter-free alternative.
External reading: Su et al. 2021 — RoFormer: Enhanced Transformer with Rotary Position Embedding · Press et al. 2022 — Train Short, Test Long: Attention with Linear Biases (ALiBi) · Amirhossein Kazemnejad — Transformer Architecture: The Positional Encoding.
The Entire Data Preprocessing Pipeline of LLMs
Lecture 12 stitches everything from the previous five lectures into a single, end-to-end pipeline that can be run on any text file: load corpus → tokenize with BPE → add special tokens → chunk into overlapping windows → look up token embeddings → add positional embeddings → yield a tensor of shape [batch_size × context_length × embedding_dim]. The lecturer also discusses dataset hygiene: deduplication (per Lee et al., 2022), PII scrubbing, and contamination analysis against benchmark test sets.
The chapter also introduces context length as a key hyperparameter and explains its cost: attention is quadratic in sequence length (O(n²)), which is why long-context research is one of the most active areas in the field (see Longformer, Linformer, FlashAttention).
External reading: Lee et al. 2022 — Deduplicating Training Data Makes Language Models Better · Dao et al. 2022 — FlashAttention: Fast and Memory-Efficient Exact Attention · Hugging Face — Preprocessing data.
Attention — The Single Most Important Idea in Modern AI
Attention is what allows a model to dynamically route information between any two positions in a sequence. This part builds the mechanism in five progressive variants, from a single matrix multiply to the full multi-head causal self-attention used in GPT-2.
Introduction to the Attention Mechanism in LLMs
Attention is a soft dictionary lookup. Given a query vector q, a set of key vectors K, and a set of value vectors V, the output is a weighted average of the values, where the weights come from a softmax over q · k dot products. In a language model, every token issues a query, every previous token offers a key, and the value is the information the previous token is willing to pass forward.
The lecture traces the conceptual history from Bahdanau attention (2014) for machine translation, through self-attention in Transformers (2017), to its use as the only sequence-mixing operator in modern LLMs — replacing recurrence (RNN/LSTM) entirely.
External reading: Bahdanau, Cho & Bengio 2014 — Neural Machine Translation by Jointly Learning to Align and Translate · Vaswani et al. 2017 — Attention Is All You Need · Lilian Weng — Attention? Attention!.
Simplified Attention Mechanism — Coded from Scratch in Python (No Trainable Weights)
To make the idea unforgettable, the lecturer implements the simplest possible attention in NumPy with no learned parameters. Given an input matrix of token vectors, the attention output is softmax(X · Xᵀ) · X — each output token is a weighted sum of every input token, with weights determined by pairwise dot product. This is sometimes called "averaging attention with similarity weights" and it instantly shows why attention is so powerful: it lets every position see every other position with a learned, data-dependent weight.
The lecture also demonstrates the famous "attention is all you need" toy example: a single attention head can learn to copy, average, or selectively route information between arbitrary positions, something a recurrent network requires many layers to achieve.
# Simplified self-attention in 4 lines of NumPy scores = X @ X.T # [n × n] dot products weights = np.exp(scores - scores.max(axis=-1, keepdims=True)) weights /= weights.sum(axis=-1, keepdims=True) # row-wise softmax out = weights @ X # [n × d]
External reading: Sebastian Raschka — Understanding and Coding Self-Attention from Scratch · e2eML — Transformers from scratch · Borealis AI — Understanding Self-Attention.
Coding Self-Attention with Key, Query, and Value Matrices
The simplified attention of Lecture 14 used the same vector for the query, the key, and the value. In a real transformer, three separate linear projections of the input are learned: Q = X · W_q, K = X · W_k, V = X · W_v. This decoupling is what gives attention its expressive power: queries learn "what am I looking for?", keys learn "what do I contain?", and values learn "what do I pass forward if I am selected?"
The lecturer implements the trainable variant in PyTorch as a nn.Module subclass and shows that the output dimension equals the input dimension — making the module a drop-in replacement inside a residual stream. He also introduces d_k = embedding_dim / num_heads and the √d_k scaling factor in the softmax argument.
External reading: torch.nn.Linear documentation · Vaswani et al. 2017 — Section 3.2.1 (Multi-Head Attention) · D2L.ai — Self-Attention and Positional Encoding.
Causal Self-Attention — The Mask That Makes Generation Work
GPT-style LLMs are causal (also called masked) language models: token i is only allowed to attend to tokens at positions ≤ i. This is enforced with an upper-triangular mask of -∞ values added to the attention scores before the softmax, so that future positions receive zero weight. Without this mask the model would "cheat" during training by looking at the token it is supposed to predict.
The lecture derives the dropout applied to the attention weights, which is one of the key regularizers in the transformer. It also discusses attention with a context larger than the training context: at inference time, you can technically attend to all past tokens, but for many models performance degrades past the training length — a phenomenon documented in Press et al. 2023.
# Causal mask in PyTorch mask = torch.triu(torch.ones(context_length, context_length), diagonal=1).bool() scores = scores.masked_fill(mask, -torch.inf) weights = torch.softmax(scores, dim=-1)
External reading: torch.nn.functional.softmax documentation · Vaswani et al. 2017 — Section 3.2.3 (Masked Attention) · Hugging Face — Decoder-only models: the workhorse of generative AI.
Multi-Head Attention Part 1 — Basics and Python Code
One attention head can only learn one kind of relationship at a time (e.g. "the previous noun"). To capture many relationship types in parallel, the embedding dimension is split into num_heads chunks, attention is computed independently in each chunk, and the outputs are concatenated and linearly projected back. In GPT-2: embedding_dim = 768, num_heads = 12, head_dim = 64.
The lecture writes the multi-head class as a wrapper around the single-head class from Lecture 15. It also introduces the concept of grouped query attention (GQA) used in Llama 2, where multiple query heads share the same key/value head, saving memory and compute with minimal quality loss.
External reading: Ainslie et al. 2023 — GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · The Illustrated Transformer — Multi-Head section · Hugging Face — GPT2Model reference.
Multi-Head Attention Part 2 — The Entire Mathematics Explained
The deeper, more mathematical companion to Lecture 17. The lecturer walks through every tensor shape in the forward pass with the b (batch), nh (number of heads), tc (token count), and hs (head size) notation that the original OpenAI GPT-2 code uses. He explains the .view and .transpose calls that turn a flat tensor of shape [b × tc × embedding_dim] into a per-head tensor of shape [b × nh × tc × hs] and back.
The lecture also derives the parameter count of multi-head attention. For GPT-2: 3 × 768² (Q, K, V) + 768² (output projection) = 2,362,368 parameters per layer, times 12 layers, = 28.3M parameters — about 23% of the 124M total. The remainder is in the feed-forward network and the embeddings.
External reading: OpenAI's reference GPT-2 model.py · Hugging Face GPT-2 source code · Ba, Kiros & Hinton 2016 — Layer Normalization (the original paper).
LayerNorm, GELU, Residuals, and the Full GPT-2 Block
Attention is the flashy half of a transformer. The other half is a stack of small, almost boring components — layer normalization, a non-linearity, and skip connections — that together make training stable, deep, and fast. Part IV dissects each piece and then snaps them together into a 124-million-parameter model.
Birds-Eye View of the LLM Architecture
The lecturer draws the entire GPT-2 architecture on the whiteboard in one sitting: a stack of N identical transformer blocks, each containing a multi-head causal self-attention sublayer and a feed-forward sublayer, with layer normalization and residual connections wrapping each sublayer. The input flows through a token embedding + positional embedding, the blocks, a final layer norm, and a linear output projection back to vocabulary size.
The lecture also introduces the central "data flow" concept: every sublayer is a function from a tensor of shape [b × tc × 768] to another tensor of the same shape. This is what allows the residual connections to be a plain element-wise addition. The "information superhighway" the residual stream creates is what makes gradient flow tractable in very deep models.
External reading: The Illustrated GPT-2 (Jay Alammar) · Hugging Face — GPT-2 model documentation · OpenAI — gpt-2 GitHub repository.
Layer Normalization in the LLM Architecture
Layer normalization is a per-token operation: for every position, subtract the mean of its 768-dimensional hidden state and divide by the standard deviation, then apply a learned scale and bias. It keeps activations from drifting during training, which is essential for stacks of 12+ transformer blocks. The paper is Ba, Kiros & Hinton (2016).
GPT-2 uses pre-norm (LayerNorm before the attention/FFN) instead of the original transformer's post-norm (LayerNorm after the residual addition). Pre-norm is now standard because it enables very deep networks to train without careful learning-rate warmup, as formalized in "On Layer Normalization in the Transformer Architecture" (Xiong et al., 2020). The lecturer also discusses RMSNorm (Zhang & Sennrich, 2019) as a faster variant that drops the mean-centering step.
# PyTorch built-in layer norm — works on the last dim self.ln = torch.nn.LayerNorm(emb_dim, eps=1e-5) # forward: x has shape [..., emb_dim] → still [..., emb_dim]
External reading: Ba, Kiros & Hinton 2016 — Layer Normalization · Xiong et al. 2020 — On Layer Normalization in the Transformer Architecture · Zhang & Sennrich 2019 — RMSNorm · torch.nn.LayerNorm documentation.
GELU Activation Function in the LLM Architecture
The feed-forward sublayer of a transformer is a two-layer MLP applied independently to every position. The non-linearity in GPT-2 is the Gaussian Error Linear Unit (GELU), not ReLU. GELU is x · Φ(x) — multiply the input by the standard-normal CDF of the input — and it has smoother gradients near zero than ReLU. The original paper is Hendrycks & Gimpel (2016).
GPT-2's feed-forward network expands the embedding dimension by 4× (768 → 3072) and projects back. The two weight matrices contain 2 × 768 × 3072 = 4,718,592 parameters per block, times 12 blocks = 56.6M parameters — almost half the model. The lecturer also covers the modern SwiGLU variant used in Llama.
External reading: Hendrycks & Gimpel 2016 — Gaussian Error Linear Units · torch.nn.GELU documentation · Shazeer 2020 — GLU Variants Improve Transformer (SwiGLU).
Shortcut Connections in the LLM Architecture
A shortcut (or residual) connection is the simplest and most powerful idea in deep learning: output = sublayer(x) + x. It guarantees that the gradient with respect to x is at least 1, so signal never fully vanishes through deep stacks. The original idea is ResNet (He et al., 2015).
In a transformer block the addition happens twice — once around attention, once around the feed-forward network. In a "pre-norm" block (the GPT-2 style) the pattern is: x = x + attn(ln(x)) followed by x = x + ffn(ln(x)). The lecturer also explains the alternative "post-norm" pattern and why modern architectures have converged on pre-norm.
External reading: He et al. 2015 — Deep Residual Learning for Image Recognition (ResNet) · Ba, Kiros & Hinton 2016 — Layer Normalization · CMU — Residual Networks explained (R. Salakhutdinov).
Coding the Entire LLM Transformer Block
Everything from the previous four lectures is now wrapped into a single TransformerBlock class in PyTorch. The forward pass is literally seven lines of code: two LayerNorms, one attention, one GELU FFN, and two additions. The lecturer also implements a dropout after the attention weights and after the FFN activations — a key regularizer that the original transformer paper found to be essential for generalization on small datasets.
The lecture also touches on the question of parameter sharing across layers (a hypothesis explored in "Deep Transformer Models Are Universal" (Dehghani et al.)) and the modern variant Mixture of Experts (MoE) used in GShard and Mixtral, where each layer contains multiple FFNs and a router picks which one to use per token.
External reading: torch.nn.Dropout documentation · Vaswani et al. 2017 — Section 3.1 (dropout = 0.1) · Jiang et al. 2024 — Mixtral of Experts.
Coding the 124-Million-Parameter GPT-2 Model
This is the climax of the architecture section: the GPTModel class wires together token embedding, positional embedding, a nn.ModuleList of 12 TransformerBlocks, a final LayerNorm, and the tied output projection. The configuration dictionary is exactly the one used in OpenAI's smallest public GPT-2 release:
GPT_CONFIG_124M = { "vocab_size": 50257, # BPE merges "context_length": 1024, # max sequence length "emb_dim": 768, # hidden size "n_heads": 12, # attention heads "n_layers": 12, # transformer blocks "drop_rate": 0.1, # dropout "qkv_bias": False # modern: no bias on Q/K/V }
The parameter count is verified: 124,412,928 trainable parameters, matching the published GPT-2 "small" checkpoint exactly. The lecture ends with a forward-pass sanity check on a dummy input of shape [2 × 4] (batch 2, four tokens) and the expected output shape [2 × 4 × 50257] (logits over the entire vocabulary at every position).
External reading: Hugging Face — gpt2 model card (124M parameters) · OpenAI's gpt-2/src/model.py reference · The Illustrated GPT-2.
Pretraining: The Training Loop, the Loss, and the First Text Generation
A model that has never seen a backward pass is just an expensive random number generator. Part V writes the training loop, the cross-entropy loss, and the evaluation harness — then asks the model to write its first paragraph of fake Shakespeare.
Coding GPT-2 to Predict the Next Token
The lecture begins with the simplest possible training objective: given a sequence of token ids, predict the next one. The model's output is a tensor of shape [b × tc × vocab_size]; the target is the input shifted one position to the left. The loss is the cross-entropy between the predicted softmax distribution and the true one-hot next token, averaged over all positions and all batch elements.
The lecturer implements two PyTorch helper functions: calc_loss_batch (computes the loss for a single batch) and calc_loss_loader (averages over a DataLoader). The shape gymnastics for the cross-entropy call — flattening the leading dimensions — are walked through step by step. The lecture also introduces F.cross_entropy, which combines log_softmax and nll_loss in a single numerically stable call.
# Cross-entropy for next-token prediction def calc_loss_batch(input_batch, target_batch, model, device): input_batch, target_batch = input_batch.to(device), target_batch.to(device) logits = model(input_batch) # [b × tc × vocab] return F.cross_entropy(logits.flatten(0, 1), # [b·tc × vocab] target_batch.flatten(),# [b·tc] ignore_index=-100)
External reading: F.cross_entropy documentation · Wikipedia: Cross-entropy · Brown et al. 2020 — Loss discussion in Section 2.2.
Measuring the LLM Loss Function
A model that has never been trained will produce a loss of exactly log(50257) ≈ 10.83 — the entropy of a uniform distribution over the vocabulary. As training progresses, the loss should drop monotonically (modulo small fluctuations from mini-batch noise). The lecture plots the training and validation loss curves of the 124M GPT-2 trained on TinyShakespeare and shows the standard sigmoidal learning curve.
Two important pitfalls are discussed: overfitting (training loss keeps falling while validation loss starts to rise) and data leakage (when evaluation text accidentally appears in the training set, producing a deceptively low validation loss). The lecturer also introduces perplexity = exp(loss) as the standard cross-dataset-comparable metric: a model that predicts the next token with perplexity 1 is perfect; one that is uniformly random over 50,257 tokens has perplexity 50,257.
External reading: Wikipedia: Perplexity of a probability model · Hugging Face — WikiText dataset · Brown et al. 2020 — Section 3 (Evaluation).
Evaluating LLM Performance on a Real Dataset — Hands-on Project with Book Data
The training corpus is upgraded from TinyShakespeare to Project Gutenberg's public-domain books. The lecturer writes a streaming-friendly load_dataset function, picks ~1 GB of clean text, tokenizes once with tiktoken, and splits the result 90/10 into train and validation tensors. The DataLoader is recreated with a 1,024-token context length.
The hands-on portion includes: setting up a learning-rate warmup (linearly increasing from 0 to the peak over the first ~1% of steps), weight decay of 0.1, and the AdamW optimizer with betas (0.9, 0.95) — the recommended configuration from "Scaling Language Models" (Kaplan et al., 2020) and the nanoGPT project. The lecturer also discusses gradient clipping (max norm = 1.0) to prevent occasional loss spikes.
External reading: torch.optim.AdamW documentation · Andrej Karpathy — nanoGPT training script · Kaplan et al. 2020 — Scaling Laws for Neural Language Models · Project Gutenberg.
Coding the Entire LLM Pre-training Loop
The full training loop is assembled from the helper functions built across the last three lectures: train_model_simple iterates for a fixed number of epochs, logs training and validation loss every eval_freq steps, saves the model checkpoint with torch.save, and prints a small sample of generated text every generate_freq steps. The lecture also covers two key engineering details: deterministic seeding (torch.manual_seed) and device placement (moving the model and batches to GPU with .to(device)).
The lecture also introduces the canonical text_to_token_ids and token_ids_to_text helpers, which wrap tiktoken's encode and decode to handle batch dimensions. By the end of the lecture, a 124M model has been trained from scratch on a single GPU for a few hours and is producing grammatically plausible, semantically confused, but recognizably "in-domain" text.
for epoch in range(num_epochs): model.train() for input_batch, target_batch in train_loader: optimizer.zero_grad() loss = calc_loss_batch(input_batch, target_batch, model, device) loss.backward() optimizer.step() tokens_seen += input_batch.numel()
External reading: PyTorch — Training a Classifier tutorial (training loop structure) · torch.cuda.amp.GradScaler (mixed precision) · nanoGPT — train.py reference.
Decoding — From Logits to Coherent Sentences
Once a model produces a probability distribution over the next token, how should we pick one? Greedy decoding, temperature scaling, and top-k sampling each reveal a different facet of the "creativity vs. coherence" trade-off.
Temperature Scaling in Large Language Models
The simplest text-generation strategy is greedy decoding: at every step, pick the token with the highest probability. It is fast, deterministic, and frequently degenerate — the model gets stuck in loops like "the the the". The next step up is sampling: sample the next token from the softmax distribution. This is what the original Transformer paper used with a temperature of 1.
Temperature scaling is a single parameter T that divides the logits before the softmax: softmax(logits / T). T = 1 is unchanged. T < 1 sharpens the distribution (more confident, more repetitive). T > 1 flattens it (more random, more creative). The optimal temperature depends entirely on the use case: factual QA prefers ~0.2, poetry prefers ~1.0, brainstorming prefers ~1.5. The technique is described in detail in "Improving Language Models by Retrieving from Trillions of Tokens" (Borgeaud et al., 2022) and is standard in every modern inference library.
External reading: Hugging Face — Generation strategies · Borgeaud et al. 2022 — Improving Language Models by Retrieving from Trillions of Tokens (RETRO) · Holtzman et al. 2020 — The Curious Case of Neural Text Degeneration (nucleus sampling).
Top-k Sampling in Large Language Models
Plain temperature sampling can still produce low-probability junk tokens if the tail of the distribution is long. Top-k sampling fixes this by zeroing out all but the k most probable tokens before renormalizing and sampling. Common values are k = 40 or k = 50. The technique was popularized by Fan, Lewis & Dauphin (2018).
The lecture also covers the more refined top-p (nucleus) sampling introduced in Holtzman et al. (2020): instead of a fixed k, keep the smallest set of tokens whose cumulative probability mass exceeds p (typically 0.9 or 0.95). Top-p adapts to the local sharpness of the distribution and is the default in most modern LLM chat interfaces. Finally, the lecturer demonstrates a generate_text_simple function that supports temperature, top-k, and top-p in a single call.
# Top-k and temperature in PyTorch def generate(model, idx, max_new_tokens, context_size, temperature=1.0, top_k=None): for _ in range(max_new_tokens): idx_cond = idx[:, -context_size:] logits = model(idx_cond)[:, -1, :] / temperature if top_k: v, _ = torch.topk(logits, min(top_k, logits.size(-1))) logits[logits < v[:, [-1]]] = -float('inf') probs = torch.nn.functional.softmax(logits, dim=-1) idx_next = torch.multinomial(probs, num_samples=1) idx = torch.cat((idx, idx_next), dim=1) return idx
External reading: Fan, Lewis & Dauphin 2018 — Hierarchical Neural Story Generation · Holtzman et al. 2020 — The Curious Case of Neural Text Degeneration (top-p) · Hugging Face — Generation strategies (sampling parameters).
Loading OpenAI's GPT-2 Weights — Standing on the Shoulders of Giants
Why train a 124M-parameter model from random initialization when OpenAI already published the weights? This part loads the official checkpoint into the model built in Part IV and verifies the arithmetic.
Loading Pre-trained Weights and Saving/Loading Model Weights Using PyTorch
The lecture opens with the most important PyTorch idiom in the entire series: torch.save(model.state_dict(), path) and model.load_state_dict(torch.load(path)). These two calls let a training run be paused, resumed, or shared. The state dict is just a Python dictionary mapping each named nn.Parameter to its tensor. Best practice is to save both the model state and the optimizer state (for resumption) and to use a versioned filename like model_pg_124M_50000_steps.bin.
The lecturer also discusses Hugging Face's save_pretrained / from_pretrained convention, which is the de facto standard for distributing open-weights models. Under the hood it is the same state_dict plus a config.json describing the architecture.
External reading: PyTorch — Saving & Loading Models tutorial · Hugging Face — Model save/load · torch.nn.Module state_dict documentation.
Loading Pre-trained Weights from OpenAI's GPT-2
OpenAI released four GPT-2 checkpoints in 2019 — 124M, 355M, 774M, and 1.5B parameters — and made them freely downloadable. The lecture downloads the smallest one (~500 MB), unpacks it with the NumPy np.load call on the provided .npz archive, and writes a load_weights_into_gpt function that maps each tensor from the OpenAI naming convention (wte, wpe, h.{i}.attn.c_attn.weight, …) to the lecture's nn.Module naming convention (tok_emb.weight, pos_emb.weight, trf_blocks[i].att.W_query.weight, …).
The trickiest mapping is the concatenated Q, K, V projection: OpenAI stores it as a single weight of shape [3 × 768 × 768] whereas the lecture's MultiHeadAttention class stores it as three separate matrices. The function splits the OpenAI tensor along its first axis and assigns each slice to W_query, W_key, W_value. After the load, the model is asked to generate text with the same prompt used in OpenAI's announcement, and the output matches almost word-for-word.
# OpenAI stores Q/K/V as one matrix, we split it q_w, k_w, v_w = np.split(c_attn_weight, 3, axis=0) model.tok_emb.weight = nn.Parameter(torch.from_numpy(wte)) model.pos_emb.weight = nn.Parameter(torch.from_numpy(wpe))
External reading: Hugging Face — gpt2 model card (link to original weights) · OpenAI — gpt-2/src/model.py (weight layout reference) · OpenAI Public — 124M checkpoint download.
From Pretrained Model to Shipping Product — Finetuning, Instruction-Tuning & Ollama
A pretrained base model is a fluent text completer — not a useful assistant. The final part of the series adapts that base model to two of the most common product surfaces (classification and instruction-following) and ships the result to a local inference runtime. It also wraps up with a 20-minute recap that ties every chapter together.
Introduction to LLM Finetuning — Python Coding with Hands-on Example
The transition from pretraining to finetuning is the single most important productization step in the LLM pipeline. The lecture defines the three canonical finetuning regimes: classification finetuning (replace the output head with a small linear classifier; train on labeled examples — e.g. spam vs. ham), instruction finetuning (keep the language-model head; train on prompt–response pairs in a chat format — e.g. Alpaca), and preference finetuning (use a pairwise preference signal — e.g. RLHF or DPO).
All three share a common pattern: take a pretrained base model, continue training it on a smaller, task-specific dataset with a much lower learning rate (e.g. 1e-5 instead of 3e-4), and only for a small number of epochs. The pretrained weights are the "prior" and the finetuning data is the "evidence". The lecturer contrasts this with training a model from scratch, which would require ~1000× more data and compute for the same end-task accuracy.
External reading: Devlin et al. 2018 — BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (the original finetuning recipe) · Hugging Face — Trainer documentation · Rafailov et al. 2023 — Direct Preference Optimization (DPO).
Dataloaders in LLM Classification Finetuning — Hands-on Project
For the classification-finetuning mini-project, the lecturer uses the SMS Spam Collection dataset — 5,572 SMS messages labeled ham or spam. The dataset is downloaded from the UCI Machine Learning Repository, loaded into a Pandas DataFrame, balanced (spam is the minority class), split 70/10/20 into train/val/test, and then tokenized with the same tiktoken BPE used during pretraining.
The PyTorch Dataset class for classification is different from the one for pretraining: each example returns a single tokenized message plus its class label (0 for ham, 1 for spam), not a (input, target) window pair. The DataLoader stacks them into a batch and pads shorter messages to the longest in the batch (with a <|endoftext|> token) so that the tensors stay rectangular. A custom custom_collate_fn handles the padding in one place.
def custom_collate_fn(batch, pad_token_id=50256, allowed_max_length=None): # Find longest sequence in batch max_len = max(len(x) for x in batch) inputs, targets = [], [] for item in batch: ids = item["input_ids"] label = item["label"] ids = ids[:allowed_max_length] ids += [pad_token_id] * (max_len - len(ids)) inputs.append(torch.tensor(ids)) targets.append(int(label)) return torch.stack(inputs), torch.tensor(targets)
External reading: Hugging Face — SMS Spam dataset · UCI — SMS Spam Collection · PyTorch — custom collate_fn documentation.
Coding the Model Architecture for LLM Classification Fine-tuning
The 124M GPT-2 is repurposed for classification by replacing the 50,257-way language-model head with a 2-way classification head. A new GPTForSequenceClassification class wraps the existing GPTModel and adds a single nn.Linear(emb_dim, num_classes) layer. The hidden state of the last token (or, equivalently, the hidden state at position -1) is fed into the new head — this is the standard "use the last position as the sequence summary" trick used in every decoder-only classifier.
Two design choices are discussed in depth. First, the out_head of the underlying GPTModel is not re-used or tied to the new classification head — they solve different problems (next-token vs. class label). Second, all other parameters (token embedding, positional embedding, every transformer block) are initialized from the OpenAI GPT-2 checkpoint loaded in Lecture 32, and only the new classification head is trained from scratch.
External reading: Hugging Face — GPT2ForSequenceClassification · Devlin et al. 2018 — Section 5.1 (classification finetuning) · torch.nn.Linear documentation.
Coding a Fine-tuned LLM Spam Classification Model from Scratch
The lecture closes the classification loop: the model is trained for five epochs with cross-entropy loss, the AdamW optimizer, a learning rate of 5e-5, and a batch size of 8. The training loop is the same one used for pretraining, only with labels of shape [b] instead of [b × tc]. After training, the model reaches 99%+ accuracy on the held-out test set and is saved to disk with torch.save.
The lecturer also covers two production considerations: calibration (a 99% accurate classifier is not necessarily well-calibrated — the expected calibration error should be measured) and robustness (a tiny character-level perturbation can flip a "spam" prediction to "ham"). Adversarial robustness is an active research area covered in "TextAttack" (Morris et al., 2020).
External reading: Morris et al. 2020 — TextAttack: A Framework for Adversarial Attacks in NLP · Wikipedia — Expected Calibration Error · TextAttack GitHub repository · scikit-learn — Calibration documentation.
Introduction to LLM Instruction Fine-tuning — Loading the Alpaca Dataset and Prompt Format
Where classification finetuning turns a base model into a labeler, instruction finetuning turns it into an assistant. The lecture introduces the Alpaca dataset — 52,000 prompt–response pairs distilled from GPT-3.5 (see Stanford CRFM announcement) — and the standard Alpaca prompt template:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
{response}
Each example is tokenized as a single sequence, the loss is computed on the response tokens only (the instruction part is masked with -100 to be ignored by cross_entropy), and the model is trained for three epochs. The result is a model that follows natural-language instructions in the style of GPT-3.5. The exact same recipe powers Dolly, OpenAssistant, and most open-weights chat models.
External reading: Stanford CRFM — Alpaca announcement · Hugging Face — Alpaca dataset · Ouyang et al. 2022 — InstructGPT (the original paper).
Data Batching in LLM Instruction Fine-tuning — Hands-on Project, Live Python Coding
Instruction-finetuning batches are built from heterogeneous-length prompt–response pairs. The lecturer writes a InstructionDataset class that: (1) loads Alpaca, (2) formats each example with the prompt template, (3) tokenizes it, (4) pads shorter sequences to the longest in the batch using the <|endoftext|> token (id 50,256), and (5) creates a target tensor identical to the input except that all prompt tokens (including the "\n### Response:" separator) are replaced with -100.
The collate function for instruction finetuning is more elaborate than the classification one: it must produce three tensors — input_ids, target_ids, and an optional attention_mask — and pad the inputs on the right (decoder-only models are trained right-padded, unlike encoder-only models like BERT which use left-padding for some tasks). The lecture ends with a sanity check that target_ids[i, j] == input_ids[i, j] everywhere except at the prompt positions.
External reading: Hugging Face — Padding and truncation · Hugging Face — Llama 2: instruction tuning details · F.cross_entropy — ignore_index parameter.
Dataloaders in Instruction Fine-tuning
Building on the previous lecture, the InstructionDataset is wrapped in a PyTorch DataLoader with a custom collate_fn. The lecturer also introduces a key engineering detail: drop_last_batch should be set to True for instruction finetuning because the variable batch shapes can confuse the Transformer if the final batch has length 1, and a final batch of length 1 is statistically inevitable with stochastic gradient descent.
The lecture also benchmarks different num_workers settings for the DataLoader. On a 16-core CPU and a single GPU, num_workers=4 usually saturates the data pipeline and yields the highest training throughput. Setting it higher wastes RAM and can actually decrease throughput due to multiprocessing overhead.
External reading: PyTorch — DataLoader documentation · PyTorch — Multiprocessing best practices · Hugging Face Datasets library.
Instruction Fine-tuning — Loading Pre-trained LLM Weights
Unlike the classification finetune of Lecture 35 — which only re-used the transformer body — instruction finetuning re-uses everything, including the original 50,257-way language-model output head. The class is GPTModel from Part IV, not GPTForSequenceClassification. The 124M OpenAI GPT-2 weights from Lecture 32 are loaded verbatim, the Alpaca dataset from Lecture 37 is fed in, and the loss is the same next-token cross-entropy as pretraining — only the data distribution and learning rate are different.
The lecture also discusses low-rank adaptation (LoRA) as a memory-efficient alternative to full finetuning. LoRA, introduced in Hu et al. (2021), freezes the pretrained weights and learns small rank-decomposition matrices (e.g. rank 8) that are added to the attention and FFN weight matrices. A 124M-parameter base model trains with only ~300K additional LoRA parameters, fitting on a single consumer GPU. The technique is implemented in the Microsoft LoRA library and the Hugging Face PEFT library.
External reading: Hu et al. 2021 — LoRA: Low-Rank Adaptation of Large Language Models · Hugging Face PEFT library · Ouyang et al. 2022 — InstructGPT.
LLM Fine-tuning Training Loop — Coded from Scratch
The instruction-finetuning training loop is the same as the pretraining loop from Lecture 28, with two small but important changes. First, the loss is computed with ignore_index=-100 so the prompt tokens do not contribute. Second, the evaluation step periodically generates a sample response to a held-out prompt and prints it — this is the only way to see whether the model is learning to follow instructions, because the loss can stay low while the model overfits to "predict the most common response token in the training set."
The lecture also introduces a critical engineering pattern: gradient accumulation. Modern instruction finetunes often use a per-device batch size of 1 and accumulate gradients over 8 or 16 micro-batches to simulate a global batch size of 8 or 16. This is essential for fitting instruction finetuning into a single consumer GPU. The pattern is implemented in Hugging Face Accelerate and in PyTorch's native mixed-precision.
# Gradient accumulation pattern optimizer.zero_grad() for i, (input_batch, target_batch) in enumerate(train_loader): loss = calc_loss_batch(input_batch, target_batch, model, device) loss = loss / grad_accum_steps # scale down loss.backward() # accumulate gradients if (i + 1) % grad_accum_steps == 0: optimizer.step() optimizer.zero_grad()
External reading: Hugging Face Accelerate · PyTorch — Automatic Mixed Precision examples · "Large Batch Optimization for Deep Learning" (Goyal et al., 2017).
Evaluating the Fine-tuned LLM Using Ollama
The final technical lecture of the series. Ollama is an open-source runtime that packages an LLM (weights + architecture + chat template) into a single .gguf file and serves it over a local REST API. The lecturer walks through: (1) installing Ollama, (2) saving the instruction-finetuned model to a local directory, (3) converting the PyTorch state dict to the GGUF format with the llama.cpp convert.py script, (4) writing a Modelfile that describes the chat template and inference parameters, and (5) calling the model with the Ollama REST API.
The evaluation methodology is qualitative but instructive. The lecturer gives the same prompt ("Write a function in Python that checks whether a string is a palindrome") to the base GPT-2, the instruction-finetuned GPT-2, and Ollama's Llama 2 7B. The base model produces unrelated text. The instruction-finetuned model produces a syntactically correct, semantically correct answer — the unmistakable signature of successful instruction finetuning. The Llama 2 model produces a more polished, commented, and tested answer — the unmistakable signature of a much larger model.
External reading: Ollama — official site · llama.cpp — GGUF converter · EleutherAI LM Evaluation Harness · Hugging Face Open LLM Leaderboard.
Build LLMs from Scratch — 20-Minute Summary
The closing lecture is a 20-minute fast recap of the entire 42-lecture arc, designed to be watched the day before a job interview or a project kickoff. The lecturer condenses each part into one or two sentences:
- Foundations — A language model is a probability distribution over token sequences; a transformer is a stack of attention + feed-forward blocks; the modern pipeline is pretrain → finetune.
- Data — BPE tokenization turns text into integers; embeddings turn integers into dense vectors; positional embeddings restore order.
- Attention — A soft dictionary lookup that lets every token "ask a question" of every previous token; multi-head attention runs many such lookups in parallel.
- Architecture — LayerNorm + residual streams + GELU + dropout = a stable, deep, trainable transformer. Stack 12 of them, you have GPT-2.
- Pretraining — Cross-entropy loss, AdamW optimizer, learning-rate warmup, weight decay, gradient clipping. Train for days on a large corpus.
- Decoding — Temperature and top-k/top-p sampling turn logits into text. Greedy is fast and bad; sampling is slow and good.
- Reuse — Don't pretrain from scratch. Load OpenAI's weights, replace the head, finetune on your data.
- Product — Classification finetuning for labelers, instruction finetuning for assistants, Ollama for shipping.
The lecturer closes with a thought: building an LLM from scratch is not the goal — the goal is to understand the artifact well enough to be unfooled, to debug, and to extend it. The code in this series is the price of admission. The intuition is the dividend. He recommends three follow-up paths: nanoGPT for minimum-viable pretraining, Hugging Face Transformers for production code, and Alpaca for instruction data.
External reading: Andrej Karpathy — nanoGPT · Hugging Face Transformers · Stanford Alpaca · Kaplan et al. 2020 — Scaling Laws for Neural Language Models.
A Note on the Reading Experience
This document is a single, self-contained HTML file: no external scripts, no external fonts, no build step. Every lecture is hyperlinked to the most authoritative public reference available — academic papers, library documentation, blog posts from the original authors, and the canonical reference implementations. Open any of the highlighted links in a new tab and the depth of each topic opens up to the level required by an actual research or engineering project.
The visual language borrows from long-form editorial design: a serif body face, an editorial red accent, double horizontal rules, a strict 12-column grid, and pull-quotes for the moments that benefit from setting them apart. The intent is to make a long technical document feel like a feature in a serious magazine — the kind you would print, bind, and read on a flight.
If you are a new reader, the recommended path is: Part I for the mental model, Part III for the single most important mechanism (attention), Part IV for the full architecture, and Part VIII for the product. The remaining parts fill in the supporting detail.
Comments
Post a Comment