Can AI Be Creative?
What happens in a brain that has a new idea, where recombination stops, and what the measurements of machine creativity show, from AlphaGo's move 37 to GPT-4 on the Alternate Uses Task
Eva Koroleva's blog
Hyperparameter tuning on a free GPU with W&B Sweeps
Spend an hour of a free GPU on hyperparameters: experiment tracking with Weights & Biases, a learning-rate range test, random search and successive halving from scratch, and W&B Sweeps with Bayesian optimization.
What Llama changed in the Transformer: RoPE, RMSNorm, SwiGLU
Pre-norm, RMSNorm, SwiGLU and RoPE: the changes Llama made to the 2017 Transformer, written in PyTorch, added to our translator one at a time and measured against what a new random seed does alone, with fused attention and a KV cache.
Evaluating machine translation with chrF and COMET
Evaluate English–Russian translation with BLEU, chrF and COMET written from scratch: multiple references, agreement with professional translators, significance tests and a contaminated test set.
Training a Transformer: warm-up, label smoothing, beam search, sampling
Train a Transformer on half a million sentence pairs on a free GPU, then decode it with greedy search, beam search and sampling.
Implementing the Transformer in PyTorch: why attention divides by √d_k
Attention, masks, heads, positions and the full encoder-decoder of the 2017 paper, written line by line in PyTorch and checked against real models.
How language models read: tokenization and BPE from scratch
What GPT-4o actually receives when you type, why Russian costs more tokens than English, and byte-pair encoding written from scratch.
Somatic Marker Hypothesis by Antonio Damasio
A patient who could reason about everything and decide nothing, a card game whose decks are rigged, and a theory that puts the body inside decisions: Damasio's somatic markers, the evidence for them and the evidence against
LLM Inference Optimization
Where the time and the memory go when a language model serves users, measured on a real GPU, and what serving engines do about it
Parameter-Efficient Fine-Tuning: Adapters, LoRA and the Rest, in PyTorch
How to adapt a pretrained transformer by training a fraction of a percent of it: what adapters, LoRA, prefix and prompt tuning, BitFit, (IA)³ and Compacter add to the model, written out in PyTorch and measured on one task
Is the Mind a Computer?
Symbols, rules and meaning: the computational theory of mind from McCulloch and Pitts to the Chinese Room, and what language models add to an old argument
Building a Transformer (Cross-Attention and MHA Explained)
Learn and implement the most iconic architecture in modern deep learning.
Understanding Byte-Pair Encoding Algorithm
Implement one of the most popular tokenization algorithms and learn how to use ready-made solutions.