pinned · cognitive science
Can AI be creative?
What happens in a brain that has a new idea, where recombination stops, and what the measurements of machine creativity show, from AlphaGo's move 37 to GPT-4 on the Alternate Uses Task
Eva Koroleva's blog
pinned · system design
LLM inference optimization: where the time and memory go
Why generating text is limited by memory rather than arithmetic, measured on a free GPU, and what serving engines like vLLM do about it: the KV cache, continuous batching, quantization and speculative decoding
pinned · finance
How accurate is Polymarket? Calibration, longshot bias and a volume trap
Polymarket's prices checked against what happened across almost three million resolved markets: how well calibrated they are, what accuracy figures hide, where longshots and favourites are mispriced, insiders, oracles and the cost of trading
November 2026
The Vision Transformer reads images as sequences of patches
How a Transformer reads a photo: every step from pixels to a sequence of 197 vectors on a real image, written from scratch and checked against Google's ViT-B/16. What the model learned about patches and positions, where its heads look, DINO's heads finding objects without labels, and ViT against a CNN on satellite images when labels are scarce.
How accurate is Polymarket? Calibration, longshot bias and a volume trap
Polymarket's prices checked against what happened across almost three million resolved markets: how well calibrated they are, what accuracy figures hide, where longshots and favourites are mispriced, insiders, oracles and the cost of trading
AI crypto trading bots: what machine learning and LLMs can and can't do
AI trading bots and LLM agents tested on six years of Binance data: what machine learning predicts, what trading fees eat, and why LLM backtests lie
October 2026
Beyond attention with Mamba, state space models and hybrids
State space models from the continuous equation up: discretisation, Mamba's selective scan written from scratch and checked against mamba-130m, what its time steps choose to remember, generation with a state of fixed size, Mamba-2 as masked attention, Mamba against Pythia on the same data, hybrids, and what selectivity buys, learned from scratch on selective copying.
September 2026
Long context with RoPE scaling, YaRN and FlashAttention
Why a language model breaks past the length it was trained on, and how RoPE scaling fixes it. Position interpolation, NTK scaling, YaRN and Llama 3.1's scaling written from scratch and checked against transformers, raced on TinyLlama; FlashAttention built from the online softmax; Qwen3 finding a needle in a novel up to 64k tokens.
The KV cache budget and DeepSeek's latent attention
What one token costs in the KV cache of ten models, and how DeepSeek's multi-head latent attention cuts it. MLA written from scratch in both its forms, why rotary positions need their own dimensions, MiniCPM3-4B with its official weights, a latent cache against a rebuilt one on a T4, Qwen3 compressed without training, and a quantized cache.
August 2026
Mixture-of-experts LLMs from scratch with OLMoE
How a dozen production mixtures of experts are built, from Mixtral to gpt-oss. The router written four ways, OLMoE in PyTorch with its official weights, the experts that English, Python, maths and Russian use, a model pruned for one kind of text, and an expert cache measured on real routing.
Modern LLM architecture and Qwen3 from scratch
What Mistral, Gemma and Qwen changed after Llama, read from a dozen configs. Grouped-query attention, sliding windows and attention sinks, QK-norm and soft-capping, Qwen3-0.6B written from scratch with its official weights, and LoRA that turns voice commands into JSON in English and Russian.
July 2026
Theories of consciousness and the hard problem
The hard problem of consciousness for readers used to physicalism. Thought experiments from Mary's room to fading qualia, positions from illusionism to panpsychism, the scientific theories, the first experiment that tested two of them against each other, and what they imply for AI
June 2026
How Karl Friston's free energy principle works and what its critics say
Karl Friston's free energy principle with exact formulas and plain words. Surprisal and variational free energy, its link to the ELBO of machine learning, predictive coding on a worked example, active inference, Markov blankets, the experimental evidence and the main criticisms
How similar is the human brain to AI models?
A decade of neuroscience has compared brains with neural networks. Language models predict the brain's language network, vision models the visual cortex and reinforcement learning the dopamine signal, while memory, bodies, energy and lifelong learning are where the two still differ most
May 2026
T5 and BART: the encoder–decoder, and a model that writes SQL
T5 written from scratch and loaded with Google's weights: relative position buckets, cross-attention, span corruption and BART's text infilling, and a T5 fine-tuned on Spider to write SQL, checked by running every query on SQLite.
April 2026
GPT from scratch: GPT-2, the KV cache and a model that learns chess
GPT-2 written from scratch and loaded with OpenAI's weights, generation with a KV cache, the logit lens, and a small GPT trained on Lichess games that learns to play legal chess and keeps a board inside, found with a linear probe.
Embeddings and search: Sentence-BERT, cross-encoders and ColBERT
How a search engine reads: BM25 written from scratch, Sentence-BERT's bi-encoder trained with in-batch negatives, a cross-encoder reranker and ColBERT's late interaction, each checked against the real weights and put together into a literature search over 64,000 machine-learning papers.
March 2026
From RoBERTa to ModernBERT: what seven years did to BERT
What seven years did to BERT: RoBERTa's training recipe, ELECTRA's replaced-token detection, DeBERTa's disentangled attention and ModernBERT's LLM-era block, each written in PyTorch and checked against the real weights, and the four compared on triaging GitHub issues of projects they never saw.
February 2026
BERT from scratch: masked language modelling and fine-tuning
BERT written in PyTorch from scratch and loaded with Google's weights: why reading in both directions needed a mask, what BERT knows and what its attention heads do, and a detector of personal data in long financial documents, against regular expressions.
Mixture of experts: why the largest LLMs use a few percent of their weights per token
A mixture-of-experts layer in PyTorch trained next to dense models: the router, load balancing, dropped tokens, what experts learn, and serving
January 2026
Hybrid attention: why new LLMs keep full attention in only a few layers
Linear attention, the delta rule and Gated DeltaNet in PyTorch, a recall test, GPU timings, and why Qwen3-Next and Kimi Linear keep few attention layers
December 2025
It works on my laptop: a reproducible Docker image of a model
Ship a model so that it runs the same next year: a lock file with hashes, an image pinned by digest, builds that give the same bytes twice, GitHub Actions publishing to GHCR, the image pulled and run without Docker, and the model exported to ONNX for a free in-browser demo.
Is it rational to be good? Game theory and the ethics of cooperation
Glaucon's challenge, Hobbes's Foole and Hume's farmers read through the prisoner's dilemma: when cooperation is rational, which virtues the models select, how much forgiveness reason allows, and what game theory cannot settle
November 2025
Serving a Transformer with FastAPI and Gradio: latency under load
Put a Transformer behind an HTTP API and measure it the way users meet it: a KV cache in beam search, int8 quantization that doesn't pay off, FastAPI and asyncio, open-loop load tests, queueing theory checked against the measurements, dynamic batching, and a Gradio page.
Your notebook is not a product: a Python package with tests and CI
Turn a model from a notebook into a Python package anyone can install: the src layout, pyproject.toml, a command line, four kinds of tests checked by mutation testing, GitHub Actions, and a model card that measures what the model can't do.
October 2025
Is Centaur a model of the mind? An LLM fine-tuned on 160 psychology experiments
In 1973 Allen Newell asked psychology for one system that performs all its tasks. In 2025 a Llama fine-tuned on 160 experiments was presented as that system. What Centaur predicts, what happens when you take away its instructions, and why its own authors went on to beat it with half a million parameters
September 2025
Hyperparameter tuning on a free GPU with W&B Sweeps
Spend an hour of a free GPU on hyperparameters: experiment tracking with Weights & Biases, a learning-rate range test, random search and successive halving from scratch, and W&B Sweeps with Bayesian optimization.
What Llama changed in the Transformer: RoPE, RMSNorm, SwiGLU
Pre-norm, RMSNorm, SwiGLU and RoPE: the changes Llama made to the 2017 Transformer, written in PyTorch, added to our translator one at a time and measured against what a new random seed does alone, with fused attention and a KV cache.
August 2025
Evaluating machine translation with chrF and COMET
Evaluate English–Russian translation with BLEU, chrF and COMET written from scratch: multiple references, agreement with professional translators, significance tests and a contaminated test set.
July 2025
Training a Transformer: warm-up, label smoothing, beam search, sampling
Train a Transformer on half a million sentence pairs on a free GPU, then decode it with greedy search, beam search and sampling.
Implementing the Transformer in PyTorch: why attention divides by √d_k
Attention, masks, heads, positions and the full encoder-decoder of the 2017 paper, written line by line in PyTorch and checked against real models.
June 2025
How language models read: tokenization and BPE from scratch
What GPT-4o actually receives when you type, why Russian costs more tokens than English, and byte-pair encoding written from scratch.
May 2025
Damasio's somatic marker hypothesis: the body in decision-making
A patient who could reason about everything and decide nothing, a card game whose decks are rigged, and a theory that puts the body inside decisions: Damasio's somatic markers, the evidence for them and the evidence against
LLM inference optimization: where the time and memory go
Why generating text is limited by memory rather than arithmetic, measured on a free GPU, and what serving engines like vLLM do about it: the KV cache, continuous batching, quantization and speculative decoding
April 2025
Parameter-efficient fine-tuning in PyTorch: adapters, LoRA and the rest
How to adapt a pretrained transformer by training a fraction of a percent of it: what adapters, LoRA, prefix and prompt tuning, BitFit, (IA)³ and Compacter add to the model, written out in PyTorch and measured on one task
Is the mind a computer? The computational theory of mind
Symbols, rules and meaning: the computational theory of mind from McCulloch and Pitts to the Chinese Room, and what language models add to an old argument
March 2025
Building a Transformer in PyTorch, layer by layer
Positional encoding, self-attention, multi-head attention and cross-attention, then the encoder and decoder layers, written in PyTorch one module at a time.
February 2025
Byte-pair encoding: writing a BPE tokenizer in Python
Byte-pair encoding written in Python step by step, trained on English–Russian sentences from Tatoeba and compared with the tokenizers of Hugging Face.
January 2025
Can AI be creative?
What happens in a brain that has a new idea, where recombination stops, and what the measurements of machine creativity show, from AlphaGo's move 37 to GPT-4 on the Alternate Uses Task
December 2024
The algebra of LoRA: how low-rank adaptation works
Why a fine-tuning update can be the product of two thin matrices: rank, the singular value decomposition and the Eckart–Young theorem, then LoRA's initialization, gradients and scale, derived by hand and checked on BERT
How large language models work, for those who missed out
What a large language model is, how it is trained in three stages, and what it could and could not do in spring 2024: a tour after Andrej Karpathy's talk, with the claims checked against their sources and measured on small open models
November 2024
How to write clean Python code
Naming, comments, modules, tests, docstrings, exceptions and keyword arguments: the basic habits that keep Python code readable and easy to change