Can AI be creative?
What happens in a brain that has a new idea, where recombination stops, and what the measurements of machine creativity show, from AlphaGo's move 37 to GPT-4 on the Alternate Uses Task
Eva Koroleva's blog
AI crypto trading bots: what machine learning and LLMs can and can't do
AI trading bots and LLM agents tested on six years of Binance data: what machine learning predicts, what trading fees eat, and why LLM backtests lie
Mixture of experts: why the largest LLMs use a few percent of their weights per token
A mixture-of-experts layer in PyTorch trained next to dense models: the router, load balancing, dropped tokens, what experts learn, and serving
Hybrid attention: why new LLMs keep full attention in only a few layers
Linear attention, the delta rule and Gated DeltaNet in PyTorch, a recall test, GPU timings, and why Qwen3-Next and Kimi Linear keep few attention layers
It works on my laptop: a reproducible Docker image of a model
Ship a model so that it runs the same next year: a lock file with hashes, an image pinned by digest, builds that give the same bytes twice, GitHub Actions publishing to GHCR, the image pulled and run without Docker, and the model exported to ONNX for a free in-browser demo.
Is it rational to be good? Game theory and the ethics of cooperation
Glaucon's challenge, Hobbes's Foole and Hume's farmers read through the prisoner's dilemma: when cooperation is rational, which virtues the models select, how much forgiveness reason allows, and what game theory cannot settle
Serving a Transformer with FastAPI and Gradio: latency under load
Put a Transformer behind an HTTP API and measure it the way users meet it: a KV cache in beam search, int8 quantization that doesn't pay off, FastAPI and asyncio, open-loop load tests, queueing theory checked against the measurements, dynamic batching, and a Gradio page.
Your notebook is not a product: a Python package with tests and CI
Turn a model from a notebook into a Python package anyone can install: the src layout, pyproject.toml, a command line, four kinds of tests checked by mutation testing, GitHub Actions, and a model card that measures what the model can't do.
Is Centaur a model of the mind? An LLM fine-tuned on 160 psychology experiments
In 1973 Allen Newell asked psychology for one system that performs all its tasks. In 2025 a Llama fine-tuned on 160 experiments was presented as that system. What Centaur predicts, what happens when you take away its instructions, and why its own authors went on to beat it with half a million parameters
Hyperparameter tuning on a free GPU with W&B Sweeps
Spend an hour of a free GPU on hyperparameters: experiment tracking with Weights & Biases, a learning-rate range test, random search and successive halving from scratch, and W&B Sweeps with Bayesian optimization.
What Llama changed in the Transformer: RoPE, RMSNorm, SwiGLU
Pre-norm, RMSNorm, SwiGLU and RoPE: the changes Llama made to the 2017 Transformer, written in PyTorch, added to our translator one at a time and measured against what a new random seed does alone, with fused attention and a KV cache.
Evaluating machine translation with chrF and COMET
Evaluate English–Russian translation with BLEU, chrF and COMET written from scratch: multiple references, agreement with professional translators, significance tests and a contaminated test set.
Training a Transformer: warm-up, label smoothing, beam search, sampling
Train a Transformer on half a million sentence pairs on a free GPU, then decode it with greedy search, beam search and sampling.
Implementing the Transformer in PyTorch: why attention divides by √d_k
Attention, masks, heads, positions and the full encoder-decoder of the 2017 paper, written line by line in PyTorch and checked against real models.
How language models read: tokenization and BPE from scratch
What GPT-4o actually receives when you type, why Russian costs more tokens than English, and byte-pair encoding written from scratch.
Damasio's somatic marker hypothesis: the body in decision-making
A patient who could reason about everything and decide nothing, a card game whose decks are rigged, and a theory that puts the body inside decisions: Damasio's somatic markers, the evidence for them and the evidence against
LLM inference optimization: where the time and memory go
Why generating text is limited by memory rather than arithmetic, measured on a free GPU, and what serving engines like vLLM do about it: the KV cache, continuous batching, quantization and speculative decoding
Parameter-efficient fine-tuning in PyTorch: adapters, LoRA and the rest
How to adapt a pretrained transformer by training a fraction of a percent of it: what adapters, LoRA, prefix and prompt tuning, BitFit, (IA)³ and Compacter add to the model, written out in PyTorch and measured on one task
Is the mind a computer? The computational theory of mind
Symbols, rules and meaning: the computational theory of mind from McCulloch and Pitts to the Chinese Room, and what language models add to an old argument
Building a Transformer in PyTorch, layer by layer
Positional encoding, self-attention, multi-head attention and cross-attention, then the encoder and decoder layers, written in PyTorch one module at a time.
Byte-pair encoding: writing a BPE tokenizer in Python
Byte-pair encoding written in Python step by step, trained on English–Russian sentences from Tatoeba and compared with the tokenizers of Hugging Face.
The algebra of LoRA: how low-rank adaptation works
Why a fine-tuning update can be the product of two thin matrices: rank, the singular value decomposition and the Eckart–Young theorem, then LoRA's initialization, gradients and scale, derived by hand and checked on BERT
How large language models work, for those who missed out
What a large language model is, how it is trained in three stages, and what it could and could not do in spring 2024: a tour after Andrej Karpathy's talk, with the claims checked against their sources and measured on small open models
How to write clean Python code
Naming, comments, modules, tests, docstrings, exceptions and keyword arguments: the basic habits that keep Python code readable and easy to change