This chapter develops the mathematical foundations of language models from the ground up, with no prior ML background assumed. A language model is a probability-conditioned autocomplete: given everything that has been said so far, it estimates the probability of every word that could come next, and samples one. Repeat the process, and you get a sentence; repeat the sentence, and you get a paragraph; repeat the paragraph, and you get a chat-bot that appears to "reason."
Figure 1. Concept map for this episode: fight, desk translation, what to lock.
Tokens, not words
The first thing to understand: language models do not operate on words. They operate on tokens - subword units chosen by a tokenisation algorithm to balance vocabulary size against sequence length. A typical tokeniser breaks English text into roughly 0.7 tokens per word (English has many common short words). The word "trading" might be a single token; "anticipation" might split into "antic" + "ipation"; rare technical terms split into more pieces. The model's vocabulary is fixed at training time, typically 30,000 to 100,000 tokens. Indian-language text usually splits into many more tokens per word than English, since most models are trained primarily on English text - which is why some multilingual models ship specialised, larger vocabularies to handle Indic scripts more efficiently. The choice of tokeniser is essentially irrevocable: changing it means retraining the entire model.
The next-token prediction task
Given a sequence of tokens, the language model outputs a probability distribution over the next token. Under the hood, the neural network produces a raw score (a "logit") for every token in the vocabulary, and a function called softmax converts those scores into genuine probabilities: it exponentiates each score and divides by the sum of all the exponentiated scores, so the results are all positive and add up to exactly 1. Bigger score gaps translate into much bigger probability gaps - a logit that's just 1 point higher than another already gets roughly 2.7 times the probability - which is what lets the model be sharply confident when it should be and spread its bets when it should be uncertain. At inference, the model samples one token from this distribution, appends it to the sequence, and predicts the next token. The process repeats until the model emits a special "end" token or hits a length limit. This is the entire core mechanism; everything else is engineering.
The transformer architecture
Modern language models are built on the transformer architecture, introduced in the 2017 paper "Attention is All You Need." The key innovation is the self-attention mechanism, which lets each token in the input attend to every other token, producing context-aware representations.
Training: pretraining and fine-tuning
Language models are trained in two phases. Pretraining: the model learns to predict the next token on a massive corpus of text (the Common Crawl, Wikipedia, code repositories, books), scored by how surprised it is by each true next word - a model that consistently puts high probability on what actually comes next is learning well. Pretraining runs for weeks on hundreds of GPUs and leaves the model with an extraordinary amount of implicit world knowledge, but it isn't particularly useful yet - it just continues text in whatever style the input suggests. Fine-tuning: a second phase adapts the pretrained model to specific tasks - chat, instruction-following, domain expertise - using far less data (thousands to millions of examples) and running in hours to days rather than weeks.
Definition
A token is the basic unit a language model operates on - typically a subword segment. The tokeniser is the algorithm that splits text into tokens; the model never sees raw text, only sequences of integer token IDs. Two common tokenisation schemes are byte-pair encoding (BPE) and SentencePiece. Both work by starting with the most common subword pieces and progressively merging them into larger pieces based on co-occurrence frequency in the training corpus. The result is a vocabulary that compresses common words into single tokens and represents rare words as sequences of smaller pieces.
Worked picture
Suppose the vocabulary has three tokens with logit scores of 2.0, 1.0, and 0.0. Softmax converts these to probabilities of about 66.5%, 24.5%, and 9.0%. The largest logit captures 66.5% of mass - why sharp finance settings (Chapter 46) matter: one wrong token in a ticker field can dominate downstream JSON validity.
What breaks in production
A common misunderstanding: people think the pretrained model is the chatbot they interact with. It is not. The pretrained model is a raw next-token predictor - if you give it a question, it might continue with more questions (because question-mark tokens often precede more questions in web text) rather than answering. What makes a pretrained model into a useful chatbot is instruction fine-tuning (a form of supervised fine-tuning on prompt/response pairs) followed by alignment (RLHF or DPO; Chapter 45). These post-training steps are what transform "very good at next-token prediction" into "very good at answering questions helpfully." When someone says "ChatGPT can do X," they usually mean "the fine-tuned and aligned version that OpenAI deploys can do X." The base model behind it would behave quite differently. This distinction matters for anyone building on open-source base models: the base model is a starting point, not a finished product.
Key idea
A language model is next-token prediction at scale - pretraining learns the corpus; fine-tuning and alignment make it useful. In a trading stack it occupies only a small slice of the total effort, and that slice pays off only if the ontology and evaluation behind it are honest.