Set your Publisher ID in src/config/ads.ts to display live ads.
Autoregressive LLMs & Next-Token Prediction
Tokenization, BPE, causal masking, and probabilistic sampling
Large Language Models are autoregressive decoder-only Transformers trained on next-token prediction: P(w_t | w_1, w_2, ..., w_{t-1}).
The Autoregressive Generation Loop
Byte-Pair Encoding (BPE): Text is broken down into sub-word tokens (~0.75 words per token in English). Words like "unbreakable" become ["un", "break", "able"].
Causal Attention Masking: The attention mechanism is constrained so that during training, token t can only attend to previous tokens 1 through t-1, preventing information leakage from the future.
Sampling Strategies: The model outputs logits for every vocabulary word (typically 32k to 128k tokens). Softmax converts logits into probabilities. Temperature, Top-K, and Top-P (nucleus sampling) determine how the next token is picked.
import numpy as np
def sample_with_temperature(logits, temperature=1.0):
# Temperature scales logits before softmax: low temp -> greedy; high temp -> creative
scaled = np.array(logits) / max(temperature, 1e-5)
exp_scaled = np.exp(scaled - np.max(scaled))
probs = exp_scaled / np.sum(exp_scaled)
return probs
logits = [2.0, 4.0, 1.0, 0.5]
print("Greedy temp (0.2):", np.round(sample_with_temperature(logits, 0.2), 3))
print("Creative temp (1.5):", np.round(sample_with_temperature(logits, 1.5), 3))Setting temperature to 0.0 results in greedy argmax decoding, producing deterministic outputs for factual extraction tasks.
Set your Publisher ID in src/config/ads.ts to display live ads.
Test Yourself With Exercise
What does a causal attention mask prevent an autoregressive decoder model from doing during pretraining?