Applied Machine Learning

A practitioner's course: statistics through to production, in 27 modules and 250 questions.

It assumes you can write Python and want to build systems that work, not pass an exam. Concepts are explained to the depth needed to use them and no further — where a topic deserves a book you get the shape of it and a pointer, and where a detail decides whether your system works you get the detail.

Route

Modules 00–09 run in order. After that each module names its prerequisites at the top, so you can take them as you need them — 17 (Tokenization) sits between 04 and 07, for instance, and 21 follows 07.

Modules 09 → 11 → 15 → 16 form the deployment chain: track it, containerise it, orchestrate it, then watch it degrade.

Module 99 collects claims that are commonly stated wrong. It is not an appendix — read it after module 03, because it teaches a habit the rest of the course relies on.

How to use it

Reading straight through will teach you less than testing yourself will. The rhythm that works:

1. Read a module once, fast, for the shape.
2. Close it. Open that module's question set and answer from memory, in writing. Out loud is fine; in your head is not — "I knew that" is unfalsifiable and always feels true.
3. Mark each question Got it or Missed. Scoring is binary and harsh: "I nearly had it" is a miss.
4. Read your misses in Your gaps, and go back to those sections.

Step 3 is the mechanism. A gap you found by failing a question is worth ten topics you picked because they sounded important.

Two habits that cost nothing and get skipped. Predict before you read — before the section on precision and recall, write down what you think the difference is; being wrong in writing is the only reliable way to notice you were wrong at all. And assume something here is wrong — module 99 lists claims that are routinely mangled, several of which will contradict something you have read elsewhere.

Your scores are stored in your browser and go nowhere else.

What to expect

Some of it will go stale. Library APIs move; module 99 treats that as a permanent condition rather than a defect, and version-specific claims are marked as true at time of writing.

Some of it is wrong. Anything this size, about a field moving this fast, has errors in it. Module 99 lists what has been corrected so far. If you find another, you have used the course correctly.

What this course covers

Every topic, and the module that teaches it. Modules 00–09 run in order; after that each module names its prerequisites at the top.

Module 99 collects claims that are commonly stated wrong, plus corrections made to this course since it was written.

Tracking your own progress

Scoring the question sets records what you missed in Your gaps — a list built from your recall rather than from a syllabus. Nothing is uploaded; it stays in your browser.

00 — Orientation

Why it matters

Most of the confusion in early ML is vocabulary, not mathematics. "Is this a classification problem?" has an answer; getting it wrong costs you a week. This module is the map — the terms you need before any other module makes sense.

What it is

Machine learning is teaching a computer to find patterns in data and act on them, rather than writing the rules yourself. You supply examples; the training process supplies the rules.

The first fork is what your data looks like, and it determines everything downstream.

The four learning types

Supervised splits again by what the label is:

• Regression — the label is a number. House price, delivery time, temperature.
• Classification — the label is a category. Spam/not-spam, which of four ticket types, which digit.

Unsupervised splits by what structure you're after:

• Clustering — group similar records. Customer segments, document topics.
• Dimensionality reduction — same records, fewer columns. Needed when you have hundreds of features and most carry no signal.
• Anomaly detection — find the records that don't fit. Fraud, faults, intrusions. (Sometimes miscalled "anomaly reduction" — the task is detection, not reduction.)
• Association — find things that co-occur. The supermarket-basket analysis that every textbook introduces with diapers and beer.

Semi-supervised deserves a caution that introductions usually skip. "Label one point and the rest label themselves" is the intent, not the guarantee. The propagation assumes points near each other in feature space share a label, and when that assumption fails you have confidently mislabelled your entire dataset.

Reinforcement learning is the odd one out: no fixed dataset, an agent acting in an environment, learning from reward and penalty. AlphaGo beating the Go champion is the canonical demonstration. It is not what you will use at work, and it is not covered further here.

Picking the type

Run these in order, and stop at the first yes:

1. Do I have labelled examples of the exact thing I want to predict? → supervised.
2. Do I have a few labels and a lot of unlabelled data? → semi-supervised, or label more data. Usually: label more data.
3. Am I looking for structure rather than a specific answer? → unsupervised.
4. Am I choosing actions whose consequences arrive later? → reinforcement.

Most business problems are (1). Many problems that look like (3) are actually (1) where nobody wanted to pay for labels.

What breaks

• Solving the wrong problem type. "Predict which tickets will be delayed" is classification if you want delayed/not-delayed and regression if you want how many hours. These need different models, different metrics, and different data. Decide before you write code.
• Assuming clustering will produce meaningful groups. K-means always returns k clusters. It has no way to tell you the structure isn't there, and it will never say "these points have nothing in common."
• Treating semi-supervised as free labels. See above. If the similarity assumption is wrong, errors propagate silently.
• Vocabulary drift. "Feature", "attribute", "variable", "column", "predictor", "independent variable" and "input" are the same thing. "Label", "target", "class", "response" and "dependent variable" are the same thing. Papers switch mid-paragraph.

Standard workflow

Every project in module 08 follows this shape:

1. Collect — load data (pandas, NumPy).
2. Analyse — look at distributions and correlations before modelling (seaborn, matplotlib). Skipping this is the most expensive shortcut available.
3. Clean — handle missing values, drop useless columns, fix types.
4. Split — train/test before any fitting. Module 03 explains why the order matters more than it looks.
5. Train — fit the model.
6. Evaluate — the right metric, on held-out data.
7. Deploy — and then monitor, which introductions routinely skip and module 16 covers.

Steps 1–3 are where the time goes. Step 5 is one line.

How to study this

Two habits, both cheap, both skipped:

Predict before you read. Before a section on precision and recall, write down what you think the difference is. Being wrong in writing is the only reliable way to notice you were wrong at all.

Assume any explanation is wrong somewhere. Module 99 collects the claims most often gotten wrong, including in this tutorial. Reading with an eye for the error is a completely different activity from reading to absorb, and it retains far better.

Feeds into: everything. Module 01 next.

01 — Statistics & Probability

Why it matters

Every model output is a claim about a population you can't see, made from a sample you can. Statistics is what tells you how much to believe the claim. You do not need to derive anything here — you need to know which measurement answers which question, and where each one lies to you.

Descriptive statistics

Describing the data you actually have.

Measures of centre

Mean income and median income differ by a lot, and which one a report uses tells you what it wants you to conclude. If mean ≫ median, you have a long right tail.

Measures of spread

• Range — max − min. Determined entirely by your two most extreme points, so it tells you almost nothing about the other n−2.
• Interquartile range (IQR) — split the sorted data into quarters; IQR = Q3 − Q1. The middle 50%. This is the robust spread measure, and the basis of the standard outlier rule (below Q1 − 1.5·IQR or above Q3 + 1.5·IQR).
• Variance — mean squared deviation from the mean. Squaring makes it penalise large deviations disproportionately, and leaves it in squared units.
• Standard deviation — the square root of the variance, which puts it back in the original units. That is the whole reason it exists. (Frequently garbled — see module 99.)

Entropy and information gain

Both come from information theory; you meet them when a decision tree picks its splits.

Entropy measures impurity — how mixed the labels are in a set.

• All one class → entropy 0. Perfectly certain.
• Two classes at 50/50 → entropy 1 (maximum for binary). Perfectly uncertain.

Information gain = entropy before a split − weighted entropy after it. It answers "how much did this feature reduce my uncertainty?" A decision tree computes it for every candidate split and takes the largest. That is the entire tree-building algorithm; module 02 has the rest.

Probability

• Probability — desired outcomes ÷ total outcomes, for equally likely outcomes. Between 0 and 1.
• Random experiment — one whose outcome you can't predict with certainty.
• Sample space — every possible outcome.
• Event — one or more outcomes.

Three kinds you must keep apart:

Bayes' theorem

P(A | B) = P(B | A) · P(A) / P(B)

It relates a conditional probability to its inverse — which matters because one direction is usually easy to measure and the other is what you want.

Medical testing is the standard illustration. P(positive test | disease) is what the lab reports and is typically high. P(disease | positive test) is what the patient cares about, and when the disease is rare it can be low even for an accurate test — because P(A), the prior, drags it down. Same numbers, opposite conclusions, and the intuition that they're interchangeable is what Bayes exists to break.

This is the whole basis of Naive Bayes classifiers in module 02.

Distributions

A probability density function (PDF) describes how probability is spread over a continuous variable. Two things surprise people:

1. The PDF value at a point is not a probability. It's a density. For a continuous variable, P(X = exactly 5.0) = 0.
2. Probability is area under the curve over a range. Total area = 1.

For a normal (Gaussian) distribution, the mean sets where the curve is centred and the standard deviation sets how spread out it is. A larger SD gives a wider, flatter curve — the peak drops because the total area is fixed at 1. (The source notes say SD "controls the height", which inverts the relationship — module 99 C-03.)

Central limit theorem — take repeated samples from any population, compute each sample's mean, and those means will be normally distributed around the population mean, for large enough samples. This is why normal-distribution methods work on data that isn't itself normal, and it's the licence for most of inferential statistics.

Inferential statistics

Reasoning from sample to population.

• Point estimate — a single value from the sample used to estimate the population parameter. Almost certainly slightly wrong.
• Interval estimate — a range instead, which is honest about that.
• Confidence interval — the range, between a lower and upper bound.
• Confidence level — the percentage (usually 95%). Its meaning is narrower than it sounds: over many repeated samples, 95% of the intervals so constructed would contain the true value. It is not "95% chance the true value is in this particular interval."
• Margin of error — how far the interval extends either side of the estimate.

Hypothesis testing

The procedure for "is this effect real or is it noise?"

• Null hypothesis (H₀) — nothing is going on. No effect, no difference.
• Alternative hypothesis (H₁) — the thing you're trying to show.
• Significance level (α) — your tolerance for rejecting H₀ when it was actually true. Conventionally 0.05. This is a Type I error — and module 03 shows it is the same object as a false positive.

Which test:

What breaks

• Mean on skewed data. Salaries, response times, and file sizes are all right-skewed. Reporting the mean of any of them overstates the typical case. Report the median, or both.
• Reading the confidence level as a probability about your interval. Extremely common, including in published work. See the definition above.
• Statistical significance treated as importance. With a large enough sample, a trivially small effect becomes significant. Significance says "probably not zero", not "big enough to care about."
• Standard deviation on a non-normal distribution. "Mean ± 2 SD covers ~95%" is a normal-distribution fact. On a bimodal or heavy-tailed distribution it can cover a region containing no data at all.

Feeds into: module 02 (entropy → trees, Bayes → Naive Bayes), module 03 (Type I/II errors → confusion matrix), module 05 (loss functions are built on these measures of deviation).

02 — Classical ML

Why it matters

Deep learning gets the attention; gradient-boosted trees win most problems that arrive as a table of rows and columns. Knowing this family well is what separates someone who reaches for a neural network because it's the only tool they have from someone who reaches for one because it's right.

Supervised: regression

Linear regression

Fit a straight line (or hyperplane) through the data. Training finds the slope and intercept that minimise squared error. Output is a number.

Use it when you want a continuous prediction and a model whose coefficients you can read and defend. It is also the baseline you should beat before claiming anything fancier is worth it.

Logistic regression

Despite the name, this is a classification model. It's linear regression's output pushed through a sigmoid:

sigmoid(x) = 1 / (1 + e^(-x))

The sigmoid squashes any real number into (0, 1), which you read as a probability, then threshold — usually at 0.5 — into a class.

The underlying relationship is linear in the log-odds: with odds = p/(1−p), log(odds) = mx + c. That is what "logistic" refers to, and why a linear model can produce a curved probability output.

Excellent default for binary classification. Fast, calibrated, interpretable.

Supervised: classification

Decision tree

A sequence of if/then splits. At each node the algorithm picks the feature and threshold with the highest information gain (module 01), splitting the data into more homogeneous groups. Handles numeric and categorical features, needs no scaling, and you can print it and show it to a non-technical stakeholder.

Its defining flaw: an unconstrained tree keeps splitting until every leaf is pure, which means memorising the training set. A single deep tree overfits nearly always.

Random forest

Build many decision trees, each on a random subset of rows and features, then vote (classification) or average (regression).

This directly fixes the tree's overfitting. Individual trees overfit in different directions because they saw different data; averaging cancels the noise and keeps the signal. This is bagging, below. Strong general-purpose default: good accuracy with little tuning.

Naive Bayes

Applies Bayes' theorem, assuming every feature is independent given the class. That assumption is essentially always false — "naive" is doing real work in the name — and the classifier performs well anyway, because it only needs the ranking of class probabilities to be right, not the values.

Used for spam filtering, news categorisation, and as the fast baseline for any text classification task. Trains in one pass.

K-nearest neighbours (KNN)

No training. To classify a point, find its k closest neighbours and take the majority vote.

Distance metrics:

Cheap to set up, expensive at prediction time — every query compares against the entire training set. Requires feature scaling, since a feature measured in thousands dominates the distance regardless of relevance. Powers a lot of recommendation systems and "find similar documents" features, and it's the same idea as the vector search in module 08.

Support vector machine (SVM)

Find the hyperplane separating the classes with the widest possible margin. Each record is a point in n-dimensional space (n = number of features); the points that define the margin are the support vectors, and only they matter — you could delete the rest of the training data.

For data not separable by a straight boundary, the kernel trick maps it into a higher-dimensional space where it is. Strong on small, high-dimensional datasets — text, gene expression. Slow on large ones.

Unsupervised

K-means clustering

Pick k centroids, assign each point to its nearest one, move each centroid to the mean of its assigned points, repeat until stable.

You must choose k in advance, and k-means will happily return 5 clusters from data with no cluster structure at all. The elbow method and silhouette score help pick k; neither will tell you the structure is absent.

Dimensionality reduction

High dimensionality causes real problems: points spread thin so distance-based methods degrade (the "curse of dimensionality"), models overfit the extra noise, and everything gets slower.

The t-SNE restriction matters and gets ignored. Distances between t-SNE clusters are not meaningful, cluster sizes are not meaningful, and re-running with a different seed gives a different picture. It is a way to look at data, not a way to transform it.

Association rules

Find items that co-occur — the Apriori algorithm and market-basket analysis. Retail layout, cross-sell recommendations.

Ensembles: bagging and boosting

Both combine weak models into a strong one. They attack opposite problems.

Bagging: train many models on different random subsets, average them. Errors that come from noise are uncorrelated between models and cancel out.

Boosting: train models in sequence, each one weighting the examples the previous ones got wrong. AdaBoost does this literally — misclassified samples get more weight next round.

Boosting usually wins on accuracy and is easier to overfit; bagging is more forgiving. The current boosting libraries:

For a new tabular problem: run logistic regression or a random forest as a baseline, then gradient boosting. If deep learning beats gradient boosting on tabular data, check for a bug before celebrating.

Choosing

Minimum workflow

Every project in module 08 is this shape with more steps:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = LogisticRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))

random_state=42 makes the split reproducible. Set it, or your accuracy changes every run and you cannot tell improvement from luck.

What breaks

• Forgetting to scale features for KNN and SVM. Both are distance-based; an unscaled feature in thousands drowns everything else. Trees don't care.
• An unconstrained decision tree. It will reach 100% training accuracy and fail on new data. Limit depth, or use a forest.
• Choosing k in k-means by eye and never revisiting it. The clusters look convincing at any k. That's the trap.
• t-SNE as preprocessing. Covered above. Visualisation only.
• Treating accuracy as the score. On imbalanced data it is actively misleading — module 03 is entirely about this.

Feeds into: module 03 (evaluating these models), module 05 (neural networks as the answer to what these can't do), module 08 (Naive Bayes and KNN reappear as baselines).

03 — Evaluation

Why it matters

This is the module that separates working models from demos. Choosing the wrong metric is not a reporting error — it produces a model optimised for the wrong thing, and the failure is invisible until production. A fraud detector with 99.9% accuracy that catches no fraud is a normal outcome, not a pathological one.

Overfitting and underfitting

Overfitting — high accuracy on training data, poor accuracy on new data. The model memorised noise instead of learning the pattern.

Signs: training accuracy far above test accuracy; accuracy that drops sharply on fresh data.

Fixes: more training data; a simpler model; regularisation; dropout (module 05); ensembles; and validating on data the model has never touched.

Underfitting — poor accuracy everywhere. The model is too simple for the structure in the data, or hasn't trained long enough.

Fixes: a more expressive model, better features, longer training.

The two are ends of one axis, and every model lives somewhere on it. You are looking for the point where test error is lowest, which is before training error bottoms out.

Train / validation / test

Three splits, three jobs:

• Train — the model fits on this.
• Validation — you tune on this: hyperparameters, model choice, when to stop.
• Test — touched once, at the end, to estimate real-world performance.

The test set is spent the moment you make a decision based on it. Tune against test and your reported number is optimistic by an unknown margin, because you have leaked test information into the model through your own choices.

Where data is scarce, k-fold cross-validation rotates the validation set across k splits and averages, so every record trains and validates. Costs k trainings.

Regression metrics

• MSE (mean squared error) — average squared difference between predicted and actual. Squaring punishes large errors heavily. Sensitive to outliers.
• MAE (mean absolute error) — average absolute difference. More robust.
• R² (coefficient of determination) — the fraction of variance in the target the model explains. 1.0 is perfect, 0 means no better than predicting the mean, and it can be negative, which means worse than predicting the mean.
• Adjusted R² — R² penalised for the number of features. Plain R² never decreases when you add a feature, even a column of random numbers, so it cannot be used to compare models with different feature counts. Adjusted R² can.

Classification metrics

The confusion matrix

Everything else is derived from this table.

Read the terms as two words: the second says what was predicted, the first says whether that prediction was right. "False positive" = predicted positive, wrongly.

The derived metrics

Accuracy — of everything, what fraction did I get right?

(TP + TN) / (TP + TN + FP + FN)

Precision — of everything I flagged, what fraction was real?

TP / (TP + FP)

Recall (sensitivity, true positive rate) — of everything that was real, what fraction did I catch?

TP / (TP + FN)

F1 — harmonic mean of precision and recall.

2 · (Precision · Recall) / (Precision + Recall)

Harmonic, not arithmetic, so it stays low unless both are decent. Precision 1.0 with recall 0.0 gives F1 = 0, where a plain average would give a respectable 0.5.

Why accuracy lies

Take 10,000 transactions, 10 fraudulent. Predict "not fraud" every time:

• Accuracy: 9,990/10,000 = 99.9%
• Recall: 0/10 = 0%

The model is worthless and its headline number is excellent. Whenever classes are imbalanced — fraud, disease, defects, spam, anything rare — accuracy is the wrong metric. Use precision, recall, F1, or precision-recall AUC.

Choosing between precision and recall

They trade off. Moving the decision threshold buys one with the other, and which you want is a question about consequences, not statistics:

• Recall matters more when missing a positive is expensive. Cancer screening: a false alarm costs a follow-up test, a miss costs a life.
• Precision matters more when a false alarm is expensive. Spam filtering: a little spam in the inbox is an annoyance, a job offer in the spam folder is not.

Ask what happens to a real person on each side of the error. That answers it.

ROC and AUC

The ROC curve plots true positive rate (recall) against false positive rate as you sweep the classification threshold across its whole range.

TPR = TP / (TP + FN)          FPR = FP / (FP + TN) = 1 − specificity

Note the x-axis: false positive rate, which is 1 − specificity, not specificity. These are routinely conflated — see module 99. Getting it backwards inverts the curve.

AUC is the area under that curve. 1.0 is perfect, 0.5 is a coin flip. It summarises performance across all thresholds, so it's useful when you haven't picked a threshold yet.

On heavily imbalanced data, prefer the precision-recall curve. ROC-AUC stays flatteringly high there, because the large TN count keeps FPR small no matter how bad precision is.

Type I and Type II errors

The statistics vocabulary from module 01, same objects:

The mnemonic is crude and it does stick.

Multi-class

Precision, recall and F1 are defined per class. To get one number:

• Macro average — unweighted mean across classes. Every class counts equally, so rare classes are fully visible. Use when the rare classes matter.
• Micro average — pool all TP/FP/FN, then compute. Dominated by frequent classes. Equals accuracy in single-label classification.
• Weighted average — mean weighted by class frequency.

classification_report in scikit-learn prints all of it, per class, with support. Use it instead of accuracy_score as a habit:

from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_true, y_pred, target_names=class_names))
print(confusion_matrix(y_true, y_pred))

The confusion matrix tells you which classes get confused with each other, which no summary metric can. Two classes bleeding into each other usually means they aren't really distinct, or the labels are inconsistent.

What breaks

• Reporting accuracy on imbalanced data. The single most common evaluation error. See the fraud example.
• Tuning against the test set. Every decision you make while looking at test results leaks into the model. Your final number becomes fiction.
• Fitting the vectoriser or scaler before splitting. fit_transform on the full dataset lets test-set statistics into training — data leakage. Fit on train, transform test. This is subtle and it happens constantly, including in the module 08 project code.
• A single metric with no confusion matrix. F1 = 0.82 tells you nothing about which errors you're making, and the errors are the actionable part.
• Comparing R² across models with different feature counts. Use adjusted R².

Feeds into: module 05 (loss functions optimise these), module 08 (every project ends in a classification report), and module 12 — none of these metrics work on generated text, which module 12 exists to solve.

04 — Text → Vectors

Why it matters

Models do arithmetic. Text is not arithmetic. Every NLP system begins by turning words into numbers, and that choice caps everything downstream — no classifier recovers meaning that the representation threw away. This module is the ladder from counting words to representing meaning.

The ladder

Five rungs, each fixing the previous one's main flaw.

1. One-hot encoding

Each word is a vector of zeros with a single 1 at that word's index. Vocabulary of 50,000 words → 50,000-dimensional vectors.

Every pair of distinct words is exactly equidistant. "cat" is as close to "dog" as it is to "bureaucracy". No similarity information exists at all, by construction.

2. Bag of Words (BoW)

Count how many times each vocabulary word appears in the document. The document becomes one vector of counts.

Simple, interpretable, works surprisingly well on small datasets. Discards word order entirely — "dog bites man" and "man bites dog" are identical vectors — and produces very sparse, very wide vectors.

3. TF-IDF

BoW weighted so that words common across all documents count for less.

• TF (term frequency) — how often the word appears in this document.
• IDF (inverse document frequency) — how rare the word is across the corpus.

Multiply them. A word that's frequent here and rare elsewhere scores high, and is probably what this document is about. "the" appears everywhere, so its IDF collapses it to near zero — you get stopword handling for free.

Still sparse, still no word order, still no semantics. But it is a genuinely strong baseline: TF-IDF into logistic regression solves more text classification problems than it gets credit for, and it trains in seconds.

4. Word embeddings (dense vectors)

Learn a dense, low-dimensional vector per word from a large corpus, so that words used in similar contexts land near each other. 100–300 dimensions typically. Word2Vec, GloVe, FastText.

This is the jump. Similarity becomes geometric — "king" is near "queen", and cosine similarity between vectors is a usable measure of relatedness.

Two limits:

• One vector per word, forever. "bank" gets a single vector averaging the river and the financial senses. Neither is right.
• Out-of-vocabulary words have no vector. (FastText patches this with subword pieces.)

5. Contextual embeddings

Generate the vector at read time, from the sentence the word appears in. BERT, GPT and everything after. 768 or 1024 dimensions typically.

"bank" in "river bank" and "bank account" now get different vectors. This is what transformers buy you, and module 07 is how they do it.

Cost: you must run a large model for every piece of text, at inference as well as training. Orders of magnitude more expensive than a TF-IDF lookup.

Summary

Cleaning text first

The classical methods need preprocessing; the pattern from the project docs:

import re
import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

nltk.download('stopwords')
nltk.download('wordnet')

stop_words = set(stopwords.words('english'))
lemmatizer = WordNetLemmatizer()

def clean_text(text):
    text = text.lower()
    text = re.sub(r'[^a-z\s]', '', text)          # strip punctuation and digits
    tokens = text.split()
    tokens = [lemmatizer.lemmatize(t) for t in tokens if t not in stop_words]
    return ' '.join(tokens)

What each step is for:

• Lowercase — so "Email" and "email" aren't separate features.
• Strip punctuation/digits — noise for count-based methods.
• Remove stopwords — "the", "and", "is" carry no topic signal.
• Lemmatise — "running" → "run", collapsing forms into one feature. (Stemming is the cruder, faster alternative: it chops suffixes and can produce non-words.)

This pipeline is wrong for transformers. BERT and friends were pre-trained on natural text, with their own tokenizers, and depend on the punctuation, casing and stopwords you just deleted. Aggressive cleaning before a transformer removes signal the model was built to use. Clean for TF-IDF; pass raw text to BERT.

The project docs apply this cleaner and then feed BERT — see module 99.

Making it concrete

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

corpus = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "quantum entanglement in condensed matter",
]

bow = CountVectorizer()
X_bow = bow.fit_transform(corpus)
print(bow.get_feature_names_out())
print(X_bow.toarray())

tfidf = TfidfVectorizer()
X_tfidf = tfidf.fit_transform(corpus)
print(X_tfidf.toarray().round(2))

Run both. In the BoW output "the" has the highest count in the first two documents; in the TF-IDF output it is heavily discounted while "quantum" and "entanglement" dominate the third. That contrast is the whole idea of IDF, and seeing it in the numbers beats reading the formula.

Documents, not just words

Doc2Vec extends word embeddings to whole documents, learning a vector per document.

The pragmatic modern approach is sentence embedding models — sentence-transformers with all-MiniLM-L6-v2 and similar. One call, one vector per sentence or paragraph, tuned so that cosine similarity means semantic similarity. This is what module 08's semantic search project runs on, and it's the standard starting point for retrieval today.

What breaks

• fit_transform on the full dataset before splitting. The vectoriser's vocabulary and IDF weights then encode test-set information. Fit on train, transform test. Data leakage, and it inflates your score.
• Cleaning aggressively before a transformer. Covered above.
• Vocabulary explosion. Unbounded CountVectorizer on a large corpus produces hundreds of thousands of features, most appearing once. Cap it — max_features, min_df.
• Euclidean distance on embeddings. Use cosine similarity. Embedding magnitude varies with things you don't care about; direction is what carries meaning. FAISS's IndexFlatIP with L2-normalised vectors gives you cosine, and that's why module 08 normalises before indexing.
• Assuming embedding similarity means relevance. "I love this product" and "I hate this product" are extremely close in embedding space. For sentiment, that's the opposite of useful.

Feeds into: module 06 (embedding layers), module 07 (contextual embeddings are what attention produces), module 08 (all three projects turn text into vectors first), and module 17 — tokenization sits underneath all of this and is where the ladder actually starts.

05 — Deep Learning

Why it matters

Classical ML needs you to hand it good features. On images, audio and text, nobody knows what the good features are — so deep networks learn them. That is the entire trade: you give up interpretability and data efficiency, and you get automatic feature extraction on data too raw to engineer by hand.

The neuron

Loosely modelled on the biological one: dendrites bring signals in, the cell body processes, the axon passes output on. The artificial version:

1. Each input is multiplied by a weight (initialised randomly).
2. The products are summed, and a bias is added.
3. The sum passes through an activation function.
4. If the result clears the activation's threshold, the neuron fires.

A perceptron is a single such neuron: a single-layer binary linear classifier. It can only separate linearly separable data — famously it cannot learn XOR.

Stack neurons into layers, stack layers, and you have a multilayer perceptron: input layer, one or more hidden layers, output layer, each node connected to every node in the next layer. Networks can have hundreds of hidden layers.

Activation functions

Without one, a network is pointless. Stacking linear layers gives you another linear layer — W₂(W₁x + b₁) + b₂ collapses to Wx + b — so an arbitrarily deep linear network has exactly the expressive power of a single one. The activation function introduces the non-linearity that makes depth mean something.

ReLU is the workhorse: it activates only above zero and is linear above it. It's cheap, and crucially its gradient is exactly 1 for positive inputs, so it doesn't shrink gradients the way sigmoid does. Its own failure mode is "dying ReLU" — a neuron pushed permanently negative outputs zero forever and stops learning. Leaky ReLU exists for this.

Training

Forward pass, loss, backward pass

1. Forward — data flows through the layers to a prediction.
2. Loss — compare prediction to truth. One number saying how wrong.
3. Backward — backpropagation computes, for every weight, the derivative of the loss with respect to that weight (∂E/∂w): how much this weight contributed to the error.
4. Update — nudge each weight against its gradient.

Repeat over the dataset. One full pass over the training data is an epoch.

Gradient descent

The update rule:

x = x − learning_rate · (dy/dx)

You are standing on an error surface and stepping downhill. The learning rate is your step size, and it is the hyperparameter that most often decides whether training works:

• Too high — you overshoot the minimum and bounce, or diverge entirely.
• Too low — training crawls, or parks in a local minimum.

Batches

How much data per update:

Mini-batch is what everyone means by "SGD" in practice. The noise is not purely a cost — it helps escape shallow local minima.

Gradient pathologies

Deep networks fail in two symmetric ways, and both come from the chain rule multiplying many terms together.

Vanishing gradient — gradients shrink toward zero as they propagate back through layers. Early layers stop updating; the network trains only its last few layers.

Fixes: ReLU (gradient of 1, doesn't shrink), residual connections (module 07), better initialisation, LSTM/GRU gating for sequences.

Exploding gradient — gradients grow without bound. Weights leap to huge values, loss becomes NaN, training dies immediately.

Fixes: gradient clipping (cap the norm at a threshold), lower learning rate, truncated backpropagation through time, layer normalisation.

These two lists get swapped constantly — ReLU prescribed for exploding gradients, clipping for vanishing. It is exactly backwards, and worth knowing why: ReLU helps vanishing because its gradient doesn't shrink; clipping helps exploding because it caps growth. Neither addresses the other problem.

Architectures

Feedforward

Information flows one way. Output depends only on the current input — no memory.

Fine for tabular data and fixed-size inputs. It cannot do anything requiring context from earlier in a sequence: predicting the next word in a sentence needs the previous words, and a feedforward net has no way to hold them.

CNN (Convolutional Neural Network)

For images, inspired by the visual cortex.

The problem it solves is concrete. A 224×224 colour image has ~150,000 values. A fully connected layer of 1,000 neurons on that input needs 150 million weights — in one layer. Too slow, too memory-hungry, and it overfits immediately.

A CNN instead slides small filters across the image. Each filter has few weights and is reused at every position, which encodes a true fact about images: an edge is an edge wherever it appears.

Four layer types:

1. Convolution — slide filters, produce feature maps. Early layers learn edges and colours; deeper layers learn textures, then parts, then objects.
2. ReLU — non-linearity.
3. Pooling — downsample (usually max over a 2×2 window). Shrinks the representation and adds tolerance to small shifts.
4. Fully connected — the actual classification, at the end.

This is automatic feature extraction: you never wrote an edge detector, and the network built one because edges were useful.

RNN (Recurrent Neural Network)

For sequences. The output of one step is fed back as an input to the next, so the network carries a hidden state — a memory of what it has seen.

Handles variable-length input, and order matters. Text, time series, speech.

Its weakness is long-range dependency. Information from 50 steps back has been multiplied through 50 transformations and is effectively gone — this is the vanishing gradient problem in its most acute form. If the subject of a sentence is far from its verb, a plain RNN loses the link.

LSTM (Long Short-Term Memory)

An RNN with gates that explicitly control memory. Sigmoid layers (outputting 0–1, "how much of this passes?") and tanh layers (−1 to 1, candidate values) decide what to forget, what to store, and what to output. A cell state runs along the whole chain with only minor linear interactions, so information can travel a long way unchanged.

This is what makes long-term dependency learnable. LSTMs powered NLP for years and still appear where sequences are long and data is limited — module 08's first project uses one.

GRU is a simplified LSTM with fewer gates: faster, fewer parameters, usually comparable accuracy. It is named here and not explained — module 10,.

Autoencoders

Train a network to reconstruct its own input through a narrow bottleneck. The target output is the input.

Because the bottleneck is smaller than the input, the network must learn a compressed representation that keeps what matters. Uses: dimensionality reduction, image compression, denoising, anomaly detection (train on normal data; anything that reconstructs badly is anomalous).

Unsupervised — no labels needed, since the data is its own label. Compared to PCA: PCA is restricted to linear projections, an autoencoder has non-linear activations and multiple layers, so it can capture structure PCA cannot.

Where this runs out

RNNs and LSTMs process sequences one step at a time. Two consequences:

1. No parallelism during training. Step 51 needs step 50's output. GPUs sit idle.
2. Context still degrades over distance, even with gating.

Module 07 is what happens when someone removes the recurrence entirely.

What breaks

• Learning rate wrong by an order of magnitude. Loss goes to NaN (too high) or flatlines (too low). Check this before anything else when training fails.
• Forgetting to zero gradients. In PyTorch, gradients accumulate. Omit optimizer.zero_grad() and every step is polluted by every previous step. Module 06 covers the loop.
• Sigmoid in deep hidden layers. Saturates at both ends, gradient goes to near zero, vanishing gradient. Use ReLU in hidden layers.
• Deep learning on a small tabular dataset. It will lose to gradient boosting and take twenty times longer. Neural networks need data volume to pay for their flexibility.
• No non-linearity between linear layers. The network silently collapses to a linear model. It trains, it just can't learn anything a linear model couldn't.

Feeds into: module 06 (building these in PyTorch/TensorFlow), module 07 (transformers as the successor to RNNs), module 08 (LSTM and BERT classifiers).

06 — Frameworks: PyTorch and TensorFlow

Why it matters

Both frameworks are the same eight ideas with different names. Learn the ideas and you can read either; learn one framework's API and you've learned an API. This module is organised around the shared concepts, with both dialects shown.

The eight components

1. Tensors

A multi-dimensional array, like a NumPy array with GPU support and gradient tracking. Rank = number of dimensions (scalar 0, vector 1, matrix 2). Shape = size along each. Dtype = float32, int64, etc.

# PyTorch
import torch
x = torch.tensor([1, 2, 3])     # 1-D
y = torch.ones((2, 3))
z = torch.randn((4, 4))

# TensorFlow
import tensorflow as tf
a = tf.constant([[1, 2], [3, 4]])
print(a.shape)                   # (2, 2)
w = tf.Variable(tf.random.normal([2, 2]), name='weight')

tf.Variable is the mutable one — weights and biases, state that survives across training steps. tf.constant doesn't change.

2. Automatic differentiation

The thing that makes any of this feasible. You write the forward pass; the framework computes every derivative for the backward pass.

PyTorch records operations on tensors with requires_grad=True into a computational graph, then .backward() walks it:

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2
y.backward()
print(x.grad)          # 4.0  — dy/dx = 2x at x=2

Gradients accumulate in .grad. This is deliberate (it lets you sum gradients across sub-batches) and it is the source of the single most common PyTorch bug — see What breaks.

TensorFlow uses an explicit context manager:

with tf.GradientTape() as tape:
    predictions = model(x)
    loss = loss_fn(y, predictions)
gradients = tape.gradient(loss, model.trainable_variables)
optimizer.apply_gradients(zip(gradients, model.trainable_variables))

3. Layers and models

PyTorch — subclass nn.Module, define layers in __init__, define the forward pass in forward():

import torch.nn as nn

class BagOfWordsClassifier(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        return self.fc2(self.relu(self.fc1(x)))

model = BagOfWordsClassifier(input_dim=500, hidden_dim=16, output_dim=3)

Note fc2 outputs raw scores (logits), not probabilities — no softmax. That is correct, because nn.CrossEntropyLoss applies softmax internally. Adding your own softmax here applies it twice and degrades training.

Common layers: nn.Linear (fully connected), nn.Conv2d, nn.LSTM, nn.Dropout, nn.ReLU, nn.Sigmoid.

TensorFlow/Keras — stack layers:

from tensorflow.keras import layers, models

model = models.Sequential([
    layers.Dense(64, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax'),
])

Or subclass tf.keras.Model with a call() method for anything Sequential can't express — branches, multiple inputs, custom logic.

4. Optimisers

Start with Adam at lr=0.001. Move on only if you have a reason.

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

5. Loss functions

Classification

• CrossEntropyLoss — multi-class. Takes raw logits and integer class labels. Softmax is built in.
• BCELoss — binary. Takes probabilities, so apply sigmoid first. (Or use BCEWithLogitsLoss, which takes logits and is numerically more stable.)
• Hinge loss — SVM-style, maximises margin.

Regression

• MSELoss — squared error. Sensitive to outliers.
• L1Loss (MAE) — absolute error. Robust to outliers.
• Huber — quadratic near zero, linear far out. A compromise.

Similarity / ranking

• Contrastive — pull similar pairs together, push dissimilar apart.
• Triplet — anchor, positive, negative. Learns relative similarity.
• Cosine embedding loss — on cosine similarity.

These matter for embedding models — it's how the sentence-transformers in module 08 were trained.

Distributions and specialised

• KL divergence — difference between two probability distributions.
• Adversarial loss — GAN generator/discriminator.
• Dice / Jaccard — segmentation overlap.
• Focal loss — cross-entropy that down-weights easy examples so training focuses on hard ones. The standard tool for severe class imbalance.

In Keras, note SparseCategoricalCrossentropy (integer labels) versus CategoricalCrossentropy (one-hot labels). Mismatching these is a routine and confusing shape error.

6. Data pipelines

PyTorch — a Dataset says how to get one item; a DataLoader batches, shuffles and parallelises:

from torch.utils.data import Dataset, DataLoader

class TextDataset(Dataset):
    def __init__(self, X, y):
        self.X = torch.tensor(X)
        self.y = torch.tensor(y)

    def __len__(self):
        return len(self.X)

    def __getitem__(self, idx):
        return self.X[idx], self.y[idx]

loader = DataLoader(TextDataset(X, y), batch_size=32, shuffle=True)

Three methods, always the same three.

TensorFlow — tf.data:

dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
dataset = dataset.shuffle(1000).batch(32).prefetch(tf.data.AUTOTUNE)

7. The training loop

PyTorch — you write it, every time:

for epoch in range(num_epochs):
    model.train()
    for xb, yb in train_loader:
        optimizer.zero_grad()        # 1. clear old gradients
        logits = model(xb)           # 2. forward
        loss = criterion(logits, yb) # 3. loss
        loss.backward()              # 4. backward
        optimizer.step()             # 5. update

    model.eval()
    with torch.no_grad():
        for xb, yb in val_loader:
            ...

Five lines in a fixed order. model.train() and model.eval() switch the mode of dropout and batch-norm layers, which behave differently in training and inference. torch.no_grad() skips graph construction during evaluation — faster and lower memory.

TensorFlow — the high-level API writes it for you:

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.fit(x_train, y_train, epochs=10, validation_data=(x_test, y_test))

That is the real difference between the two frameworks in daily use: Keras hides the loop, PyTorch hands it to you. Use GradientTape when you need TensorFlow to hand it back.

8. Devices, saving, reproducibility

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
data = data.to(device)

torch.save(model.state_dict(), 'model.pth')
model.load_state_dict(torch.load('model.pth'))
model.eval()

torch.manual_seed(42)

Model and data must be on the same device, or you get a device-mismatch error. Saving state_dict (weights only) rather than the whole object is the recommended practice — it doesn't pickle your class definitions along with it.

TensorFlow:

model.save("path_to_model")
model = tf.keras.models.load_model("path_to_model")

Graphs: dynamic vs static

PyTorch builds the graph as the code runs, so control flow is just Python and you can put a breakpoint anywhere. TF 1.x required defining the whole graph up front and then running it, which was fast and miserable to debug. TF 2.x adopted eager execution by default and offers @tf.function to compile a function into a graph when you want the speed back.

Ecosystem

PyTorch dominates research — easier to prototype, better suited to unusual architectures. Deployment via TorchScript, ONNX, PyTorch Mobile, TorchServe. Libraries: torchvision, torchaudio, torchtext, PyTorch Lightning, and HuggingFace Transformers has first-class PyTorch support.

TensorFlow has the broader production ecosystem: TFX for pipelines, TensorBoard for visualisation, TF Lite for mobile/embedded, TF Serving for production, TF.js for browsers.

The deployment gap that once justified choosing TensorFlow has largely closed. Pick either; most published models you'll want to use are PyTorch.

TensorBoard is worth using regardless — add it as a callback in model.fit(), launch with tensorboard --logdir=logs, and watch loss curves live instead of squinting at printed numbers. It works with PyTorch too.

What breaks

• Missing optimizer.zero_grad(). PyTorch accumulates gradients, so without it every step includes every previous step's gradient. Training degrades in a way that looks like a bad learning rate. The most common PyTorch bug there is.
• Softmax before CrossEntropyLoss. Applied twice. Output the raw logits.
• Forgetting model.eval() at inference. Dropout stays active, so predictions are randomly degraded and non-reproducible.
• Model on GPU, data on CPU. Loud error, easy fix; move both.
• SparseCategoricalCrossentropy with one-hot labels (or the reverse) in Keras. Shape error whose message doesn't point at the cause.
• Copying API calls from old tutorials. Both frameworks have moved substantially. Embedding(input_length=...) and input_dim positional patterns from Keras 2 no longer apply in Keras 3 — see module 99. Check the current docs rather than trusting a code block from a blog post.

Feeds into: module 07 (transformers are built from these pieces), module 08 (all three projects use one or both frameworks).

07 — Transformers

Why it matters

Every current language model is a transformer. The architecture replaced RNNs in about three years, and the reason is not that it understands language better in principle — it's that it can be trained in parallel, so it can be trained on enough data to look like it does.

This is the densest module in the tutorial. Read it slowly.

What was wrong with RNNs

Three problems:

1. Sequential processing. Step 51 needs step 50's output, so training can't parallelise. Slow, forever.
2. Long-range context decays. If the first sentence of an essay bears on a paragraph three pages later, an RNN has lost it.
3. One embedding per word. "Apple" gets a single vector whether the sentence is about juice or phones. Word embeddings are fixed before the sentence is read.

Transformers fix all three with one mechanism.

Self-attention

The core idea: every word looks at every other word in the sentence and builds a new embedding for itself from what it finds.

"I love apple juice" and "I have an apple phone" contain the same token "apple". After self-attention they don't. In the first, "apple" has attended to "juice" and shifted toward the fruit sense; in the second it has attended to "phone" and shifted toward the company. The output is a contextual embedding — module 04's fifth rung, and this is the machinery that produces it.

A nice illustration of what this buys beyond disambiguation: asked for "leafy cloud", a system with self-attention doesn't produce a leaf and a cloud side by side. It can attach the leafy property to the cloud and produce something that never existed. Composition, not lookup.

Query, Key, Value

The naive version — just compare every word embedding to every other with a dot product — has two fatal defects:

1. No trainable parameters. Nothing about the comparison can be learned for the task at hand.
2. Symmetry. If "apple" attracts "phone", "phone" attracts "apple" equally hard. But you wanted to move "apple" toward the tech sense, not drag "phone" toward fruit. Relationships in language are directional; a raw dot product isn't.

The fix is three learned weight matrices, producing three vectors per token:

Queries are matched against keys to produce attention weights; those weights are applied to values. Because Q and K come from different learned matrices, the match is asymmetric — solving defect 2 — and because the matrices are learned, attention becomes task-specific — solving defect 1.

The computation

1. Project each token embedding into Q, K, V by linear transformation.
2. Dot-product every query against every key → a raw score per pair.
3. Scale by dividing by √d_k (below).
4. Softmax the scores into probabilities that sum to 1 — these are the attention weights.
5. Multiply each value vector by its weight and sum → the new embedding.

Linear transformation

A' = A·W + b. Beyond reshaping, this does two jobs: it reduces dimensionality (512×1 through a 64×512 matrix gives 64×1), and the learned weights amplify the features that matter for the task while suppressing the rest.

Why divide by √d_k

The variance of the dot product grows with the dimension of the vectors. In high dimensions, raw scores get large, and large scores fed into softmax produce a near one-hot distribution — one weight at ~1.0, the rest at ~0.

That's a training disaster: softmax gradients vanish when the distribution is saturated. Dividing by √d_k keeps the variance in a workable range. It does not reduce dimensionality; it reduces magnitude. This is scaled dot-product attention, and the scaling is the entire content of the word "scaled".

Multi-head attention

One attention operation captures one kind of relationship. Sentences have several at once.

"She saw the man with the telescope." Who has the telescope? Both readings are grammatical, and a single attention head must commit to one. Run several heads in parallel, each with its own Q/K/V matrices, and different heads can track different relationships — syntactic role, spatial, temporal, coreference. One head can hold "man with telescope" while another holds "she saw, using telescope".

Mechanically: run h attention operations in parallel, concatenate the outputs, pass through one more linear transformation to mix them and return to the expected dimension.

A neater example: "The keys on the table belong to..." — resolving what belongs to what requires tracking possession and location simultaneously. One head each.

Positional encoding

Attention processes all tokens simultaneously. That's the whole speed advantage, and it means the model has no idea what order the words are in. "Dog bites man" and "man bites dog" are identical bags of tokens to it.

So position must be injected into the embeddings. The constraints, each ruling out an obvious approach:

• Not discrete integers (1, 2, 3…) — unbounded, and they'd swamp the embedding values with skew.
• Bounded, so long sequences don't blow up.
• Continuous, so nearby positions have similar encodings.
• Unique per position, so no two positions collide.
• Must not corrupt the semantic embedding they're added to.

The solution is sinusoidal: a 512-dimensional vector built from 256 sine/cosine pairs at geometrically decreasing frequencies. Any single sine wave repeats — two different positions can share a value — but the combination across 256 different frequencies is unique for every position, exactly like a binary counter where each bit flips at a different rate. High-frequency components distinguish nearby positions; low-frequency ones distinguish distant regions of a long document.

These vectors are added to the token embeddings before the first encoder block. Only there — not repeated at each layer.

Layer normalisation and residual connections

Two supporting mechanisms applied after every sub-layer. Together they're the "Add & Norm" box in the architecture diagram.

Residual connection — add a sub-layer's input to its output: output = sublayer(x) + x. This works because input and output have the same dimension, which is why the architecture keeps dimensions constant throughout. The benefit is gradient flow: gradients reach early layers through the addition even when the sub-layer's own gradients are small. This is what makes a 6-block (or 96-block) stack trainable at all — it's the vanishing-gradient fix from module 05, applied structurally.

Layer normalisation — normalise activations to a consistent scale. Same motivation as feature scaling in classical ML: if one feature is "number of rooms" (0–10) and another is "price" (0–1,000,000), the large one dominates purely by magnitude. Normalising keeps the layers in a stable range and training converges much faster.

The encoder

A block, stacked. The original paper used 6, with separate weights per block.

Each encoder block, in order:

1. Multi-head self-attention — context-aware representations.
2. Add & Norm — residual + layer norm.
3. Feed-forward network — two dense layers, ReLU(A·W + b), expanding 512 → 2048 → 512.
4. Add & Norm — again.

Input and output dimensions are identical, which is what lets blocks stack arbitrarily.

Why the feed-forward expansion? Two reasons. First, non-linearity: without a ReLU between them, consecutive linear layers collapse into a single linear layer (module 05), so attention alone would be a stack of linear operations. Second, capacity: expanding to 2048 gives the network room to compute richer intermediate features before projecting back to 512.

Positional encoding is added once, at the input to the first block only.

The decoder

Generates output. Three differences from the encoder.

Masked self-attention

A decoder generating word 5 must not see words 6 onward — at inference they don't exist, and at training they're the answer. Letting the model see them means it learns to copy rather than predict.

The mask adds negative infinity to the future positions in the attention score matrix. Softmax turns −∞ into exactly 0, so those positions contribute nothing. Elegant: no special-case code, just an additive mask before the softmax.

Cross-attention

Queries come from the decoder, keys and values from the encoder's output. This is how the decoder consults the input — "given what I've generated so far, what in the input is relevant now?"

It is also the natural seam for multi-modal input: the encoder side can carry something other than text.

Training parallel, inference sequential

This asymmetry surprises people.

At inference ("inference time" = generation time), generation is inherently sequential: predict a token, append it, predict the next. No way round it.

At training, the entire correct output is known. So the decoder is fed the ground-truth previous tokens rather than its own predictions — every position can be computed at once, in parallel. Each block's output feeds backpropagation but is not passed forward as the next position's input. This is teacher forcing, and combined with masking it's what makes transformer training fast enough to be worth doing.

Masking is what makes it legitimate. Without the mask, feeding the whole labelled sequence in would let each position read its own answer.

Transfer learning

The property that changed the economics of NLP. Training happens in two phases:

1. Pre-training — a general task (predict the masked/next token) on an enormous unlabelled corpus. Expensive, done once, by someone else.
2. Fine-tuning — adapt to your specific task with a small labelled dataset.

You need far less data and far less compute than training from scratch, because the general language understanding is already there. This is why module 08 can load bert-base-uncased and get a competent email classifier from a few thousand examples.

What breaks

• Forgetting positional encoding. The model trains, loss decreases, and it has learned a bag of words. Word order is invisible to it.
• Missing or wrong causal mask in a decoder. Training loss drops suspiciously fast and generation is incoherent — the model learned to copy the next token, which isn't available at inference.
• Attention is O(n²) in sequence length. Double the context, quadruple the compute and memory. This is the hard constraint behind every context-window limit, and module 21.
• Fine-tuning with too high a learning rate. Catastrophic forgetting — the pre-trained knowledge is overwritten and you're worse off than a fresh model. Fine-tuning rates are typically 10–100× lower than training-from-scratch rates.
• Cleaning text before a transformer tokenizer. Module 04's warning. The model wants raw text.

Feeds into: module 08 (BERT for classification, sentence transformers for search). module 22, 13, 14 — encoder-only vs decoder-only variants, tokenization, and attention efficiency all sit adjacent to this module and are absent from this tutorial so far.

08 — Projects

Why it matters

Three builds, deliberately ordered: supervised deep learning, then retrieval, then the two combined into a served API. Each teaches a different way of getting an answer out of text, and the third teaches the decision most projects make implicitly and never state — when not to train a model.

A note on the code. Library APIs move; several calls below will have drifted by the time you read them, and module 99 explains why that is a permanent condition rather than a defect. Treat every block as a design to implement, not a snippet to paste. Each project also carries a deliberate bug, kept because the bug is the most instructive part.

Project 1 — Email classification with an LSTM

Goal: classify emails into SPAM / Phishing / Marketing / Relevant.

Teaches: the full supervised deep-learning pipeline end to end.

Pipeline

raw email → clean → tokenize → pad → embed → LSTM → dense → softmax → class

Preprocessing

Beyond module 04's standard cleaning, email needs domain-specific work — quoted reply chains and headers are noise that appears in every message:

text = re.sub(r"On.*wrote:", "", text)       # quoted reply headers
text = re.sub(r"From:.*\n", "", text)
text = re.sub(r"Subject:.*\n", "", text)
text = re.sub(r"http\S+|www\S+", "", text)   # URLs

Whether URLs should go is a real judgement call, not an obvious cleanup: for phishing detection, the presence and shape of a URL is one of your strongest signals. Stripping it may delete the thing you're trying to detect. Consider replacing with a token like <URL> instead.

Tokenize and pad

Neural networks need fixed-size input; sentences vary. So: map words to integer IDs, then pad or truncate every sequence to the same length.

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

tokenizer = Tokenizer(num_words=5000, oov_token='<OOV>')
tokenizer.fit_on_texts(train_texts)          # fit on TRAIN only

sequences = tokenizer.texts_to_sequences(train_texts)
padded = pad_sequences(sequences, padding='post', maxlen=50)

oov_token gives unseen words a home instead of dropping them silently. num_words=5000 keeps the 5,000 most frequent.

Fitting the tokenizer on the whole dataset before splitting is data leakage (modules 03 and 18) — fit on train, transform test.

Labels

from sklearn.preprocessing import LabelEncoder

label_encoder = LabelEncoder()
labels = label_encoder.fit_transform(df['label'])     # 0, 1, 2, 3
labels = tf.keras.utils.to_categorical(labels)        # one-hot

One-hot pairs with categorical_crossentropy. Keeping integers and using sparse_categorical_crossentropy is equivalent and saves memory — just don't mix them (module 06).

Model

model = Sequential([
    Embedding(input_dim=5000, output_dim=64),
    LSTM(64),
    Dropout(0.5),
    Dense(32, activation='relu'),
    Dropout(0.5),
    Dense(4, activation='softmax'),
])

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

Layer by layer:

• Embedding — integer IDs to dense 64-d vectors, learned during training. (Older code passes input_length=50; that argument is gone in Keras 3.)
• LSTM — reads the sequence in order, carrying state. Order matters in email: "your account has been compromised, reset your password" is phishing; the same words shuffled are noise.
• Dropout(0.5) — randomly zeroes half the units each training step, forcing redundant representations. Active in training, off at inference — this is what model.eval() / Keras's automatic mode switch handles.
• Dense(4, softmax) — four class probabilities, summing to 1.

Evaluate

y_pred_classes = np.argmax(model.predict(X_test), axis=1)
y_true = np.argmax(y_test, axis=1)

print(classification_report(y_true, y_pred_classes,
                            target_names=label_encoder.classes_))

Per-class, always (module 03). Then plot training vs validation accuracy — the gap between the two curves is your overfitting diagnostic, and with dropout at 0.5 you want to see them roughly tracking.

Upgrades

Bidirectional LSTM — one line, reads the sequence both ways:

from tensorflow.keras.layers import Bidirectional
Bidirectional(LSTM(64))

For classification you have the whole email up front, so there's no reason not to. (For generation there is: you can't read the future.)

BERT — replace the learned-from-scratch embeddings with a pre-trained transformer:

from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
bert_model = TFBertModel.from_pretrained('bert-base-uncased')

input_ids = Input(shape=(128,), dtype=tf.int32)
bert_output = bert_model(input_ids)[1]        # pooled [CLS] output
x = Dense(64, activation='relu')(bert_output)
x = Dropout(0.3)(x)
output = Dense(4, activation='softmax')(x)

This is transfer learning (module 07): BERT already understands English, you supply the four-class head. Two cautions the snippet above omits — pass raw text to BERT's tokenizer, not NLTK-cleaned text (module 17), and BERT needs the attention mask alongside input_ids.

Realistic expectations. A handful of synthetic examples produces a model that has memorised a handful of examples. Real use needs thousands.

Project 2 — Semantic knowledge search

Goal: search documents by meaning, not keyword. "Where is the password reset link?" should find "Hi John, your password has been reset successfully" without sharing search terms.

Teaches: embeddings, vector indexes, similarity search — the retrieval half of every RAG system.

Why it works

Sentence-BERT maps sentences into a space where semantically similar sentences land near each other (module 04). Nearest-neighbour search in that space is semantic search. No training, no labels — just encode and index.

With FAISS

from sentence_transformers import SentenceTransformer
import faiss

model = SentenceTransformer("all-MiniLM-L6-v2")

texts = [
    "How to reset my password?",
    "What are today's meeting notes?",
    "Your invoice for last month.",
    "Team outing has been rescheduled to Friday.",
]

embeddings = model.encode(texts, convert_to_numpy=True).astype("float32")

index = faiss.IndexFlatIP(embeddings.shape[1])
faiss.normalize_L2(embeddings)
index.add(embeddings)

def semantic_search(query, top_k=3):
    q = model.encode([query], convert_to_numpy=True).astype("float32")
    faiss.normalize_L2(q)
    scores, idx = index.search(q, top_k)
    return [(texts[i], float(scores[0][k])) for k, i in enumerate(idx[0])]

Two details that matter:

IndexFlatIP + normalize_L2 = cosine similarity. Inner product on unit vectors is cosine similarity. This is why the normalisation isn't optional — skip it and you're ranking by magnitude, which encodes nothing you want.

Normalise the query too. Easy to forget; it's in the function above for a reason.

IndexFlatIP is exact brute-force search: every query compares against every vector. Correct and fast enough to hundreds of thousands of documents. Beyond that you want an approximate index (IndexIVFFlat, HNSW) which trades a little recall for a lot of speed. module 14.

With Chroma and LangChain

Persistence and metadata, at the cost of a dependency:

from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma
from langchain_text_splitters import CharacterTextSplitter
from langchain_core.documents import Document

docs = [Document(page_content=t, metadata={"source": f"doc_{i}"})
        for i, t in enumerate(texts)]

splitter = CharacterTextSplitter(chunk_size=200, chunk_overlap=20)
split_docs = splitter.split_documents(docs)

embedding_model = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
db = Chroma.from_documents(split_docs, embedding_model,
                           persist_directory="./chroma_knowledge")

results = db.similarity_search("How do I reset my password?", k=3)

LangChain split into provider packages, so older langchain.embeddings / langchain.vectorstores imports no longer apply; .persist() is likewise gone, since persistence is automatic when persist_directory is set. Module 99.

Chunking is the part that decides whether retrieval works. chunk_size=200 with chunk_overlap=20: too large and one chunk covers several topics so its embedding averages them into mush; too small and it loses the context that makes it interpretable. Overlap stops a sentence being severed at a boundary. There is no correct value — it depends on your documents, and tuning it beats almost any other change you can make.

Choosing a store

Start with FAISS or Chroma. Module 14 covers the choice in depth.

Project 3 — RAG classification API

Goal: classify ticket activity text into delay_start / delay_end / no_delay, served over HTTP.

Teaches: RAG, and the decision of when to retrieve instead of train.

Why RAG here

The premise: instead of training a classifier, retrieve similar past examples and ask an LLM to classify by analogy. No training, no labelled dataset of any size, and new categories need only new examples in the index.

Read that claim critically. For a stable 3-class problem with a few thousand labelled rows, TF-IDF plus logistic regression will be more accurate, ~10,000× cheaper per call, faster, and deterministic. RAG earns its place when labels are scarce or expensive, when categories change often, or when you need an explanation alongside the label.

Making that comparison — for your own problem, out loud — is the most valuable thing in this project. Module 19 is the full decision.

The pipeline

text → clean → embed → FAISS index
                          ↓
new text → embed → retrieve top-k similar → build prompt → LLM → label

Building the index

from langchain_openai import OpenAIEmbeddings
import faiss, numpy as np

embedding_model = OpenAIEmbeddings()
embeddings = embedding_model.embed_documents(cleaned_texts)

index = faiss.IndexFlatL2(len(embeddings[0]))
index.add(np.array(embeddings).astype('float32'))

IndexFlatL2 is Euclidean distance. Fine, but for embeddings prefer normalised inner product as in project 2 — module 04 explains why direction beats magnitude.

The bug worth studying

A natural-looking implementation builds a RetrievalQA chain and calls it like this:

# This does not do what it appears to
prompt = f"""Classify this activity as one of ['delay_start', 'delay_end', 'no_delay'].
Activity: "{activity_text}"
Answer with only the label."""
return qa_chain.run(prompt).strip()

The retrieval query is the entire instruction block, not the activity text — so the nearest-neighbour search runs against a string dominated by boilerplate that is identical for every call. Worse, the retrieved examples are inserted as undifferentiated context with no labels attached, so even when retrieval works, the LLM is shown similar sentences without being told what any of them were classified as. The retrieved context can't do the job the design assigns it.

The fix — retrieve on the text alone, and put the retrieved examples in the prompt with their labels:

def classify_activity(activity_text, k=3):
    neighbours = vectorstore.similarity_search(activity_text, k=k)
    examples = "\n".join(
        f'- "{d.page_content}" → {d.metadata["label"]}' for d in neighbours
    )
    prompt = f"""Classify the activity into exactly one of:
delay_start, delay_end, no_delay.

Similar past activities and their labels:
{examples}

Activity: "{activity_text}"

Answer with only the label."""
    return llm.invoke(prompt).content.strip()

Now the retrieved neighbours are labelled examples, which is what makes this few-shot classification rather than decoration. This is the difference between RAG that works and RAG that appears to.

Note the labels must be in the vector store's metadata for this to be possible — a schema decision made at index time, which is why it's easy to design yourself out of.

Serving with FastAPI

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class ActivityRequest(BaseModel):
    activity_text: str

@app.post("/predict")
async def predict_delay(request: ActivityRequest):
    return {"predicted_label": classify_activity(request.activity_text)}

@app.get("/health")
async def health():
    return {"status": "ok"}

uvicorn app:app --reload

curl -X POST "http://127.0.0.1:8000/predict" \
     -H "Content-Type: application/json" \
     -d '{"activity_text":"System failure caused delay at start"}'

Pydantic gives you request validation for free — a request missing activity_text gets a 422 with a useful message rather than a 500. FastAPI also generates OpenAPI docs at /docs automatically, which makes the API testable in a browser without writing a client.

Load the model once, at startup, not per request. Module-level setup does this correctly; building the index inside the handler would rebuild it on every call.

What breaks

• Building the vector index inside the request handler. Seconds per request instead of milliseconds. Load at startup.
• Retrieving on the wrong string. Project 3's bug. If retrieval quality seems random, print what you actually sent to the retriever.
• Retrieved context without labels. Also project 3's bug, and subtler: the system runs, returns plausible labels, and the retrieval is contributing nothing. It will fail silently on exactly the cases you added retrieval for.
• Chunk size chosen once and never revisited. The highest-leverage knob in any retrieval system.
• Fitting the tokenizer/vectoriser before splitting. All three projects, and it inflates every reported score.
• No evaluation for project 3. Projects 1 and 2 have accuracy and classification reports. Project 3 has an API that returns a string, and nothing measures whether the string is right. module 12.
• Costs invisible until the bill. Every project-3 request is two API calls, embedding plus completion. At 100k requests/day that's a real number nobody computed.

Feeds into: module 09 (tracking, scale). Gaps 3, 5, 6, 8, 15 in the course roadmap all emerge from this module — these projects end at a laptop, and production is not yet described.

09 — Scale & Ops

Why it matters

Module 08 ends with three things that work on your laptop. This module covers the two problems that arrive next: data that doesn't fit in memory, and the fact that after twenty experiments you cannot remember which one produced the good number.

It's the shortest module here and the one most courses stop before. Almost everything in modules 11 and 15 onward is downstream of this page.

PySpark: when data outgrows one machine

PySpark is the Python API for Apache Spark, which distributes computation across a cluster. Reach for it when pandas runs out of memory, or when the data already lives in a distributed store.

Components

Prefer DataFrames to RDDs. DataFrames go through Spark's query optimiser (Catalyst); RDDs don't, so identical logic can run several times slower.

The workflow

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MyApp").getOrCreate()

df = spark.read.csv("data.csv", header=True, inferSchema=True)

df.printSchema()
df.show(5)

df_filtered = df.filter(df["age"] > 25).select("name", "age")

df.createOrReplaceTempView("people")
result = spark.sql("SELECT name FROM people WHERE age > 25")

Lazy evaluation

The single most important thing to understand about Spark, and many explanations don't mention it.

Transformations (filter, select, join, withColumn) do nothing when called. They build a plan. Only an action (show, collect, count, write) executes it.

This is what makes Spark fast — the optimiser sees your whole chain before running any of it, and can reorder and fuse operations. It's also why your timings look wrong: a filter over ten billion rows returns instantly, and then show() takes four minutes.

The dangerous action is collect() — it pulls the entire distributed dataset into the driver's memory, which is the exact thing you brought in Spark to avoid. Use show(), take(n), or write to storage.

MLlib

from pyspark.ml.feature import VectorAssembler
from pyspark.ml.classification import LogisticRegression

assembler = VectorAssembler(inputCols=["feature1", "feature2"],
                            outputCol="features")
data = assembler.transform(df)

train, test = data.randomSplit([0.8, 0.2])

lr = LogisticRegression(labelCol="label", featuresCol="features")
model = lr.fit(train)

predictions = model.transform(test)

VectorAssembler is the required first step and has no scikit-learn equivalent: MLlib estimators take a single vector column, not many feature columns. Every MLlib pipeline starts by assembling.

When not to use Spark

Spark has real fixed overhead — JVM startup, serialisation, shuffles across the network. Below roughly single-machine memory, pandas is faster and simpler. "We might need to scale later" is not a reason to start with Spark; migrating later is cheaper than debugging distributed code you didn't need.

It integrates with HDFS, Hive, Kafka, Delta Lake, and via connectors with TensorFlow/Keras.

MLflow: tracking experiments

You will run the same model with twenty parameter settings. Without tracking, you will not know which produced the number you're about to report. This is not a discipline problem; it's a tooling problem with a small solution.

A minimal tracked training run:

import mlflow
import mlflow.sklearn
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

def train():
    data = load_iris()
    X_train, X_test, y_train, y_test = train_test_split(
        data.data, data.target, test_size=0.2, random_state=42
    )

    model = RandomForestClassifier(n_estimators=100, random_state=42)
    model.fit(X_train, y_train)
    acc = accuracy_score(y_test, model.predict(X_test))

    with mlflow.start_run():
        mlflow.log_param("n_estimators", 100)
        mlflow.log_metric("accuracy", acc)
        mlflow.sklearn.log_model(model, name="model")

The three primitives:

• log_param — an input you chose. Hyperparameters, data version, model type.
• log_metric — an output you measured. Accuracy, loss, F1.
• log_model — the trained artefact itself, so a run is reproducible rather than merely described.

mlflow ui gives you a sortable table of every run. That table is the point.

Three things to fix in the original version:

1. Hardcoded parameters. n_estimators=100 is written twice — once in the model, once in the log. They will drift. Set a variable and use it in both.
2. The artefact path argument. The source passes "model" positionally; current MLflow expects the name= keyword. Errata module 99.
3. Logging is not registering. log_model stores an artefact under a run. Promoting a version to a named, staged model needs mlflow.register_model(...) or registered_model_name=. Code that claims to register a model and only logs one is a common and confusing pattern — check which it actually does.

The distinction is worth keeping straight, because it's the boundary between experiment tracking and deployment: the tracking server holds runs, the model registry holds versioned models with stages, and only the registry answers "what is in production right now?"

What comes next

The honest state of things: the material take you to a trained, tracked model and a FastAPI app on a laptop. Between here and production sit at least these, none of which any source covers.

Gap 15 is the one that bites hardest and is least visible. A deployed model degrades silently: inputs drift away from training data, the relationship the model learned stops holding, and accuracy falls without a single error in the logs. Module 03's metrics require labels, and in production labels arrive late or never. Solving that is its own discipline.

What breaks

• collect() on a large DataFrame. Driver runs out of memory. This is the classic Spark failure.
• Spark on small data. Slower than pandas, plus a cluster to maintain.
• Untracked experiments. Twenty runs, one good number, no idea which run produced it. Not recoverable after the fact.
• Logging a metric and not the parameters that produced it. Half a record is close to no record.
• Assuming log_model deployed something. It logged an artefact. See above.
• A model in production with no measurement. It may have stopped working months ago and nothing would say so.

Feeds into: module 11 (containers), module 15 (orchestration) and module 16 (monitoring) — the chain that carries a tracked model into production and watches it degrade.

10 — GRU & the RNN Family

Extends module 05.

Why it matters

Module 05 names GRU and moves on, which leaves you unable to answer the question that actually comes up: you have a sequence model to build, and three architectures claim to do it. The useful knowledge is not what a GRU is — it's the shape of the trade against LSTM, and the fact that neither reliably wins.

There's a second payoff. Working out why gating fixes vanishing gradients gives you the mechanism behind residual connections in module 07. It's the same idea twice.

The vanilla RNN, and why it fails

One recurrence, no gates:

h_t = tanh(W · [h_{t-1}, x_t] + b)

Every step's state passes through a weight matrix and a tanh. To backpropagate from step 50 to step 1, the chain rule multiplies 50 Jacobians together.

That product is the whole problem. Each factor has the form Wᵀ · diag(tanh′), so its size is set by two things: tanh's derivative, which is at most 1 and smaller whenever the unit is saturated, and the recurrent weight matrix W.

W is what decides the direction. If its largest singular value is below 1, the factors shrink and 50 of them multiply to approximately nothing — vanishing gradient, module 05's version in its purest form. Above 1, they compound instead and you get explosion. The tanh term can only ever shrink the product; W is the term that can do either, which is why the same architecture produces both pathologies depending on initialisation.

Either way there is no path around the multiplication, because every step's state is transformed by the next.

Gating exists to build that path.

GRU

Two gates, one state. Introduced by Cho et al. (2014).

z_t = σ(W_z · [h_{t-1}, x_t])          # update gate
r_t = σ(W_r · [h_{t-1}, x_t])          # reset gate
h̃_t = tanh(W · [r_t * h_{t-1}, x_t])   # candidate state
h_t = (1 − z_t) * h_{t-1} + z_t * h̃_t  # final state

Read the sigmoids as "how much passes through", between 0 and 1 per element.

The update gate z interpolates between keeping the old state and taking the new candidate. z ≈ 0 keeps the past unchanged; z ≈ 1 overwrites it.

The reset gate r controls how much of the past the candidate is allowed to see. r ≈ 0 computes a candidate from the current input alone — "start fresh here", useful at a phrase or sentence boundary.

Why this fixes the gradient

Look at the last line with z ≈ 0:

h_t ≈ h_{t-1}

The state is carried forward by addition, not transformation. Gradients flow back through that path without being multiplied by a weight matrix at every step, so they survive across many steps.

This is the mechanism. It is also exactly what a residual connection does in a transformer (module 07): output = sublayer(x) + x, creating an additive route for gradients past a stack of transformations. Two architectures, twenty years apart, same fix — if a gradient must survive depth, give it a path that adds rather than multiplies.

The gates make that path learned rather than fixed: the network decides, per element and per timestep, when to hold and when to update.

GRU vs LSTM

LSTM (module 05) has three gates — forget, input, output — and two states: hidden h and cell state c. GRU compresses this.

Three differences that matter:

Coupled forget and input. LSTM decides separately how much to forget and how much to add. GRU's single z couples them: (1−z) old plus z new, always summing to 1. It cannot both retain the old state and write substantial new information into the same element. Less flexible, fewer parameters.

No separate cell state. LSTM's c runs along the chain with only minor linear interaction and is protected from the rest of the network; h is what gets exposed. GRU has one state doing both jobs.

No output gate. GRU passes its full state on. LSTM can maintain internal state it declines to expose — occasionally useful, one more thing to learn.

Which is better

Neither, reliably.

Chung et al. (2014) (arXiv:1412.3555) compared GRU, LSTM and plain tanh units on polyphonic music modelling and speech signal modelling. Both gated units beat tanh clearly; GRU and LSTM came out comparable, with no consistent winner between them. Note the domain — this is music and speech, not a broad NLP benchmark, so read it as evidence rather than proof.

Greff et al. (arXiv:1503.04069; IEEE TNNLS 2017) ran the large-scale version: eight LSTM variants across speech recognition, handwriting recognition and polyphonic music, ~5,400 runs, importance assessed with fANOVA. None of the eight improved significantly on vanilla LSTM, and the forget gate and the output activation function were its most critical components — removing either significantly impaired performance.

That second paper contains a result worth pulling out, because it bears directly on the design above: coupling the input and forget gates simplified LSTM without significantly reducing performance. Coupling those two decisions is exactly what GRU's single update gate does. So the compression GRU applies is not merely cheaper — it removes flexibility that a systematic search found LSTM was not meaningfully using.

The honest summary remains that the choice is a hyperparameter, not a principle.

Both papers verified against their abstracts, August 2026. Conclusions stated qualitatively; no benchmark figures quoted.

Weak priors, and treat them as weak:

• GRU — less data, smaller model, faster iteration. Fewer parameters means less to overfit.
• LSTM — very long sequences and plenty of data, where the separate protected cell state and the extra flexibility have room to pay for themselves.

Try both. It is two lines of code and one training run.

Where the RNN family still belongs

Module 07 says transformers replaced RNNs, which is true for most NLP and misleading as a general rule. Recurrent models retain a real structural advantage worth knowing:

Constant memory and cost per step. A GRU carries a fixed-size hidden state, so generating token 10,000 costs exactly what token 10 did. A transformer attends over everything before it — O(n²) in sequence length, with a KV cache that grows as it goes (module 07's What breaks, and module 21).

That makes recurrent models the better fit for:

• Streaming and online inference — endless input, bounded memory.
• Very long sequences where quadratic attention is simply unaffordable.
• Small edge models — a GRU with a few hundred thousand parameters runs on a microcontroller.
• Time series with modest sequence lengths, where a GRU often matches a transformer at a fraction of the cost.

What you give up is the thing that made transformers win: training parallelism. Step 51 needs step 50, so GPUs idle. That is a training constraint, not an inference one, which is why the trade can favour recurrence at deployment even where it lost at training time.

Code

PyTorch — the API mirrors nn.LSTM with one deliberate difference:

import torch
import torch.nn as nn

gru = nn.GRU(input_size=64, hidden_size=128, num_layers=2, batch_first=True)

x = torch.randn(32, 10, 64)          # (batch, seq_len, input_size)
output, h_n = gru(x)                 # output: (32, 10, 128)
                                     # h_n:    (2, 32, 128)  — per layer

nn.LSTM returns output, (h_n, c_n) — a tuple, because of the cell state. GRU returns output, h_n. Swapping one for the other without changing the unpacking is the first thing that breaks.

Keras — a drop-in replacement for the module 08 classifier:

from tensorflow.keras.layers import GRU

model = Sequential([
    Embedding(input_dim=5000, output_dim=64),
    GRU(64),                          # was: LSTM(64)
    Dropout(0.5),
    Dense(4, activation='softmax'),
])

Stacking recurrent layers requires return_sequences=True on every layer but the last — each one must emit a sequence for the next to consume, and the final layer emits a single vector for the dense head. Same in both frameworks, and forgetting it produces a shape error one layer later than the mistake.

What breaks

• Unpacking a GRU's return as if it were an LSTM's. output, (h_n, c_n) = gru(x) fails — there is no cell state. The reverse swap fails the same way. The single most common error when trying GRU as an alternative.
• Expecting a large speedup from 25% fewer parameters. Both are sequential, so wall-clock is dominated by sequence length rather than parameter count, and both have heavily optimised GPU kernels. Expect a modest gain, not a proportional one.
• Falling off the fast GPU kernel without noticing. Keras uses the optimised cuDNN implementation only under specific conditions — default tanh activation and sigmoid recurrent activation, recurrent_dropout=0, unroll=False, use_bias=True, reset_after=True. Change one and training silently drops to a much slower generic path with identical results. If a GRU suddenly trains an order of magnitude slower, check this first. (Conditions as documented at time of writing — verify against current docs.)
• reset_after confusion. Two GRU formulations exist: the original applies the reset gate before the recurrent matrix multiply, the cuDNN-compatible variant applies it after. Keras defaults to reset_after=True. Weights are not interchangeable between the two, which bites when loading old checkpoints.
• Reaching for a GRU on a task with no sequential structure. If order doesn't matter, a recurrent model spends capacity learning that it doesn't — and loses to a feedforward network or gradient boosting on tabular data.
• Missing return_sequences=True when stacking. Shape error, reported one layer downstream of the cause.

Feeds into: module 07 (the additive-gradient-path insight is the residual connection), module 08 (GRU as a one-line swap in project 1), and module 21 — the O(n²) comparison here is what motivates the attention-efficiency work there.

11 — Docker for ML

Prerequisites: modules 08 and 09.

Why it matters

Module 08 ends with a FastAPI app that classifies tickets. It runs on your laptop, with your Python version, your installed packages, and a model file at a path only you have. That is not a deployment — it's a demo with a hostage.

A container turns the whole thing into one artifact that runs the same anywhere. That matters more in ML than in ordinary software, because the dependency stack is worse: CUDA versions tied to driver versions, packages that need compilers, and model weights that must match the code that loads them. "Works on my machine" has more ways to be true here, and more ways to stop being true.

This is also the gap that blocks the rest of the syllabus. Orchestration (gap 8) and monitoring (gap 15) both assume a container exists.

Container, image, and what this isn't

A container is not a virtual machine. A VM emulates hardware and runs its own kernel — gigabytes, seconds to boot. A container shares the host kernel and isolates only the filesystem, process tree, and network. That's why it starts in milliseconds.

Two words to keep apart:

• Image — the built artifact. Immutable, layered, versioned. What you push to a registry.
• Container — a running instance of an image. Disposable.

Images are built from layers, one per instruction, and each is cached. Change an instruction and every layer after it rebuilds. This is not a performance footnote — it's the single fact that determines whether your builds take ten seconds or ten minutes, and the Dockerfile below is ordered around it.

A Dockerfile for the module 08 API

FROM python:3.11-slim

WORKDIR /app

# Only if some dependency lacks a prebuilt wheel — check before keeping this.
# Most ML packages ship binary wheels, and a compiler you don't need is dead
# weight in every layer below it.
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
    && rm -rf /var/lib/apt/lists/*

# Dependencies FIRST — this layer caches across code changes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Code AFTER — editing app.py doesn't reinstall torch
COPY app.py .
COPY models/ ./models/

RUN useradd -m appuser && chown -R appuser:appuser /app
USER appuser

EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Four lines are doing real work:

COPY requirements.txt before COPY app.py. The highest-leverage line in any ML Dockerfile. Copy your code first and every one-character edit invalidates the pip layer, so you reinstall PyTorch — gigabytes — on every build. Split this way, dependencies reinstall only when requirements.txt actually changes.

--host 0.0.0.0. Uvicorn defaults to 127.0.0.1, which inside a container means the container's own loopback. The server starts, logs happily, and is unreachable from the host. See What breaks.

--no-cache-dir. pip otherwise keeps a wheel cache inside the image that nothing will ever read again.

USER appuser. Containers run as root by default. A process that doesn't need root shouldn't have it.

And .dockerignore, which matters as much as the Dockerfile:

.git
__pycache__/
*.pyc
.venv/
data/
notebooks/
*.ipynb
.env

Without it the build context — everything sent to the daemon — includes your .git history and your datasets. On an ML repo that is routinely gigabytes of files that will never enter the image.

Build, tag, run

docker build -t ticket-classifier:0.1.0 .

docker run -p 8000:8000 \
           -e OPENAI_API_KEY="$OPENAI_API_KEY" \
           ticket-classifier:0.1.0

-p 8000:8000 maps host port to container port. Without it the container's port is private and nothing reaches it.

Tag with a version, never latest. latest is a mutable label that says nothing about what it points at. You cannot roll back to it, and you cannot answer "what's running in production?" — which is the same argument module 09 makes for the model registry over a logged artefact.

Model weights: bake in or mount?

A genuine decision with no default answer.

Bake in for production serving. The image is the version, so rollback is one command and there is no way for code and weights to drift apart. That coupling is the feature, not the cost.

Mount for development, where you're swapping models constantly and a rebuild per experiment is intolerable:

docker run -p 8000:8000 -v "$(pwd)/models:/app/models:ro" ticket-classifier:0.1.0

The trap in the mounted approach is silent: nothing enforces that the weights on disk match what the code expects. A model file replaced underneath a running service gives you wrong predictions, not an error.

Image size

ML images get large by default, and large images mean slow deploys, expensive registry storage, and slow cold starts under autoscaling.

Three techniques, in order of payoff:

Use a slim base. python:3.11-slim over python:3.11 costs you nothing you were using.

Install CPU-only wheels when you aren't using a GPU. The default pip install torch pulls the CUDA build with bundled NVIDIA libraries — several gigabytes. If you're serving on CPU, that is all dead weight:

--index-url https://download.pytorch.org/whl/cpu
torch==2.3.0

This is usually the largest single saving available, and most people shipping CPU inference are carrying CUDA they never call.

Multi-stage builds when you need compilers to build wheels but not to run them:

FROM python:3.11 AS builder
COPY requirements.txt .
RUN pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt

FROM python:3.11-slim
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links=/wheels /wheels/*

The build toolchain stays in the discarded first stage.

Measure your own images with docker images and docker history rather than trusting any figure — the numbers move with every library release.

Pinning

# requirements.txt
fastapi==0.111.0
uvicorn==0.30.1
scikit-learn==1.5.0
torch==2.3.0

Unpinned dependencies destroy the exact property you containerised to get. pip install scikit-learn in March and in June give you different models from the same code, and pickled models are famously unhappy about version skew on load.

Pin transitive dependencies too, via pip-compile or uv pip compile producing a lockfile. A pinned direct dependency with floating transitives is still not reproducible.

Secrets

The module 08 RAG project needs an API key. Never put it in the image.

ENV OPENAI_API_KEY=sk-...        # do not do this

Image layers are inspectable by anyone who can pull the image — docker history will show it — and a layer that adds a secret keeps it even if a later layer deletes the file. Deletion in a subsequent layer hides it from the filesystem, not from the image.

Pass secrets at runtime: -e, --env-file, or a secret manager. Keep .env in .dockerignore.

GPU access

Containers get no GPU by default. You need the NVIDIA Container Toolkit on the host and an explicit flag:

docker run --gpus all -p 8000:8000 my-model:0.1.0

With a CUDA base image:

FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04

Use the runtime variant, not devel, unless you're compiling — devel carries the full toolkit. The container's CUDA version must be supported by the host's driver; the driver is not containerised and cannot be upgraded from inside.

Multi-service with Compose

Module 08's search project needs a vector store alongside the API:

services:
  api:
    build: .
    ports: ["8000:8000"]
    environment:
      - QDRANT_URL=http://qdrant:6333
    depends_on: [qdrant]

  qdrant:
    image: qdrant/qdrant:latest
    volumes:
      - qdrant_data:/qdrant/storage

volumes:
  qdrant_data:

Three things worth noting. Services address each other by service name (http://qdrant:6333), not localhost — each has its own network namespace. The named volume is what keeps your index alive across docker compose down; without it, container removal deletes the data. And depends_on controls start order only — it waits for the container to start, not for Qdrant to be ready to serve. Your API will come up and fail its first queries unless it retries on connect or you add a healthcheck with condition: service_healthy.

Version tags above (qdrant:latest, and the CUDA and torch pins elsewhere in this module) are illustrative — verify current tags before using them, and pin latest to something real in anything but a local experiment.

What breaks

• Binding to 127.0.0.1 inside the container. The server starts, logs normally, and refuses every connection from the host. Nothing in the output suggests a problem. Bind 0.0.0.0. The most common containerised-API failure there is.
• Copying code before installing dependencies. Every code edit reinstalls the entire ML stack. Builds go from seconds to many minutes and people start avoiding rebuilds, which is worse.
• Secrets in image layers. Covered above — deleting the file in a later layer does not remove it from the image.
• Forgetting --gpus all. The container falls back to CPU and works, just far slower. Like module 08's retrieval bug, it fails silently: nothing errors, the numbers are right, and only the clock tells you. Assert torch.cuda.is_available() at startup and refuse to boot if you expected a GPU.
• OOM-killed with no traceback. A model that loads fine locally exceeds the container's memory limit and Linux kills the process — exit code 137, no Python stack, no error message. Looks like a mysterious crash. Check docker stats and raise the limit.
• No .dockerignore. Multi-gigabyte build contexts, slow builds, and datasets or .git history baked into a shipped image.
• Unpinned dependencies. The image is no longer reproducible, which was the point.
• Containerising a per-request model load. Docker doesn't fix module 08's startup-versus-handler mistake — it just packages it.

Feeds into: module 15 (Kubernetes assumes a working image and a health endpoint) and gap 15 (monitoring a deployed model presumes it's deployed). Module 09's registry argument is the same versioning argument made about weights rather than images.

12 — Evaluating LLM & Gen-AI Systems

Prerequisites: modules 03 and 08.

Why it matters

Module 03 gave you precision, recall and F1, and every one of them needs a single correct answer to compare against. Generated text has no such thing: a summary can be excellent in a hundred different wordings, and the string-equality check that underpins accuracy is useless.

This is not an academic gap. Module 08's RAG project returns a label from an LLM and nothing in it measures whether the label is right — the retrieval bug in module 08's Project 3 survived precisely because no measurement would have caught it. You cannot fix what you cannot score, which is why this module comes before the rest of the RAG work in the course roadmap.

First: is your output actually open-ended?

The most common mistake in LLM evaluation is reaching for exotic metrics when the task has a fixed answer set.

If the model must return one of delay_start, delay_end, no_delay, you have a classification problem with an unusual classifier. Module 03 applies unchanged: build a labelled test set, run predictions, print a classification report and a confusion matrix.

Three adjustments for the LLM being the classifier:

• Set temperature=0. Otherwise the same input gives different outputs and you cannot reproduce a score.
• Parse and constrain the output. The model may return "delay_start.", "The label is delay_start", or something outside your label set. Normalise, and count unparseable responses as their own failure category rather than silently dropping them — a model that returns garbage 5% of the time is a real problem that averaging will hide.
• Score retrieval separately if you are doing RAG. See below.

Do this before anything in the rest of this module. A large fraction of "LLM evaluation" problems are classification problems that nobody built a test set for.

Evaluating retrieval

RAG has two components and they fail independently. If you measure only the final answer, you cannot tell a retrieval failure from a generation failure — and the fixes are completely different.

Retrieval is ranking, and ranking has established metrics. You need a set of queries with known-relevant documents.

Recall@k is usually the one that matters for RAG. If the answer isn't in the retrieved context, no amount of generation quality recovers it — the model will either refuse or invent. Precision matters less than you'd think, because a capable model tolerates some irrelevant context, though not unlimited amounts.

This is also the cheapest diagnostic you have. Before blaming the model, check whether the right chunk was ever retrieved. On module 08's project 3, Recall@3 against the activity text versus against the full prompt string would have shown that bug immediately.

Evaluating generated text

Now the hard part. Three families, in ascending order of cost and usefulness.

1. Overlap metrics

Compare generated text to a reference by counting shared words.

• BLEU — precision-oriented n-gram overlap. From machine translation.
• ROUGE — recall-oriented. ROUGE-N for n-gram overlap, ROUGE-L for longest common subsequence. From summarisation.
• METEOR — overlap with stemming and synonym matching.

They are fast, deterministic, and free. They also correlate weakly with human judgement on open-ended tasks, for an obvious reason: they measure surface word overlap, so a correct answer phrased differently from the reference scores badly, and a fluent wrong answer that reuses the reference's vocabulary scores well.

Use them for tasks with tight reference answers (translation, extractive summarisation) and for regression detection — a sudden ROUGE drop between two versions is a real signal even when the absolute number means little. Don't use them to decide whether a system is good.

2. Model-based similarity

Compare embeddings rather than strings.

• BERTScore — token embeddings from a pre-trained model, matched pairwise. Credits semantic similarity that overlap metrics miss.
• Semantic similarity — cosine between sentence embeddings of output and reference (module 04).

Better correlation with human judgement than BLEU/ROUGE, and still cheap. But module 04's warning applies directly: "I love this product" and "I hate this product" are extremely close in embedding space. A metric built on embedding similarity inherits that blindness, which makes it unreliable for anything where negation or factual polarity is the point.

3. LLM-as-judge

Ask a strong model to score the output against criteria.

This is what most production evaluation actually runs on, because it's the only approach that scales to open-ended quality. It is also the one most often done badly. What makes it work:

• Specific criteria, not "rate this 1–10." Score one dimension at a time — faithfulness to context, relevance to the question, completeness — with each point on the scale defined.
• Binary or few-point scales. Judges are unreliable at fine distinctions. "Is every claim supported by the context: yes/no" beats a 1–10 quality score.
• Require the reasoning before the score, not after. A judge that commits to a number first will rationalise it.
• Pairwise comparison beats absolute scoring when comparing two systems. "Which is better, A or B" is a much easier question than "score A from 1 to 10", and more stable across runs.
• Validate the judge against humans. Score 50–100 examples by hand, check the judge agrees. Without this you have a number of unknown meaning, and skipping this step is the single most common failure.

Known biases, all documented and all worth designing around: judges prefer longer answers, prefer their own outputs over other models', and are sensitive to position in pairwise comparisons — so randomise A/B order and check both directions.

RAG-specific dimensions

For a retrieval-augmented system, the useful decomposition:

Faithfulness is the one that catches hallucination, and it's checkable without a reference answer: extract each claim from the output, ask whether the context supports it. That property matters enormously in practice, because reference answers are expensive and context is already there.

These four are roughly what the RAG evaluation frameworks implement — Ragas is the common one, with DeepEval, TruLens and promptfoo covering similar ground. Framework APIs move quickly; treat any specific call signature as needing verification.

Quality is not the only axis

Everything above measures whether the output is good. Two more dimensions decide whether the system is viable, and they belong in the same evaluation run:

• Cost per query. Module 08's project 3 makes two API calls per request — embedding plus completion — and nothing in it computes what that costs at volume. Measure it per query and multiply by projected traffic before deploying, not after the invoice.
• Latency, at p50 and p95. A retrieval step, a generation step, and a judge in the loop each add to it. Tail latency is what users experience.

These trade against quality directly: a larger model, more retrieved context, or a reranking pass each buy accuracy with money and milliseconds. An evaluation that reports only quality makes that trade invisible, and it is usually the trade the decision actually turns on.

Build the test set first

Everything above assumes labelled examples, and this is where projects stall.

Start smaller than feels respectable. 50 examples covering your real failure modes beats 5,000 random ones. Build it from:

• Real queries from logs, if you have them. Always better than invented ones.
• Deliberate edge cases: ambiguous inputs, out-of-scope questions, adversarial phrasings, inputs where the correct answer is "I don't know."
• Every bug you've fixed, added as a regression case. This is how the set earns its keep over time.

Version it alongside the code. A test set that drifts silently gives you scores that aren't comparable across time, which is worse than no scores.

What breaks

• No evaluation at all. The default state, and module 08's project 3 is the worked example — a system that returns confident labels with a broken retrieval step. It ran for as long as anyone cared to look at it.
• Evaluating end-to-end only on a RAG system. You learn the score dropped, not whether retrieval or generation caused it. Measure both.
• Non-zero temperature during evaluation. Scores move between runs and you cannot tell a real regression from sampling noise.
• An unvalidated LLM judge. You have a number; you don't know what it means or whether it tracks anything you care about. Validate against human labels on a sample first.
• BLEU/ROUGE on open-ended generation. They measure surface overlap. A correct answer worded differently scores badly and you will optimise toward the reference's phrasing rather than toward being right.
• Embedding similarity where polarity matters. Inherits module 04's blindness to negation.
• Silently dropping unparseable outputs. A model failing to follow the output format 5% of the time is a headline failure, not a rounding error.
• A test set that grows by adding easy cases. Numbers improve, the system doesn't. Add cases that currently fail.

Feeds into: module 13 (RAG properly — now measurable), gap 9 (fine-tuning vs prompting vs RAG, a decision that needs a metric to settle), and gap 15 (monitoring, which is this module's metrics run continuously against production traffic).

13 — RAG, Properly

Prerequisites: modules 04, 08 and 12.

Why it matters

Module 08 built a RAG classifier that retrieves context and never puts it to work — see module 08, Project 3, and it returned plausible labels the whole time it was broken. Module 12 supplied the metrics that would have caught it. This module is the rebuild.

The framing that matters: RAG is a retrieval system with a language model attached, not a language model with search bolted on. Most RAG that underperforms does so because retrieval was treated as a solved detail — one embedding model, one chunk size, top-k, done. Retrieval is where the quality is.

Chunking

The highest-leverage decision in the system, and the one most often made once and never revisited.

The tension is fixed: chunks must be small enough that their embedding represents one thing, and large enough to be interpretable alone. A chunk spanning three topics produces an embedding that averages them and matches nothing well. A chunk of one sentence retrieves cleanly and tells the model nothing.

Start with recursive character splitting. Move to structure-aware if your documents have structure worth respecting — for anything with headings, it is usually a bigger win than tuning size.

Overlap exists so an idea severed at a boundary survives in one of the two chunks. 10–20% of chunk size is the usual starting point.

Metadata is not optional

Store, alongside every chunk: its source document, its position or section, any timestamp, and any field you will need downstream.

That last clause is the second half of module 08's retrieval bug. Module 08's classifier needed each retrieved example's label to do few-shot classification, and the labels weren't in the store — so no amount of prompt fixing could have rescued it. The schema decision at index time silently determined what was possible at query time.

Metadata also buys you filtering: restrict to a date range, a document type, a permission scope. Retrieval quality is often a filtering problem rather than an embedding problem.

Retrieval: dense, sparse, hybrid

Module 04 taught embeddings, which gives you dense retrieval — and dense retrieval alone is the most common weakness in a first RAG build.

Dense (embeddings, cosine): finds paraphrase and concept matches. Fails on exact strings it has no concept for — error codes, product SKUs, surnames, delay_start. Rare tokens get averaged away.

Sparse (BM25, keyword): term-frequency scoring. Nails exact matches and rare terms. Fails completely on paraphrase — "how do I reset my password" won't match "credential recovery procedure".

Their failure modes are close to complementary, which is why hybrid retrieval beats either on most real corpora. Run both, fuse the rankings.

The fusion detail matters: BM25 scores and cosine similarities are not on the same scale and normalising them is fiddly. Reciprocal Rank Fusion sidesteps this by using only positions:

score(d) = Σ  1 / (k + rank_i(d))        # k ≈ 60, sum over retrievers

Ranks are comparable across systems in a way raw scores never are, which is why RRF is the common default for hybrid search in vector databases. (Ecosystem claim — check what your store actually implements.)

Reranking

The single biggest quality-per-effort win available, and absent from every source document.

Retrieval uses a bi-encoder: query and document embedded separately, then compared. That separation is what makes it fast — document embeddings are computed once at index time — and it's also what limits it, since the model never sees query and document together.

A cross-encoder takes query and document as one input and outputs a relevance score. Far more accurate, because it can attend across both (module 07). It cannot be precomputed, so scoring a whole corpus is impossible.

The two compose into the standard pattern:

query → bi-encoder retrieval → top 50 → cross-encoder rerank → top 5 → prompt

Retrieve broadly and cheaply, then rerank a shortlist expensively. Recall@50 is much higher than Recall@5, so the reranker gets to fix ordering mistakes the retriever couldn't avoid — you are trading a little latency for most of the ranking quality you were missing.

Query transformation

The user's question is often a poor search query. Three cheap fixes:

• Multi-query — have the model write 3–4 variations, retrieve for each, fuse. Covers vocabulary the original phrasing missed.
• HyDE — generate a hypothetical answer, embed that, and search with it. Works because answers resemble documents more than questions do, and embedding spaces reward that resemblance.
• Decomposition — split a multi-part question into sub-questions and retrieve for each. The only one of the three that handles "compare X and Y" properly.

Each costs an extra model call. Measure before keeping them.

Building the prompt

Where module 08's retrieval bug actually died. Retrieved context has to arrive in a form the model can use.

def classify_activity(activity_text, k=3):
    neighbours = vectorstore.similarity_search(activity_text, k=k)
    examples = "\n".join(
        f'- "{d.page_content}" → {d.metadata["label"]}' for d in neighbours
    )
    prompt = f"""Classify the activity into exactly one of:
delay_start, delay_end, no_delay.

Similar past activities and their labels:
{examples}

Activity: "{activity_text}"

Answer with only the label."""
    return llm.invoke(prompt).content.strip()

Four rules the broken version violated:

1. Retrieve on the content, not the prompt. The original searched using the whole instruction block — boilerplate identical across every call.
2. Label the context. Retrieved examples without their labels are decoration; the model cannot infer how similar cases were classified.
3. Mark the boundary between retrieved material and the live question, so the model doesn't treat context as part of the instruction.
4. Say what to do when context doesn't help. Without an explicit "if the context does not support an answer, say so", the model will invent one — this is most of what people call hallucination in RAG.

For generative RAG, add: "cite the source for each claim", with source IDs in the context. Citations make faithfulness (module 12) checkable by a human in seconds instead of minutes.

Access control

Retrieval does not respect document permissions. Nothing about embedding a document preserves who was allowed to read it, so an index built over a company's files will happily surface a salary review or an unannounced acquisition to anyone who phrases a query well.

This is worth stating plainly because the failure is quiet and the fix is structural. The model never "leaks" anything — retrieval hands it a document it was entitled to receive, and it summarises it faithfully. Every safeguard at the generation layer is downstream of the mistake.

Two workable approaches:

• Filter at query time. Store the permission scope in chunk metadata and filter the search by the caller's identity. Requires a store with real metadata filtering, and requires the filter to run inside the search rather than on its results — post-filtering a top-k list silently returns fewer results than requested, or none.
• Partition the index. Separate collections per permission boundary. Simpler to reason about and harder to get wrong; costs duplication when documents are shared across boundaries.

Whichever you choose, the permission data has to be captured at index time — the same lesson as module 08's retrieval bug's missing labels, with worse consequences for getting it wrong.

Closing the loop

Everything above is a hypothesis until measured, and module 12 is how:

1. Build a small test set — 50 real queries with known-relevant documents.
2. Measure Recall@k first. If the right chunk isn't retrieved, nothing downstream matters, and this is where most RAG problems actually are.
3. Change one thing — chunk size, hybrid on, reranker in. Re-measure.
4. Then measure faithfulness and answer relevance on the generation side.

Chunking and retrieval changes are cheap and usually pay more than prompt changes. Most teams do the reverse, because prompts are more fun to edit.

What breaks

• Dense retrieval only. Silent failure on exact identifiers, error codes and rare terms. Add BM25 and fuse — usually the largest single improvement after chunking.
• Retrieving on the wrong string. module 08's retrieval bug. If retrieval seems random, print what you actually sent to the retriever before theorising.
• Context without the fields you need. Decided at index time, discovered at query time, fixed only by re-indexing everything.
• No refusal instruction. The model answers from parameters when context is unhelpful, fluently and wrongly.
• Chunk size set once. The highest-leverage knob, usually touched least.
• Reranking everything. Cross-encoders are too slow for a full corpus. Shortlist first — that's the whole point of the two-stage design.
• Tuning against a test set of easy queries. Numbers rise, the system doesn't. Module 12's warning, and it applies with full force here.
• Treating RAG as a substitute for evaluation. Grounding in retrieved documents reduces hallucination; it does not eliminate it, and a system nobody measures is a system nobody knows is working.

Feeds into: module 14 (index types and filtering), gap 7 (LangChain/LangGraph, the orchestration layer for these pipelines), and gap 9 (fine-tuning vs prompting vs RAG, now decidable because both sides can be measured).

14 — Vector Databases

Prerequisites: modules 04, 08 and 13.

Why it matters

Module 08 uses FAISS, Chroma and Qdrant across three projects and never says why each was chosen — because nothing chose them. Module 13 assumed an index that exists, works, and never changes. This module covers what actually holds those systems up: how similarity search stays fast at scale, and what happens when the documents underneath it change.

The second half is the part people discover late. An index is not a build artifact, it is a replica of a dataset that drifts out of sync, and nothing warns you when it has.

Exact vs approximate search

IndexFlatIP from module 08 is exact: every query is compared against every vector. Perfect recall, and cost that grows linearly with corpus size. This is fine much further than people expect — hundreds of thousands of vectors on one machine — and you should not abandon it before measuring.

Beyond that, you trade recall for speed with an approximate nearest neighbour (ANN) index. The trade is real and worth stating plainly: ANN indexes return wrong answers sometimes. A vector that should have been in your top 5 is occasionally missed. You accept this because a 20 ms search over 50M vectors is worth a small recall loss, but you have to know you accepted it.

Two knobs carry most of the tuning:

• IVF's nprobe — how many clusters to search. Higher means better recall and slower queries. The relevant failure: a vector near a cell boundary is missed entirely if its cell isn't probed.
• HNSW's ef_search — how wide the graph traversal is. Same trade.

Both are set at query time in the common implementations, so you can tune recall against latency without rebuilding. Build-time parameters (nlist, M, ef_construction) require a rebuild, so get those roughly right first.

Measure the recall you're actually getting. Run a sample of queries against a flat index and against your ANN index, and compare the result sets. That number is the thing you traded away, and most teams never compute it.

Choosing a store

The honest default: start with FAISS or Chroma, and move when a specific thing hurts. The usual specific things are concurrent writes, metadata filtering at scale, and not wanting to own persistence.

pgvector deserves more attention than it gets. If your data already lives in Postgres, keeping vectors there removes an entire system from your architecture, along with the consistency problem of keeping two stores in agreement. It is slower than a dedicated store at large scale and that is frequently irrelevant.

Which stores exist, which are popular, and which parameter names they expose all move quickly. Treat the tables above as a shape to reason with, and check current docs for anything you are about to depend on.

Memory is the cost

Vector search cost is dominated by RAM, because the fast index types need the vectors resident. Module 12 made cost an evaluation axis; here is where the money actually goes.

Rough shape of it, and worth computing for your own corpus before committing:

raw vectors ≈ n_vectors × dimensions × 4 bytes     (float32)

A million 768-dimension vectors is about 3 GB before any index structure. HNSW then adds its graph on top — a substantial multiplier, not a rounding error — and IVF adds centroids and lists. Three levers, in the order worth trying:

• Fewer dimensions. Cost is linear in dimension, and many embedding models now support truncation to a shorter vector with modest quality loss. Halving dimensions halves the bill.
• Quantisation. PQ or scalar quantisation compresses vectors several-fold at some accuracy cost. This is what makes large corpora affordable.
• Fewer vectors. Larger chunks mean fewer of them — which trades directly against module 13's chunking quality. There is no free choice here, only an informed one.

The reason to do this arithmetic early: memory cost scales with corpus size while your evaluation set stays small, so quality looks stable right up to the point the bill or the OOM arrives.

Filtering

Real queries are rarely "find similar" — they're "find similar among documents this user may see, from the last year, of type invoice." How a store handles that determines whether it works.

• Pre-filtering — restrict the candidate set, then search within it. Correct results, but it can defeat the ANN structure: filter to 0.1% of the corpus and the graph or cell layout no longer helps.
• Post-filtering — search, then discard non-matching results. Fast, and silently returns fewer than k results, sometimes zero, when the filter is selective. This is the module 13 access-control trap in its general form.

Good stores do filtered search inside the index traversal rather than at either end. When evaluating one, this is the capability to ask about — not raw query speed.

Keeping the index current

The half nobody plans for. Documents change; your index is a copy that doesn't.

Adds, updates, deletes

Adding is easy in most stores. The other two are where the sharp edges are:

• Updates are usually delete-then-insert. If your chunk IDs aren't stable and derived from the original document, you cannot find the old chunks to remove, and you end up with both versions in the index — retrieving contradictory answers with no error anywhere.
• Deletes are often soft: the vector is tombstoned, not removed. It stops being returned but still occupies memory and can still degrade the graph structure. Periodic compaction is a real operational task.
• HNSW deletion is genuinely awkward — the graph is built assuming its nodes exist. Heavy churn degrades recall over time in a way that no query-time knob fixes.

Design for this at index time: derive chunk IDs deterministically from (document ID, chunk index) so an update can find and replace exactly the chunks it owns.

Re-embedding

The one that catches everyone. Change your embedding model and every vector in the index becomes meaningless — not degraded, meaningless. The old and new vectors occupy different spaces, and distances between them are noise.

This includes changes you might not think of as changes: a different model version, a different provider, sometimes a different truncation or pooling setting. There is no incremental path. You re-embed the entire corpus.

So: record which embedding model and version produced the index, alongside the index itself. Then a model upgrade is a planned rebuild rather than a mysterious quality collapse that nobody connects to a dependency bump three weeks earlier.

The safe procedure is a blue-green swap — build the new index alongside the old, evaluate both against module 12's test set, then switch. Rebuilding in place means downtime and no way back.

Staleness

Even with correct update handling, there's a lag between a document changing and its chunks being re-embedded. Decide what that lag may be and measure it — a support bot answering from last month's policy is wrong in a way that reads as confident and correct.

What breaks

• Reaching for ANN too early. Flat search handles more than people assume, is exact, and has no tuning surface. Measure before complicating.
• Never measuring ANN recall. You traded correctness for speed and don't know the exchange rate. Compare against a flat index on a query sample.
• Post-filtering a top-k search. Returns fewer than k, or nothing, exactly when the filter matters most. Module 13's access-control problem in general form.
• Non-deterministic chunk IDs. Updates can't remove the old version, so the index accumulates contradictory duplicates and retrieval quality decays with no error surfaced.
• Changing the embedding model without rebuilding. Every stored vector becomes noise. The failure looks like a sudden inexplicable quality drop, and the cause is usually weeks upstream.
• Assuming deletes free memory. Most are tombstones. Index size and latency keep climbing; compaction is a scheduled job someone has to own.
• Treating the index as a build artifact. It is a replica of a live dataset. Anything true of database replication — lag, consistency, reconciliation — is true here, and nothing in the vector store will remind you.

Feeds into: module 16 (monitoring — index staleness and recall decay are production metrics), and gap 8 (Kubernetes, where a stateful vector store is the hard part of the deployment).

15 — Kubernetes for ML

Prerequisites: modules 09, 11 and 14.

Why it matters

Module 11 produced an image. An image is a thing that can run; it is not a thing that stays running, recovers from a crash, scales with load, or updates without downtime. That is what an orchestrator is for.

This is deliberately not a Kubernetes tutorial — there are better ones, and most of Kubernetes is the same whatever you deploy. What follows is the part that is different for models, which is where nearly all the pain is: model containers start slowly, hold gigabytes of memory, need GPUs, and don't scale on the signals Kubernetes measures by default. Every failure mode below comes from one of those four facts.

Reach for it when

Honest first: you may not need this. A single container on a VM behind a load balancer, or a managed serverless container platform, handles a great deal. Kubernetes earns its complexity when you need several of:

• Restart on failure and rescheduling when a node dies
• Rolling updates with rollback
• Horizontal scaling driven by load
• Bin-packing several models onto shared, expensive hardware
• One deployment story across many services

If you need one of those, use the simpler thing. The cluster is not free — someone maintains it.

The objects, briefly

Pod — containers scheduled together, the unit of execution. Deployment — "N replicas of this pod", with rollout and rollback. Service — stable name and load balancing across healthy pods. Ingress — HTTP routing from outside. ConfigMap / Secret — config and credentials injected at runtime. StatefulSet + PVC — stable identity and persistent storage.

Your API is a Deployment behind a Service. Your vector store, if self-hosted, is a StatefulSet — the hard part, below.

Probes: the ML-specific trap

Kubernetes asks a container three different questions, and conflating them is the single most common way model deployments fail.

The trap: a model that takes three minutes to load will be killed at ninety seconds by a default liveness probe, restarted, and killed again — forever. The logs show a container starting repeatedly with no error, because there is no error. It was simply not asked the right question.

A startup probe is what fixes it. It grants a long boot budget without loosening the liveness check afterwards:

startupProbe:
  httpGet: { path: /health/live, port: 8000 }
  periodSeconds: 10
  failureThreshold: 60          # up to 10 minutes to load
livenessProbe:
  httpGet: { path: /health/live, port: 8000 }
  periodSeconds: 20
  failureThreshold: 3           # strict once running
readinessProbe:
  httpGet: { path: /health/ready, port: 8000 }
  periodSeconds: 5

The two endpoints must answer different questions:

@app.get("/health/live")       # is the process wedged?
async def live():
    return {"status": "ok"}    # deliberately trivial

@app.get("/health/ready")      # can I serve a real request?
async def ready():
    if model is None:
        raise HTTPException(503, "model not loaded")
    return {"status": "ready"}

Pointing both at the same endpoint is the second common error, and it converts every transient dependency blip into a restart loop. If readiness checks a downstream service and you have wired it to liveness, a slow vector store restarts your API — which cannot possibly help, and drops in-flight requests.

Rule of thumb: liveness should test almost nothing. Only "is this process still capable of responding at all?" Everything else is readiness.

Resources

resources:
  requests:                     # used for scheduling
    memory: "4Gi"
    cpu: "1"
  limits:                       # enforced at runtime
    memory: "6Gi"
    cpu: "2"

Requests decide which node the pod lands on. Limits are enforced while it runs — and the two resources behave completely differently when exceeded:

• Memory over limit → the container is killed. OOMKilled, exit 137, no Python traceback. This is module 11's exit-137 failure, now with a second cause: not the machine running out of memory, but your own declared ceiling.
• CPU over limit → throttled, not killed. The container simply runs slower. This is worse in one way: nothing fails, latency just degrades, and the cause is invisible unless you look at throttling metrics.

Model memory is easy to underestimate. Weights are the floor, not the total — activations, the request batch, the framework's allocator and the tokenizer all add. Measure actual usage under load, then set the limit above the peak with headroom, not above the model size.

GPUs

resources:
  limits:
    nvidia.com/gpu: 1

Requires the NVIDIA device plugin running on the cluster. Two things surprise people:

• GPUs are not shared by default. One pod holds one whole GPU. Two replicas need two GPUs, and a GPU sitting at 8% utilisation is still unavailable to anything else. Sharing needs MPS or time-slicing, configured deliberately.
• GPU nodes are usually tainted, so ordinary pods stay off expensive hardware. Your pod needs a matching toleration, and a pod that "won't schedule" with GPUs available is nearly always this.

Scaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric: { name: inference_queue_depth }
        target: { type: AverageValue, averageValue: "5" }

The default CPU-based HPA is close to useless for model serving. A GPU inference pod can be saturated at 20% CPU — the CPU is waiting on the GPU. Scale on something that reflects the actual bottleneck: queue depth, request latency, or GPU utilisation, exposed through a metrics adapter.

Scaling models is also slow. A new replica must pull a multi-gigabyte image, then load the model — potentially minutes before it serves anything. So:

• Keep minReplicas above your trough load. Scaling from zero is a cold start measured in minutes.
• Scale up early and aggressively, down slowly, using stabilisation windows.
• Pre-pull images onto nodes where you can.

What this costs

Two sections above told you to keep minReplicas above trough load and to give each pod a whole GPU. On GPU nodes those are expensive instructions, and the bill runs whether or not requests arrive.

The economics that shape the decisions:

• GPU nodes bill continuously, not per request. minReplicas: 2 on GPU hardware is a fixed monthly cost — and it is often still correct, because a cold start is measured in minutes. That is a deliberate trade, not an oversight, and it should be made explicitly.
• A GPU at 8% utilisation costs the same as one at 95%. Utilisation is the number to watch. Low utilisation across several models argues for time-slicing, MPS, or consolidating onto fewer pods.
• CPU inference is often the right answer. A quantised or distilled model on CPU can be an order of magnitude cheaper per request and fast enough. Measure before assuming the GPU is required.
• Autoscaling saves less than expected when scale-up takes minutes: you must over-provision against the response lag, so the trough never drops as low as the graph suggests.

Module 12 made cost an evaluation axis. This is where that number is actually determined — not by the model, but by how much idle hardware the latency requirement forces you to hold.

Stateful: the vector store

If you self-host the store from module 14, it is a StatefulSet with a PersistentVolumeClaim, and it is the hardest part of the deployment. Stateless API pods can be killed freely; this one cannot.

Three things to get right:

• Storage. The PVC must survive pod restarts and rescheduling, and be large enough for the index plus its tombstones (module 14) plus rebuild headroom.
• Rebuild time. If the volume is lost, how long to re-embed the corpus? That number is your real recovery time, and it is usually much larger than anyone guessed.
• Memory. HNSW wants the index resident. Module 14's sizing arithmetic is now a resources.requests.memory value, and getting it wrong means OOMKill loops on a component that holds your data.

Managed vector stores exist largely to avoid this. That is a legitimate reason to use one.

Rolling updates

strategy:
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

maxUnavailable: 0 keeps capacity constant during rollout, which matters when replicas are expensive and slow to replace. The readiness probe is what makes this safe: a new pod receives no traffic until it says it is ready, so a model that fails to load never serves.

Set progressDeadlineSeconds generously — the default can mark a rollout failed while a large model is still legitimately loading.

Manifest fields and API versions shown here are current at time of writing; Kubernetes deprecates and moves them, so check against your cluster version.

What breaks

• Liveness probe killing a slow-loading model. Endless restart loop, no error in the logs. Add a startup probe. The most common ML-on-Kubernetes failure by a wide margin.
• Liveness and readiness pointing at the same endpoint. Transient dependency failures become restart loops, dropping in-flight requests and fixing nothing.
• Memory limit set to the model size. Activations and batches push past it and the pod is OOMKilled — exit 137, no traceback. Measure under load.
• CPU limits throttling silently. No crash, no error, just latency. Invisible without throttling metrics.
• HPA on CPU for GPU inference. The pod is saturated at 20% CPU, so it never scales up under load that is genuinely overwhelming it.
• minReplicas: 0. Cold start means image pull plus model load. The first user after a quiet period waits minutes.
• No resource requests. The scheduler assumes near-zero, overcommits the node, and everything on it degrades together.
• Losing the vector store's volume. Recovery is a full re-embed of the corpus, and nobody has timed it until the day it matters.

Feeds into: module 16 (monitoring — a deployed model is now something that can silently degrade), and gap 2 (serving: what a request actually costs, and the async trap in a model API, are still uncovered).

16 — Monitoring, Drift & Retraining

Prerequisites: modules 03, 09, 12 and 15.

Why it matters

Module 15 got a model into production. This module is about the fact that it will get worse there, quietly, and that nothing you have built so far would tell you.

The distinction that makes this its own discipline: software fails loudly, models fail silently. A broken endpoint returns 500s and pages someone. A model whose accuracy has fallen from 0.94 to 0.71 returns 200s at the same latency with the same payload shape. Every dashboard is green. The only symptom is that the predictions are wrong, and nothing in the system is looking at whether predictions are right.

Module 09 flagged this as the gap that "bites hardest and is least visible". This is it.

The problem with your metrics

Module 03's metrics — accuracy, precision, recall, F1 — all need labels. In production, labels arrive late, partially, or never.

Worse, labels are often biased by the model's own decisions. A loan model that declines an applicant never learns whether they'd have repaid. A fraud filter that blocks a transaction never sees whether it was genuine. You only get outcomes for the cases you let through, so your production "test set" is systematically distorted by the thing you're testing.

So production monitoring is built in layers, from what you can measure instantly to what you can measure eventually.

Layer 1 — the model is running

Standard service telemetry, still necessary: request rate, error rate, latency (p50/p95/p99), saturation. Plus the model-specific ones module 15 implies — OOMKills, restarts, GPU utilisation, queue depth.

This catches outages. It catches nothing about correctness.

Layer 2 — inputs (data drift)

Available immediately, needs no labels, and is the highest-value thing most teams aren't doing.

Data drift is a change in the distribution of inputs. Your model learned a relationship on last year's data; this month's traffic doesn't look like it.

Track, per feature:

• Summary statistics — mean, standard deviation, min/max
• Missing-value rate
• Category frequencies, and new categories that didn't exist in training
• Distribution distance from the training baseline

Common distance measures: PSI (population stability index), KL divergence, and the Kolmogorov–Smirnov statistic for continuous features. Any of them works; consistency and a stored baseline matter more than the choice.

For text and embeddings, track embedding distribution — mean pairwise distance to training centroids, or the fraction of inputs beyond a distance threshold. A sudden shift means your users are asking about something new, and module 14's index may not contain it.

Store the training baseline as an artifact alongside the model. Drift is defined relative to it, and without it you can only compare today to yesterday — which misses slow drift entirely, and slow drift is the normal kind.

Layer 3 — outputs (prediction drift)

Also label-free. Watch what the model says:

• Predicted class distribution. A fraud model that flagged 2% and now flags 11% has changed behaviour, whether or not fraud has.
• Confidence distribution. Falling average confidence often precedes falling accuracy.
• Rate of refusals, fallbacks, or unparseable outputs for LLM systems (module 12).

Output drift with stable input drift usually means something changed in the pipeline — a preprocessing bug, a feature computed differently, a model version swapped. Input drift with stable outputs can mean the model is ignoring the change, which is its own problem.

Layer 4 — outcomes (concept drift)

The real thing, and the slowest to arrive. Concept drift is a change in the relationship between inputs and labels — the world changed, not the traffic. Your inputs can look identical while the right answer has moved.

Fraud is the clearest case: fraudsters adapt specifically to your model. The feature distribution may be stable while the meaning of those features inverts.

Measuring it requires labels, so:

• Log every prediction with its inputs, model version, timestamp and a prediction ID. Without this you can never join outcomes back to predictions, and that join is the entire game.
• Join labels as they arrive, and compute module 03's metrics on a rolling window.
• Accept that the picture is weeks behind. It is still the ground truth.
• Hold out a small random sample from automated decisions where ethics and cost allow — a control group that bypasses the model gives you unbiased labels and is the only clean answer to the feedback-loop problem above.

What logging everything costs

"Log every prediction with its inputs" is the single most important instruction in this module, and it has two prices worth naming before you follow it.

Storage and ingest. At high request volume, prediction logs with full input payloads become one of the larger line items in the system, and they grow forever unless someone decides otherwise.

Privacy. For module 08's projects, the inputs are emails and support tickets. Logging every prediction means retaining personal data, indefinitely, in a system built for debugging rather than for handling it. That is a compliance question and a breach-surface question, not just an engineering one.

Both are addressed the same way:

• Sample. Full logging of 1–5% of traffic supports drift detection perfectly well — these are distributional measures, and distributions survive sampling. Log metadata for everything, payloads for a sample.
• Store derived features rather than raw input where you can. A hash, an embedding, or the feature vector supports drift detection without retaining the message text.
• Set a retention policy and enforce it, rather than discovering the question during an audit.
• Log the prediction ID for everything, always. It is small, it is not personal data, and without it the eventual label cannot be joined back.

Retraining

Three triggers, in ascending order of sophistication:

• Scheduled — retrain monthly regardless. Simple, predictable, and either wasteful or too slow, since it isn't coupled to anything real.
• Triggered — retrain when drift or accuracy crosses a threshold. Better, and requires the monitoring above to exist first.
• Continuous — retrain on a rolling window automatically. Powerful and genuinely dangerous, because a feedback loop can now degrade the model with no human in the path.

Whichever you choose, a retrained model is a new model and must be evaluated like one. Automatic retraining that deploys without a gate is a pipeline for shipping regressions at machine speed.

The minimum gate:

1. Train on the new window.
2. Evaluate against a fixed, held-out reference set that does not move with the training window — otherwise you're measuring against drifted data and cannot see decline.
3. Compare to the incumbent on that set. Deploy only on improvement.
4. Keep the previous version and the ability to roll back to it (module 09's registry, module 15's tagged images).

Shadow deployment — running the new model alongside the old on live traffic, serving the old one's answers — is how you get real-traffic evidence without risk. Expensive, and worth it for anything consequential.

What breaks

• Green dashboards, wrong predictions. Latency, error rate and uptime all healthy while accuracy has halved. The default state of an unmonitored model.
• No prediction log. Labels eventually arrive and cannot be joined to what the model actually said. Unrecoverable after the fact — log from day one.
• No stored training baseline. Drift is only measurable against a reference. Comparing today to last week hides slow drift, which is the common kind.
• Feedback loops treated as ground truth. You only see outcomes for decisions you allowed, so your production metrics flatter the model. A held-out control group is the fix.
• Retraining on a moving evaluation set. Both training and evaluation data drift together, so the score stays flat while real performance falls.
• Automatic retraining with no deployment gate. Regressions ship automatically, faster than anyone notices.
• Alerting on every drift signal. Some drift is seasonal and harmless. Alerts nobody acts on get muted, and the muted channel is where the real one arrives.
• No model version in the logs. An accuracy change cannot be attributed to a deployment, so you can't tell a bad model from a changed world.

Feeds into: closes the deployment chain that began at module 09. What remains is module 25 (responsible AI — the feedback-loop bias above is as much a fairness problem as a metrics one) and gap 2 (serving).

17 — Tokenization

Prerequisites: modules 04 and 07.

Why it matters

Module 04 climbed a ladder from one-hot to contextual embeddings, and every rung assumed text had already been split into units. Module 07 built attention over "tokens" without ever saying where tokens come from. This module is the missing rung between them.

It is also the single best explanation of otherwise baffling model behaviour. Why a model that can write a sonnet cannot count the letters in "strawberry", why the same paragraph costs twice as much in Hindi as in English, why "context length" is not measured in words — all of it is tokenization, and none of it makes sense without this module.

Why not words

The obvious approach — split on whitespace — fails in three ways at once.

The vocabulary explodes. Every inflection is a separate entry: "run", "runs", "running", "ran". Add proper nouns, typos, and product codes and you are past a million entries, each needing its own embedding row.

Out-of-vocabulary words have no representation at all. A word unseen in training maps to <unk> — the model is told "a word was here" and nothing else. For a system processing support tickets full of error codes and product names, that is most of the signal.

Related forms are unrelated entries. Nothing tells the model "running" has anything to do with "run"; it must learn that from co-occurrence, separately, for every word pair.

Characters fix all three and create a worse problem. A vocabulary of ~100, no OOV ever — but sequences become five to ten times longer, and module 07's attention is O(n²) in sequence length. You have multiplied your compute by an order of magnitude to give the model units that individually mean nothing, so it must learn to compose words from scratch.

Subword tokenization is the compromise, and it is what everything uses. Common words stay whole. Rare words split into meaningful pieces. The vocabulary is fixed and moderate — typically 30k–100k — and nothing is ever out of vocabulary.

Vocabulary sizes have trended upward as multilingual coverage improved, and the per-language cost gap described below is narrowing with it — both are worth re-checking rather than taking from this page.

BPE

Byte-Pair Encoding began as a compression algorithm and was adapted for translation models. Training is four steps:

1. Start with a vocabulary of individual characters.
2. Count every adjacent pair in the corpus.
3. Merge the most frequent pair into one new token.
4. Repeat until the vocabulary reaches the target size.

**The ordered list of merges is the tokenizer.** To tokenize new text, apply the merges in the order they were learned.

Worked small, on a corpus of low low low lower lowest:

start          l o w   l o w   l o w   l o w e r   l o w e s t
merge (l,o)    lo w    lo w    lo w    lo w e r    lo w e s t
merge (lo,w)   low     low     low     low e r     low e s t
merge (e,r)    low     low     low     low er      low e s t

((l,o) and (o,w) are equally frequent here; implementations break such ties by a fixed rule, so which goes first is deterministic but arbitrary.)

"low" earned its place by being frequent; "lowest" stays split because it isn't. That is the whole idea — frequency buys wholeness.

Byte-level BPE

Modern GPT-family tokenizers run BPE over UTF-8 bytes rather than Unicode characters. The base vocabulary is then exactly 256, and every possible string is representable.

This is why those models never emit an unknown token. Emoji, Cyrillic, corrupted bytes, a binary blob pasted into chat — all of it tokenizes into something. The guarantee is total coverage, and the price is that unusual input fragments into many tokens.

WordPiece

BERT's tokenizer. Same shape as BPE, different merge criterion: instead of raw frequency it picks the merge that most increases the likelihood of the training data — roughly, it favours pairs that co-occur more than their parts' individual frequencies would predict.

The practical difference you will see is notation. WordPiece marks continuation pieces with ##:

playing  →  play  ##ing

The ## says "this attaches to the previous token", which makes detokenization unambiguous.

Unigram and SentencePiece

Unigram works in the opposite direction from BPE: start with a large candidate vocabulary and prune it, repeatedly removing the tokens whose loss hurts corpus likelihood least. Used by T5 and many multilingual models.

SentencePiece is not an algorithm — it is an implementation that can run BPE or Unigram, and conflating the two is a common confusion. What it contributes is treating input as a raw character stream including spaces, encoding them as ▁:

▁Hello ▁world

Two consequences. Tokenization is fully reversible — you can reconstruct the exact input, spacing included. And it needs no whitespace pre-tokenization, which matters enormously for Japanese, Chinese and Thai, where whitespace does not mark word boundaries and the "split on spaces first" assumption simply fails.

Special tokens

Reserved entries with structural meaning rather than content:

Module 08's BERT classifier took bert_output[1], the pooled [CLS] representation, as its sentence vector. This is where that comes from — and why the padding mask module 08 omitted matters, since without it the model attends to [PAD] as though it were content.

Look at it yourself

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
text = "Tokenization is unintuitive: 3.14159, naïve, 🙂"

print(tok.tokenize(text))                 # the pieces
print(len(tok(text)["input_ids"]))        # what you are actually billed for

Run it on your own text rather than trusting a description. The splits are routinely surprising — long words fragment, numbers behave inconsistently, and anything non-English or symbolic expands more than you expect. Try the same string against a GPT-family tokenizer and compare; the differences are the point.

What this explains

The payoff section. Each of these is a direct consequence of the above.

Models cannot count letters. "strawberry" arrives as two or three tokens. The model never sees individual characters, so "how many r's" asks about information absent from its input. This is a representation limit, not a reasoning failure.

Non-English text costs more. Tokenizer vocabularies are trained on predominantly English corpora, so an English word is often one token while the same meaning elsewhere takes several. The identical message costs more, runs slower, and fits less of the context window — an inequity built into the representation layer, invisible unless you measure it. If you serve multiple languages, measure tokens per language before pricing anything.

Context length is tokens, not words. For English, roughly 4 characters or 0.75 words per token — a heuristic, not a rule, and materially wrong for code, numbers, and other languages. Count with the tokenizer.

Arithmetic is unreliable. Numbers tokenize inconsistently — the same digit string may be one token or several depending on context and tokenizer. Digit manipulation is being asked of a representation that does not reliably expose digits.

The tokenizer is part of the model. Token ID 5,432 means whatever that model's vocabulary says it means. Pair a tokenizer with mismatched weights and you get fluent nonsense, often with no error — the same class of failure as module 14's embedding-model swap, and just as silent.

Do you ever do anything about it?

Everything above is description. The decision it implies is narrower than it looks, and the default is firmly "use what the model ships with".

Almost always: don't touch it. The tokenizer is part of the pre-trained model. Replacing it invalidates every learned embedding, which means retraining from scratch. The convenience of a nicer vocabulary is never worth that.

Training your own is justified when you are pre-training anyway and your domain tokenizes badly under general vocabularies — genomics, chemical formulae, source code, or a language poorly served by existing models. The symptom is measurable: tokenize a representative sample and compare tokens-per-word against a general corpus. A domain that consistently fragments is paying for it on every request, in both cost and effective context.

Vocabulary extension — adding domain tokens to an existing tokenizer and resizing the embedding matrix — sits between the two. The new rows start untrained and need fine-tuning to mean anything. Workable, occasionally worth it, and more fiddly than it sounds.

One curiosity worth knowing about, because it looks like a model failure and isn't: glitch tokens. A vocabulary is learned from one corpus while the model trains on another, so some tokens end up in the vocabulary having barely appeared in training. Their embeddings are essentially random, and prompting a model with one can produce strange, evasive, or garbled output. Rare, mostly a curiosity — but if a specific unusual string makes a model behave bizarrely and reproducibly, this is a candidate explanation before you reach for anything more elaborate.

What breaks

• Mismatched tokenizer and model. IDs are looked up in a different vocabulary. Output is garbage and nothing raises. Always load both from the same checkpoint.
• Budgeting context in characters or words. Off by enough to truncate. Count tokens with the actual tokenizer.
• Silent truncation at max_length. The end of a long input is dropped without warning — and for RAG (module 13) that means retrieved context you paid to retrieve never reaches the model. Check whether truncation fired.
• Assuming uniform cost across languages. See above; it is a real budget and fairness problem.
• Asking a model to manipulate characters. Reversing strings, counting letters, spelling backwards. Do it in code.
• Cleaning text before a pre-trained tokenizer. Module 04's warning and errata module 99 — the tokenizer was trained on natural text and expects punctuation and casing.
• Trailing whitespace before a completion. It changes the token boundary and can measurably degrade output, because the model has been trained on text where the space belongs to the following token.

Feeds into: module 07 (attention operates on these units, and O(n²) is quadratic in tokens — so tokenizer efficiency is directly a compute cost), module 12 (cost per query is cost per token), and module 21, where context length becomes the binding constraint.

18 — Feature Engineering & Data Leakage

Prerequisites: modules 02, 03 and 08.

Why it matters

Module 03 taught you to measure. This module is about the most common reason a measurement lies to you.

Leakage is information reaching the model during training that will not be available when it predicts. The model learns to use it, scores beautifully on your held-out set, and collapses in production — because the thing it was relying on is not there. The offline number was never wrong; it was answering a different question from the one you asked.

Introductory material rarely covers it, which is understandable: courses describe what to build, and leakage is a thing that happens to what you built.

Part 1 — Feature engineering

Turning raw columns into inputs that expose the signal. Module 05 noted that deep networks do this automatically for images and text; for tabular data they do not, and good features still beat a better model.

Numeric

• Scaling — required for distance-based models (KNN, SVM) and neural networks, irrelevant to trees. Module 02 has the reason: trees split one feature at a time, so a monotonic rescale changes nothing.
• Log transform — for right-skewed features (salaries, response times, file sizes; module 01's skew discussion). Compresses the tail so a few extreme values stop dominating.
• Binning — continuous into bands. Loses information; occasionally worth it when the relationship is genuinely stepped, e.g. legal age thresholds.

Categorical

Target encoding replaces a category with the mean target value for that category. It is powerful and it is the most common way people leak without noticing: the encoding is computed from the target, so computing it before splitting hands each training row a value derived partly from its own label. It must be fit inside the cross-validation fold, like any other transform.

Datetime

The richest raw column and the most under-used. A timestamp is not a feature; what you extract from it is: hour, day of week, month, is-weekend, is-holiday, days-since-signup, time-since-previous-event.

Cyclical encoding deserves a note. Hour 23 and hour 0 are adjacent in reality and maximally distant as integers. Encode as a pair:

df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

Two components because one sine wave alone maps two different hours to the same value. If that reasoning feels familiar, it is exactly module 07's argument for sinusoidal positional encoding — same problem (represent position on a cycle, bounded and continuous), same solution.

Interactions and domain knowledge

Ratios and combinations are where subject knowledge enters. price_per_sqm can carry more signal than price and area separately, because it encodes a relationship the model would otherwise have to infer. This is the part that doesn't generalise across problems and is worth the most.

Missingness is a feature

Whether a field is missing is often predictive — an unfilled optional field says something about the user. Impute and add an is_missing indicator, rather than imputing silently and discarding the signal.

Which features to keep

Creating features is easy; the harder question is which survive. More features is not better — module 02's curse of dimensionality applies, and every extra column is another chance to leak, another thing to compute at serving time, and another thing to break.

Three approaches, in the order worth trying:

• Domain reasoning first. Drop what cannot plausibly cause the target. This costs nothing and catches leaky features before any statistic does.
• Model-based importance — tree importances or permutation importance. Prefer permutation importance: it measures the drop in score when a feature is shuffled, on held-out data, which is closer to the question you care about. Built-in tree importances are biased toward high-cardinality features.
• Recursive elimination — fit, drop the weakest, refit. Effective, expensive, and easy to do wrong.

The trap: feature selection is itself a leakage vector. Selecting features by looking at their relationship to the target across the whole dataset lets test labels influence which features exist, and the resulting score is optimistic — for exactly the reason module 03 gives about tuning on the test set. Selection belongs inside the pipeline, refit per fold, like every other transform.

Part 2 — Leakage

Five kinds. The first is conceptual; the rest are procedural.

1. Target leakage

A feature that is a consequence of the target rather than a predictor of it.

Predicting churn with account_closed_date. Predicting fraud with chargeback_filed. Predicting hospital readmission with discharge_summary_length, which is written after the outcome is known. Each gives spectacular offline accuracy and is useless in production, because at prediction time the value does not exist yet.

The test, applied to every feature:

At the moment I need a prediction, in production, will this value exist — and will it have this value?

The second clause is the one people skip. A field may exist and be populated later; your training snapshot shows the final value, production shows the value at decision time. Those differ.

2. Train–test contamination

Fitting any transform on the whole dataset before splitting. The scaler learns the test set's mean, the vectoriser learns its vocabulary, the imputer learns its medians, feature selection sees its labels.

Module 03 and module 04 both flagged this; module 08's projects all commit it. Fit on train, transform test. Always.

3. Temporal leakage

Random train/test splits on time-ordered data train on the future to predict the past. Offline performance is excellent and meaningless.

If your data has a time dimension and you will deploy forward in time, split chronologically — train on everything before a cutoff, test after it — and validate with TimeSeriesSplit or an expanding window rather than KFold.

4. Group leakage

The same entity appearing in both splits. Multiple rows per patient, per user, per ticket, per document. The model memorises the entity rather than learning the pattern, and looks excellent on entities it has already seen.

Split by group with GroupKFold or GroupShuffleSplit. Ask: what is the unit I will actually be generalising to? Split on that.

5. Duplicates

Near-duplicate rows landing on both sides of the split. Common in scraped data, augmented data, and anything assembled from multiple exports. Deduplicate before splitting, not after.

Detecting it

Leakage rarely announces itself. The signals:

• The score is too good. AUC 0.99 on a problem experts find hard is not a triumph, it is a symptom. Treat surprisingly good results as a bug report until proven otherwise — this single habit catches most leakage.
• One feature dominates importance. Inspect it and apply the timing test.
• Offline and production performance diverge sharply with no drift.
• Performance drops sharply when you remove one feature that shouldn't be that load-bearing.

The structural fix

Vigilance does not scale. Make leakage hard instead:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score

pre = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])

pipe = Pipeline([("pre", pre), ("model", RandomForestClassifier())])
scores = cross_val_score(pipe, X, y, cv=5)

Because every transform lives inside the pipeline, cross_val_score refits all of it on each fold's training portion only. Contamination becomes structurally impossible rather than something you remember to avoid.

It is a general lesson, not a scikit-learn one: when a failure keeps recurring despite everyone knowing better, the fix is a structure that prevents it, not another reminder.

scikit-learn API names above are stable but do move; check current docs before depending on a signature.

Training–serving skew

Leakage's production twin, and rarer in tutorials than it should be.

The model was trained on features computed by an offline script — pandas over a warehouse extract. In production, features are computed by a different service, in a different language, under latency pressure. The two implementations drift: a different null-handling rule, a timezone, a rounding difference, a window computed as "last 30 days" in one and "last 30 complete days" in the other.

Nothing errors. The model receives inputs subtly unlike its training data and degrades quietly — indistinguishable from drift (module 16) unless you check.

Mitigations, in ascending order of effort: compute features once and share the code path between training and serving; log the actual feature vectors used in production and compare their distributions to training; or adopt a feature store, which exists largely to make this one guarantee.

A note on proxies

Removing a protected attribute does not remove it from the model. Postcode correlates with ethnicity; device type correlates with income; first name correlates with gender and origin. A model with those inputs can reconstruct what you excluded and act on it.

This is a fairness problem with a feature-engineering cause, which is why it appears here — the full treatment belongs to module 25.

What breaks

• Celebrating a suspiciously high score. The most expensive mistake in this module. Investigate before you present.
• fit_transform on the full dataset. Every project in module 08. Use a Pipeline and the problem disappears.
• Target encoding computed before splitting. Each row gets a value derived partly from its own label. Fit inside the fold.
• Random splits on time-ordered data. Training on the future. Split by time.
• Ignoring groups. Same user in train and test; the model memorises rather than generalises.
• Features that don't exist at prediction time. Apply the timing test to every feature, including the second clause about when the value is populated.
• Imputing missingness away. You deleted a signal. Add the indicator.
• Assuming offline and online features match. They diverge silently; measure rather than assume.

Feeds into: module 20 (hyperparameter tuning — cross-validation is only trustworthy once leakage is handled, so this module is its prerequisite), gap 16 (proxies and fairness), and module 16, where a leaked model's production collapse is easily mistaken for drift.

19 — Prompting vs RAG vs Fine-tuning

Prerequisites: modules 08, 12, 13 and 17.

Why it matters

Every project in module 08 made this decision and none of them stated it. The ticket classifier fine-tuned BERT; the delay predictor reached for RAG on five rows of training data; nobody wrote down why.

It is the highest-leverage architectural choice in an LLM system — it decides your cost, your latency, your data requirements, and how you change the system's behaviour six months from now. It is also routinely made by defaulting to whichever technique the team read about most recently.

Modules 12, 13 and 14 built the parts. This module is the choice between them.

What each actually changes

The three techniques are often presented as a difficulty ladder. They are not — they modify different things, and the useful framing is what part of the system you are changing.

The most useful single sentence: RAG adds knowledge, fine-tuning adds behaviour. Most failed projects picked the wrong one of those two.

Fine-tuning to teach facts is the classic error. It appears to work, then the model states outdated facts fluently, cannot cite sources, and needs full retraining to correct a single one. Facts belong in retrievable context, where they can be updated, cited, and permission-scoped (module 13).

Conversely, RAG to teach format is wasteful: you spend context tokens on every request demonstrating something the weights could hold for free.

The order to try them

Ascending cost, and each step assumes the previous one failed measurably.

1. Prompting. Minutes to change, no infrastructure, no training data. Add clear instructions, output constraints, and a few worked examples. Few-shot prompting is far stronger than most people expect and is the correct baseline for almost everything.

2. RAG. When the model lacks knowledge — your documents, recent events, private data. Module 13 is the build; module 12 is how you know it worked.

3. Fine-tuning. When you need consistent behaviour that prompting cannot hold, or you need to cut the per-request cost of a long prompt. Requires labelled examples — typically hundreds to thousands — and a retraining path.

4. Both. Fine-tune for behaviour, retrieve for facts. This is what mature systems look like, and reaching it before you have measured steps 1–3 is premature.

The rule that matters: do not skip to 3. Fine-tuning is the step that feels like real engineering, which is exactly why teams reach for it first. It is also the only step that is expensive to undo.

When fine-tuning is genuinely right

Not never — the pendulum has swung far enough that it is now under-used for the things it is good at:

• Consistent structured output, where prompting gets you 95% and you need 99.9%.
• A house style or voice that takes 500 tokens to describe on every request.
• Latency and cost at volume. A small fine-tuned model can match a large prompted one on a narrow task, an order of magnitude cheaper per call. At high request volume this alone can justify it.
• A specialised domain whose vocabulary and conventions are thinly represented in pre-training — clinical notes, legal drafting, an internal DSL.
• Distillation — using a large model's outputs to train a small one for the specific task you actually run.

Note what these have in common: none is "the model needs to know things".

Parameter-efficient fine-tuning

Full fine-tuning updates every weight, needs enough memory to hold the model plus optimiser state, and produces a full-sized model per task.

LoRA and related methods instead freeze the base model and train small low-rank adapter matrices injected into its layers. Vastly less memory, minutes to hours rather than days, and adapters are small files you can swap per task against one shared base model.

For most teams this is what "fine-tuning" now means in practice, and it changes the economics enough to move the decision boundary — options that were prohibitively expensive when full fine-tuning was the only route are now reasonable.

QLoRA adds quantisation of the frozen base, cutting memory further.

This area moves fast; treat specific method names and their relative standing as current-at-writing rather than settled.

Deciding

Work down this list and stop at the first match:

1. Does the model lack facts it needs? → RAG. Not fine-tuning.
2. Is the output format or style wrong? → Prompting first; fine-tuning if prompting plateaus below your requirement.
3. Is it too slow or too expensive at volume? → Fine-tune a smaller model on the narrow task.
4. Does it need private or per-user data? → RAG, with permission filtering (module 13).
5. Does the knowledge change often? → RAG. Anything you'd have to retrain to update belongs in the index.
6. Do you have fewer than a few hundred labelled examples? → You are not fine-tuning yet. Prompt, and collect data.

Two questions worth adding, because they are usually decisive and rarely asked:

• How will you change this in six months? Editing a prompt is minutes; reindexing is hours; retraining is a project with a data pipeline behind it. You are choosing a change-cost, not just a capability.
• Who needs to understand why it answered that? RAG can cite its sources. A fine-tuned model cannot tell you why it produced anything. In regulated or contested settings that decides it on its own.

What it costs

Each option moves cost to a different place, which is why "cheapest" has no context-free answer:

Fine-tuning trades a large fixed cost for a low marginal one, so it wins at high volume and loses badly at low volume. The crossover is worth computing rather than guessing: take module 12's measured cost per request and multiply by realistic traffic.

Where the data ends up

A dimension that can eliminate an option outright, and rarely appears in the comparison.

Fine-tuning writes your training data into the weights, permanently. There is no delete. If you fine-tune on customer records and a customer later exercises a right to erasure, you cannot remove them from the model — you retrain without them, or you don't comply. The same holds for anything mistakenly included: secrets, a mislabelled batch, data you turned out not to have rights to.

RAG keeps data in an index you can delete from. Remove the document, re-index, and it is genuinely gone from future answers — the mechanism module 14 describes for updates is also your erasure path. Permission scoping works for the same reason (module 13).

Prompting keeps data in the request, which means it lives in whatever logs the request, and module 16's retention argument applies.

Where personal data is involved, this often decides the architecture before any of the capability arguments do. Worth asking early, because discovering it after a training run is expensive.

Revisiting module 08

The ticket delay project chose RAG for a three-class problem with a handful of labelled rows, and never asked whether it should have. Applying this module:

• The model does not lack facts — the task is classification with a fixed label set, and nothing needs retrieving to know what delay_start means.
• Labels were scarce, which does argue against fine-tuning and for few-shot.
• But at 5,000 labelled rows and stable categories, module 08's own analysis is right: TF-IDF into logistic regression beats all three, at a fraction of the cost and fully deterministic.

That is the outcome this module should produce most often. The fourth option is not using an LLM, and it is the correct answer more frequently than the literature suggests — a classical classifier (module 02) on a fixed label set with adequate labels is cheaper, faster, reproducible, and easier to evaluate.

What breaks

• Fine-tuning to add knowledge. It appears to work, then states stale facts confidently with no citation, and correcting one requires retraining.
• Skipping to fine-tuning before establishing a prompted baseline. You now cannot tell whether it helped, and you have taken on a retraining pipeline.
• RAG for a task needing no external knowledge. Retrieval cost and a second failure mode, purchased for nothing.
• Fine-tuning on a few dozen examples. Overfits, and degrades general capability in exchange.
• Ignoring the change-cost. The system that is cheapest to build is often the most expensive to modify, and modification is most of the lifetime.
• No baseline to compare against. All three are unfalsifiable without module 12's test set — you cannot know a change helped.
• Never considering a classical model. The most expensive omission in the list, and the easiest to check.

Feeds into: module 22 (which transformer variant to fine-tune — encoder-only for classification, decoder-only for generation), gap 10 (hyperparameter tuning applies to fine-tuning runs), and module 16, since a fine-tuned model drifts on a retraining schedule while a RAG index drifts continuously.

20 — Hyperparameter Tuning & Validation

Prerequisites: modules 02, 03 and 18.

Why it matters

Module 03 taught you to measure a model. Module 18 taught you to trust the measurement. This module is what you do with it: search the space of settings you chose rather than learned, without fooling yourself in the process.

The fooling-yourself part is the hard bit. Tuning is selection on a score, and every selection makes that score optimistic. A team that tunes 200 configurations against a validation set and reports the best one has reported the luckiest one, and the gap between that and reality is invisible without the right procedure.

Parameters vs hyperparameters

Parameters are learned from data: weights, biases, split thresholds. Hyperparameters are set before training: learning rate, tree depth, number of estimators, regularisation strength, k in k-means.

The distinction matters because they need different machinery. Parameters have gradients. Hyperparameters do not — you can only try a value, train, and see. That makes every hyperparameter evaluation cost a full training run, which is why this module is really about spending a compute budget well.

Cross-validation, correctly

k-fold splits the data into k parts, trains on k−1 and validates on the remaining one, k times, and averages. Every record validates exactly once. Cost: k trainings.

Which variant depends on your data, and the wrong one silently invalidates everything (module 18):

Stratification is the one most often skipped. On imbalanced data, plain KFold can produce a fold containing almost none of the minority class, and your variance across folds becomes noise about sampling rather than signal about the model.

The optimism problem, and nested CV

Here is the subtlety that catches good practitioners.

You run 5-fold CV across 200 configurations and take the best mean score. That score is biased upward. You selected the maximum of 200 noisy estimates, and the maximum of noisy estimates is higher than the truth — the more configurations you try, the more optimistic it gets.

Nested cross-validation separates the two jobs:

• Inner loop — tune. Search hyperparameters within the training portion.
• Outer loop — estimate. Score the whole tuning procedure on data the inner loop never touched.

The outer score answers "how well does my process generalise", which is the honest number. Cost is outer × inner trainings, which is why it is often skipped — but if you tuned heavily and then reported the tuned CV score as your expected performance, you have overstated it, and nested CV is how you find out by how much.

A cheaper approximation: hold out a final test set, never touch it during tuning, and score once at the end. Less statistically efficient, far cheaper, and much better than nothing.

Search strategies

Random search beats grid search, and the reason is worth internalising rather than memorising: in most problems only a few hyperparameters actually matter, but you do not know which. A grid spends its budget evaluating many distinct values of the parameters that do not matter, because it varies everything systematically. Random search spends the same budget sampling many distinct values of every parameter, including the ones that turn out to matter.

Put concretely: a 5×5 grid over two parameters tries 25 combinations but only 5 distinct values of each. Twenty-five random samples try 25 distinct values of each. If only one parameter matters, the grid did 5 useful experiments and random search did 25.

Hyperband is the right choice when partial training predicts final performance — most neural network training does. Train many configurations for one epoch, keep the top third, train longer, repeat. You spend most of the budget on configurations that already look good.

What to tune, in order

Budget is finite, so order matters far more than coverage.

Neural networks: learning rate first, and it is not close — it routinely moves results more than everything else combined (module 05). Then batch size, then architecture width and depth, then regularisation (dropout, weight decay), then the learning-rate schedule.

Gradient-boosted trees: learning_rate and n_estimators together, because they trade off directly — halve the rate and you need roughly twice the trees. Then max_depth or num_leaves, then subsampling rates, then the regularisation terms.

Classical models: regularisation strength (C, alpha) usually dominates.

Tune few things at once. Every added dimension multiplies the space and dilutes the budget. Two or three at a time, with the rest fixed at sensible defaults, beats a nine-dimensional search on the same budget.

Early stopping

The cheapest tuning available: monitor validation score each epoch or boosting round, stop when it stops improving, keep the best checkpoint.

It removes "how long to train" from the search entirely, and it is a regularisation method in its own right — you stop before the model has time to memorise. Requires a validation set the model is not otherwise using, and beware that the stopping point is itself selected on that set.

What it costs

Tuning is the most compute-hungry thing in a normal ML workflow, and the arithmetic is easy to get wrong by an order of magnitude:

total trainings = configurations × k folds × (outer folds, if nested)

Fifty configurations, 5-fold CV, nested with 5 outer folds is 1,250 training runs. At ten minutes each that is over eight days of compute.

Which is why the practical answers are: use random search or Hyperband rather than grid; use early stopping; tune on a stratified subsample before the full dataset; and be honest that most tuning yields a small improvement over sensible defaults. A better feature (module 18) usually beats a better hyperparameter, and costs less to find.

Log every trial with its parameters, score, and seed (module 09). Untracked tuning means you cannot reproduce the winner, and a tuning run you cannot reproduce was not an experiment.

What breaks

• Reporting the tuned CV score as expected performance. Optimistic by an amount that grows with the number of configurations tried. Use nested CV or a never-touched holdout.
• Preprocessing outside the CV loop. The scaler or encoder sees every fold's validation data. Module 18's Pipeline is the fix, and it is the most common tuning bug there is.
• Plain KFold on imbalanced classification. Folds vary wildly in class balance; your variance estimate measures sampling, not the model.
• Random splits on grouped or time-ordered data. Module 18 again — tuning amplifies leakage rather than causing it, so a leaky split makes tuning actively harmful.
• Tuning nine hyperparameters at once. The budget spreads too thin to distinguish anything from noise.
• Grid search by default. Wastes most of its budget on parameters that don't matter.
• Not fixing the random seed. You cannot tell a real improvement from run-to-run variance, and much of what looks like tuning progress is noise.
• Tuning before the baseline works. Squeezing 1% out of the wrong model.

Feeds into: module 19 (fine-tuning runs need this same discipline), and module 25 — a tuning objective is a value judgement about which errors matter, and optimising a single aggregate metric hides that choice.

21 — Attention Efficiency & Context Length

Prerequisites: modules 07 and 17.

Why it matters

Module 07 noted that attention is O(n²) and moved on. Module 17 sharpened it: the n is tokens, so your tokenizer's efficiency is directly a compute cost.

This module is the consequence. Every context-window limit you have ever hit, every "why does this cost so much at long inputs", and the entire architecture of modern inference servers descends from one quadratic term. It is also where the choice between long context and retrieval actually gets decided — a question modules 13 and 19 answered on capability grounds and this one answers on cost.

The quadratic

Self-attention compares every token to every other token. For n tokens that is n² pairs, so:

• Double the context → quadruple the attention compute.
• The attention score matrix is n × n. At 100k tokens that matrix has 10¹⁰ entries, per head, per layer.

Naively materialising that matrix is what makes long context hard. Note this is the attention term only — the feed-forward layers stay linear in n — so at short contexts attention is not the bottleneck and at long contexts it is all that matters.

The KV cache

The constraint people actually hit, and it is a memory problem rather than a compute one.

Generating token by token, the keys and values for every previous token get recomputed at each step unless you cache them. So you cache them — and that cache grows linearly with sequence length and batch size, and it lives in GPU memory for the whole generation.

KV cache ≈ 2 × layers × heads × head_dim × seq_len × batch × bytes_per_value

The factor of 2 is keys and values. For a large model at long context and any real batch size this reaches tens of gigabytes — routinely exceeding the weights themselves.

Three consequences worth holding onto:

• Batch size at inference is usually limited by KV cache, not by weights.
• Long-context requests are memory-expensive for their whole lifetime, not just at the moment of processing.
• Module 15's throughput problem is mostly this: you cannot batch more requests because the cache will not fit.

Making it cheaper

Four approaches, and the first distinction is whether they change the answer.

Exact: FlashAttention

FlashAttention is not an approximation. It computes exactly the same attention and is simply a better implementation — it tiles the computation so the n × n matrix is never written to slow GPU memory, keeping intermediate work in fast on-chip SRAM.

The insight is that attention is memory-bandwidth bound, not compute bound: the GPU spends most of its time moving the score matrix around rather than doing arithmetic on it. Avoid the round trips and you get a large speedup and a large memory saving with identical outputs.

This is why it was adopted universally and instantly. Free wins with no accuracy cost are rare.

Fewer keys and values: MQA and GQA

Standard multi-head attention gives every head its own K and V projections. But the KV cache scales with the number of heads, so:

• Multi-Query Attention (MQA) — all query heads share a single K/V head. Cache shrinks by roughly the head count. Some quality loss.
• Grouped-Query Attention (GQA) — query heads share K/V in groups. Sits between MHA and MQA, recovering most of the quality at most of the saving.

GQA is the common choice in current large models, and the reason is exactly the cache arithmetic above: it attacks the term that actually binds.

Smaller cache entries: KV quantisation

If the cache is the binding constraint, storing it in fewer bits attacks the constraint directly. Keeping keys and values at 8 bits instead of 16 halves the cache, which roughly doubles the batch size you can hold.

Quality loss is usually small, and unevenly distributed — keys tend to be more sensitive to quantisation than values, so asymmetric schemes exist. This composes with GQA rather than competing with it: fewer K/V heads and smaller entries.

Worth knowing because it is the cheapest remaining lever once you have already adopted GQA and paged attention, and because it trades a resource you are short of for one you can usually spare.

Approximate: sparse and linear attention

Give up attending to everything:

• Sliding window — each token attends only to a local neighbourhood. Linear in n, but information travels only as far as the window per layer.
• Global tokens — a few designated tokens attend everywhere and everything attends to them, providing long-range paths on top of local windows.
• Linear attention — reformulate so cost is linear, via kernel approximations. Generally weaker on tasks needing precise retrieval from context.

These change the answer. They earn their place at extreme sequence lengths where exact attention is simply unaffordable.

Serving: paged caches and continuous batching

An inference-server concern rather than a model one, and it is where most real-world throughput comes from.

Paged attention manages the KV cache in fixed-size blocks rather than one contiguous reservation per request — the same idea as virtual memory paging. It removes the fragmentation and over-reservation you get from allocating for each request's maximum possible length, which in practice allows substantially larger batches.

Continuous batching admits new requests into the batch as others finish, instead of waiting for the whole batch to complete. Since generation lengths vary wildly, this alone is a large throughput gain.

Named methods here are current at time of writing and the area moves quickly; the underlying constraints — quadratic attention, linear KV cache, bandwidth-bound kernels — are the durable part.

Context length is not context use

A model advertising a very long context window will accept that much input. It does not follow that it uses it evenly.

Retrieval accuracy from long contexts is consistently better at the beginning and end of the input than in the middle — the "lost in the middle" effect. A fact buried at 60% depth in a very long prompt may be missed entirely, while the same fact at the start is found reliably.

Two practical consequences:

• Put the important material at the edges. In a RAG prompt (module 13), the question at the end and the most relevant chunk near the top beats burying either.
• A large window is not a substitute for good retrieval. Fewer, better chunks beat many mediocre ones — which is the same conclusion module 13 reached about reranking, arrived at from the model's side.

Long context or retrieval?

The decision this module exists to inform. Modules 13 and 19 argued it on capability; here is the cost argument.

Stuffing everything into a long context costs quadratic attention plus a large KV cache on every single request. It is simple, needs no index, and gets expensive precisely in proportion to how much you rely on it.

Retrieval pays an up-front indexing cost, then sends a small number of tokens per request. Cost per request stays roughly flat as the corpus grows — because the corpus size affects the index, not the prompt.

So: long context wins for one-off analysis of a document you already have, or when the whole input genuinely must be reasoned over at once. Retrieval wins for a corpus, for anything at volume, and whenever attribution matters.

The crossover is not close for a large corpus, and it is worth computing rather than assuming. As context windows get cheaper the boundary moves — but quadratic scaling means "just use a longer context" stops being an answer at some size, and where it stops is a number you can calculate.

What breaks

• Assuming a longer window solves a retrieval problem. It buys you the chance to include the right text, not the model's attention to it.
• Budgeting inference memory for weights only. The KV cache frequently exceeds them, and it is what caps your batch size.
• Burying the critical fact mid-prompt. Lost in the middle. Put it at an edge.
• Assuming context length is measured in words. Module 17 — it is tokens, and non-English and code are worse than you expect.
• Treating FlashAttention as an approximation and worrying about accuracy, or conversely treating sparse attention as free when it does change the answer.
• Benchmarking throughput on uniform request lengths. Real traffic has wildly varying generation lengths; that variance is the whole argument for continuous batching, and a uniform benchmark hides it.

Feeds into: module 15 (KV cache is why GPU inference pods scale on queue depth rather than CPU), and modules 13 and 19, whose retrieval-versus-context arguments this one completes with the cost side.

22 — Transformer Variants

Prerequisites: modules 07, 17 and 19.

Why it matters

Module 07 built the transformer as an encoder–decoder pair. Almost nothing you will use is one. Module 08 reached for BERT to classify email without saying why, and module 19 told you when to fine-tune without saying what.

The family splits three ways, and the split is not cosmetic — it determines what a model can see when it makes each prediction, which decides what it is good for and what it cannot do at all.

The three shapes

Everything below follows from that first column.

Encoder-only (BERT family)

No causal mask (module 07). Every token attends to every other, in both directions, so each token's representation is informed by its entire context — words before and after.

Pre-trained by masked language modelling: hide ~15% of tokens and predict them. That objective only makes sense bidirectionally, which is precisely why the architecture forgoes the mask.

This is what makes it right for classification. To label an email you have the whole email; a representation built from full context is strictly better than one built left-to-right. Module 17's [CLS] token exists for exactly this — a slot whose final embedding summarises the sequence, which you attach a classifier head to.

What it cannot do is generate. There is no next-token objective and no causal mask, so there is nothing to sample from.

Use for: classification, named entity recognition, extractive question answering, and — importantly — as the encoder half of retrieval (modules 13 and 14). Sentence-transformers are encoder models.

Decoder-only (GPT family)

Causally masked (module 07). Each position sees only what precedes it.

Pre-trained by next-token prediction on enormous corpora. That single objective turned out to scale further than anyone expected, which is why this shape now dominates.

The strength is generality: because the task is "continue this text", almost any task can be expressed as one. Classification becomes "the label is ___". That framing is what makes prompting (module 19) possible at all.

The cost is bidirectionality. When classifying with a decoder-only model, the representation of an early token cannot be informed by a later one — so for a fixed model size, an encoder is usually the better classifier. Scale has largely papered over this, which is a statement about compute rather than architecture.

Encoder–decoder (T5 family)

The original shape. The encoder reads the input bidirectionally, the decoder generates causally, and cross-attention (module 07) connects them.

Natural fit where input and output are different sequences with a strong dependency: translation, summarisation, and structured transformation. The encoder gets full bidirectional access to the source while the decoder still generates properly.

Less common now than it was, largely because a sufficiently large decoder-only model does these tasks acceptably and one architecture is simpler to scale than two.

Choosing

The question that resolves nearly every case: do you need to generate text?

• No, you need a label, a span, or a vector → encoder-only. Smaller, faster, cheaper, and better at it. This is where module 08's classifier belongs and why BERT was the right call there even though the module never said so.
• Yes, open-ended → decoder-only.
• Yes, and the output is a transformation of a specific input → encoder–decoder is a natural fit, though a decoder-only model will also do it.

Two follow-ups that decide the remaining cases:

• Is the task narrow and high-volume? A fine-tuned encoder is typically orders of magnitude cheaper per request than a prompted large decoder (module 19's cost table). At volume this dominates.
• Do you need one system for many tasks? A decoder-only model handles them all through prompting; a fleet of task-specific encoders does not.

The classification cost trap

Worth stating plainly, because the default has quietly inverted.

The instinct now is to reach for a large decoder-only model and prompt it to classify. It works, needs no training data, and demos in minutes. It is also frequently 100–1000× more expensive per request than a fine-tuned encoder that would be more accurate on the same fixed label set — and slower, and non-deterministic.

Module 19's decision list already covers this. Restated here for the specific case: for a stable label set with a few thousand labelled examples, a fine-tuned encoder beats a prompted decoder on accuracy, cost, latency and reproducibility simultaneously. The prompted decoder wins when labels are scarce, categories churn, or you need an explanation with the label.

And module 19's fourth option still applies — TF-IDF into logistic regression (module 02) may beat both.

What "variant" also means

Three axes get conflated under this heading. Worth separating, because they are chosen independently:

• Architecture shape — the three above. Determines what the model can see.
• Size — parameter count. Determines capability and cost, within a shape.
• Post-training — instruction tuning and preference optimisation, which turn a next-token predictor into something that follows instructions. A base model and an instruct model of identical architecture and size behave completely differently in a chat setting.

Reaching for an instruct-tuned model when you plan to fine-tune on your own task is often the wrong start: you may be fighting the post-training. Reaching for a base model when you plan to prompt is almost always wrong.

Which specific families lead on which task moves constantly; the three shapes and what each can see are the durable part.

What breaks

• Prompting a large decoder to classify a fixed label set at volume. Works, demos well, and costs orders of magnitude more than a fine-tuned encoder that would be more accurate.
• Expecting an encoder-only model to generate. No next-token objective, no causal mask, nothing to sample.
• Using a decoder-only model for embeddings without checking. Causal masking means the representation of each token lacks right-hand context; encoder models are trained for the job. Purpose-built embedding models exist — use one.
• Fine-tuning an instruct model when you wanted a base model. You are training against post-training that is pulling the other way.
• Comparing model sizes across shapes. A 300M encoder can beat a 70B decoder at classification. Parameter count only compares within a shape and task.
• Assuming bigger is better for a narrow task. Module 19's argument again: at volume, the smallest thing that clears your accuracy bar is the right answer.

Feeds into: module 19 (this is the "what to fine-tune" that decision leaves open), and modules 13 and 14, where the retrieval half is an encoder and the generation half a decoder — a fact that makes the two-model architecture of a RAG system obvious rather than arbitrary.

23 — Serving Models

Prerequisites: modules 08, 11 and 15.

Why it matters

Module 08 built a FastAPI app and never said why FastAPI. Module 11 put it in a container, module 15 scheduled it on a cluster — and neither addressed what happens inside the process when a request arrives.

That gap matters because model serving breaks the assumptions ordinary web serving is built on. A normal request handler is I/O-bound and takes milliseconds. A model handler is CPU- or GPU-bound and takes hundreds of milliseconds to seconds, holding the interpreter the whole time. Frameworks tuned for the first case fail in specific, confusing ways in the second.

Flask or FastAPI

The question module 08 skipped.

FastAPI is the better default, for three reasons that all matter in practice:

• Request validation is free. A malformed request gets a 422 with a useful message instead of a 500 from deep inside your handler.
• Generated OpenAPI docs at /docs make the service testable in a browser with no client.
• Async matters when your handler waits on something else — an LLM API, a vector store, a feature service. Module 08's RAG project makes two network calls per request and spends nearly all its time waiting.

But the third point has a trap that catches almost everyone.

The async trap

@app.post("/predict")
async def predict(req: Request):
    return model(req.text)          # blocking, CPU-bound — WRONG

async def puts the handler on the event loop, which is a single thread. A blocking CPU-bound call inside it does not yield — so it blocks every other request in the process for its whole duration, including the health checks module 15 depends on.

This is worse than plain synchronous code, because at least a sync framework's workers block independently.

The fix is to know which kind of work you have:

@app.post("/predict")
def predict(req: Request):          # note: def, not async def
    return model(req.text)          # FastAPI runs it in a thread pool

Rule: async def when the handler awaits I/O. Plain def when it does blocking or CPU-bound work — FastAPI will run it in a thread pool and the event loop stays free.

For genuinely CPU-bound Python work the thread pool only partly helps, because the GIL serialises Python bytecode. Native inference libraries usually release the GIL during the actual computation, so this works better than it sounds — but scaling CPU inference means more processes, not more threads.

What a request costs

Nobody measures this until the bill or the pager arrives. Three components:

• Model load — seconds to minutes. Startup only. If it appears in your per-request path, that is module 08's mistake and no amount of scaling fixes it.
• Inference — the real work. Roughly linear in input size for encoders, linear in generated tokens for decoders (module 21).
• Memory per worker — and this is the one that surprises people.

Each worker process holds its own copy of the model. Four workers with a 2 GB model is 8 GB, not 2 GB. This interacts directly with module 15's memory limits: a pod sized for one model that runs four workers gets OOMKilled, and the exit code 137 tells you nothing about why.

So worker count is not a free throughput dial. It is bounded by memory, and the arithmetic is workers × model_size + overhead, which you should compute before setting it.

Batching

The highest-leverage optimisation for GPU serving, and the one most services skip.

A GPU processing one request at a time is mostly idle — the work is small relative to the cost of dispatching it. Processing 16 requests together takes barely longer than one, so throughput rises by nearly 16×.

Dynamic batching collects incoming requests for a few milliseconds, runs them as one batch, and returns the results separately. The trade is explicit: you add a small fixed latency to every request in exchange for large throughput gains.

Two things to get right:

• The window is a latency budget. 5 ms is invisible to users and fills batches under load; 100 ms is not invisible. Set it from your p99 target, not by feel.
• Under light traffic it does nothing but add delay, because batches never fill. Good implementations dispatch early when the queue is empty.

For LLM serving specifically, module 21's continuous batching supersedes this — requests join and leave a running batch rather than being grouped up front.

Model servers

You do not have to write this yourself. Dedicated servers give you batching, multi-model hosting, versioning and metrics without hand-rolling any of it — TorchServe, TensorFlow Serving, Triton, and for LLMs specifically the vLLM-style servers built around paged attention (module 21).

The trade is a framework to learn and a deployment shape to accept, against batching and metrics you would otherwise write badly. Rough guide: a single model behind a simple API is fine in FastAPI; several models, real throughput requirements, or GPU batching justifies a model server.

Specific servers and their relative standing move quickly — the capabilities above are what to compare on.

The health endpoints module 15 needs

Serving-side implementation of module 15's probe semantics:

model = None

@app.on_event("startup")
def load():
    global model
    model = load_model("/app/models/classifier")   # once, at startup

@app.get("/health/live")
async def live():
    return {"status": "ok"}            # deliberately trivial

@app.get("/health/ready")
async def ready():
    if model is None:
        raise HTTPException(503, "model not loaded")
    return {"status": "ready"}

Liveness tests almost nothing — it only answers "is this process wedged", and its failure means restart. Readiness answers "can I serve", and its failure means "take me out of rotation". Both are async def because both are trivial and non-blocking, which is exactly when async def is correct.

What breaks

• async def around a blocking model call. Blocks the entire event loop — every concurrent request and the health checks. Use plain def.
• Loading the model per request. Seconds per call instead of milliseconds. Load at startup.
• Setting worker count without the memory arithmetic. Each worker holds a full copy; four workers OOMKill a pod sized for one.
• Binding to 127.0.0.1 in a container. Module 11 — the server starts happily and is unreachable.
• No batching on GPU. Leaves most of the throughput unused.
• A batching window set by feel. It is a latency budget; derive it from your p99 target.
• Serving without per-request metrics. Latency percentiles, queue depth and error rate are what module 15's autoscaling and module 16's monitoring both need. Without them you are scaling blind.

Feeds into: module 15 (worker memory arithmetic sets resources.requests, and queue depth is the scaling signal), module 16 (per-request metrics are layer 1 of monitoring), and module 21, whose continuous batching is the LLM-specific successor to dynamic batching.

24 — Orchestration Frameworks

Prerequisites: modules 13, 19 and 23.

Why it matters

Module 13 built a RAG pipeline out of parts: retrieve, rerank, build a prompt, call a model, parse the result. Chaining those steps is what orchestration frameworks do, and module 08's projects used one without ever explaining what it was for.

This module is deliberately about concepts, not imports. Framework APIs in this area churn faster than almost anything else in the tutorial — module 99's "stale by design" argument applies with full force. What survives is the set of problems these libraries solve, and the judgement about when a library is the right answer.

What they actually give you

Strip away the API surface and there are five things:

• Composition — chain steps so one's output feeds the next, without writing the glue each time.
• Provider abstraction — swap model or vector store without rewriting everything downstream.
• Prompt templating — parameterised prompts with variable substitution, versioned as objects rather than scattered f-strings.
• Structured output — coerce free text into a schema, with retries when the model does not comply.
• Observability — trace what each step received and produced, which matters enormously because a chain fails in the middle and the symptom appears at the end.

That last one is the most under-appreciated and the hardest to retrofit. Module 13's retrieval bug — the wrong string sent to the retriever — is exactly the class of failure that tracing makes obvious in seconds and reasoning makes invisible for weeks.

Chains and graphs

The distinction that matters, and it is why there are two names.

A chain is a directed sequence: A → B → C. Fine for retrieve → prompt → generate, which covers most of what module 13 built.

A graph allows cycles, branches and conditional edges. That is what you need once the system makes decisions about its own control flow:

• Retry with a different strategy when the first retrieval returns nothing relevant.
• Route to different handling based on the query type.
• Loop — call a tool, examine the result, decide whether to call another.
• Reflect — critique a draft and revise it.

The moment a system decides what to do next based on what just happened, you have a graph. Trying to express that as a chain produces a mess of nested conditionals, which is the honest signal that you should have reached for the other abstraction.

Agents, briefly

An "agent" in this sense is a graph with a loop: the model chooses a tool, the tool runs, the result comes back, the model decides again — until it decides it is done.

The two things to know before building one:

• They are hard to evaluate. Module 12's metrics assume a fixed input and one output. An agent takes a variable path through variable tools, so you evaluate the trajectory as well as the answer, and there is no settled way to do it.
• They fail expensively. A loop that does not terminate makes model calls until something stops it. Always set a hard step limit and a cost ceiling — these are not optional guards, they are load-bearing.

Reach for a fixed chain unless you genuinely need decisions at runtime. Most tasks described as needing an agent are a chain with two branches.

State

The reason graph frameworks exist as separate things rather than a helper function.

A multi-step system needs somewhere to keep conversation history, intermediate results, which steps have run, and accumulated tool outputs. Graph frameworks model this as an explicit state object passed between nodes, each returning an update.

Making state explicit is what buys you the useful properties:

• Resumability — a run that fails at step 7 restarts from step 7, not step 1. Worth a great deal when steps cost money.
• Inspectability — you can see exactly what the system knew at each point.
• Persistence — a conversation resumes tomorrow, from durable storage rather than process memory. Module 23's workers are stateless and disposable, so state in a Python variable does not survive a restart or a second replica.

That last point is the one that bites when a working prototype meets module 15.

When not to use one

The honest section, and the one most such discussions omit.

Frameworks cost you: a dependency that changes fast, an abstraction between you and the model API, and debugging that means reading someone else's stack. Against that, the composition benefit for a three-step pipeline is close to zero — retrieve, format a prompt, call a model is twenty lines of plain code that you fully understand.

Use plain code when:

• The pipeline is a handful of linear steps.
• You use one provider and will not swap.
• You want to see exactly what is sent to the model, with nothing in between.

Use a framework when:

• Control flow is genuinely dynamic — branching, looping, retrying.
• You need durable state across requests or sessions.
• You want tracing without building it.
• You are swapping providers or comparing models regularly.

A defensible middle path: write the first version in plain code, learn the actual shape of the problem, and adopt a framework when a specific pain appears. Adopting one before you have the pain means learning its opinions about a problem you have not met.

Evaluating the frameworks themselves

Since the specifics will be stale, evaluate on properties instead:

• Can I see the exact prompt sent? If getting the final string is hard, you will not be able to debug it. This is the single most important question.
• How much magic is between me and the API call? Abstractions that hide token counts hide cost (module 12).
• Does it support streaming? Retrofitting it later is painful.
• What happens on failure mid-chain? Retry semantics, partial results, resumability.
• Does tracing come with it, or is it a paid add-on?

Framework names, APIs and relative standing all move fast enough that anything specific here would be wrong soon. The properties above are what to compare on.

What breaks

• Adopting a framework for a three-step pipeline. A dependency and an abstraction bought for glue you could have written in twenty lines.
• Modelling dynamic control flow as a chain. Nested conditionals accumulate until the code is worse than the graph would have been.
• Agents without a step limit or cost ceiling. A non-terminating loop calls a paid API until something external stops it.
• State in process memory. Works in the prototype, breaks at the second replica or the first restart — module 15's pods are disposable.
• Not logging the final prompt. Module 13's bug survived because nobody printed what was actually sent. Whatever else you skip, log that.
• Copying framework code from tutorials. This ecosystem churns faster than any other in this course; module 99's rule applies double.
• Assuming a chain that runs is a chain that works. Every step "succeeding" while the retrieval contributes nothing is the exact failure mode of module 08's project 3.

Feeds into: module 12 (tracing is where evaluation data comes from), module 16 (traces are the prediction log), and module 25 — a system that takes actions through tools raises questions of authority and accountability that a system which only answers does not.

25 — Responsible AI

Prerequisites: modules 03, 16, 18 and 19.

Why it matters

This module has been accumulating throughout the tutorial. Module 18 noted that dropping a protected attribute does not remove it. Module 13 found that retrieval ignores document permissions. Module 16 showed that production labels are biased by the model's own decisions. Module 19 established that fine-tuning makes data permanent.

Each was flagged and deferred here. They are not separate topics — they are the same subject appearing wherever a system meets a person.

The framing that makes this engineering rather than sentiment: these are failure modes with technical causes and technical mitigations. A model that denies loans by postcode is not being unethical; it is fitting a proxy variable, which is a data problem with a data fix. Treating it as a values debate rather than a bug is how it survives to production.

Bias

Bias enters through the data, and mostly before anyone writes a model.

Historical bias — the data faithfully records a world with unequal outcomes, so a model trained to predict what happened learns to reproduce it. A hiring model trained on a decade of hiring decisions learns that decade's preferences. Nothing is broken; the model works exactly as specified, and the specification was the problem.

Representation bias — some groups are thinly represented, so the model's accuracy is worse for them. Aggregate accuracy hides this completely: a model 90% accurate overall can be 95% on a majority group and 60% on a minority one, and module 03's warning about aggregate metrics applies with force.

Measurement bias — the label is a proxy for what you actually care about, and the proxy is unevenly wrong. "Arrested" proxies "committed a crime"; "clicked" proxies "found useful"; "hired" proxies "would have performed well". Where policing or promotion differed by group, the proxy's error differs by group too.

Proxy features — module 18's point. Postcode, device type, first name and purchase history all correlate with protected attributes. Removing the attribute leaves the model able to reconstruct it.

Measuring it

You cannot manage what you do not measure, and the measurement is straightforward: compute your metrics per group, not just in aggregate.

for group in df["group"].unique():
    mask = df["group"] == group
    print(group, classification_report(y_true[mask], y_pred[mask]))

That one loop surfaces most representation and measurement bias, and almost nobody runs it.

Beyond it, several formal fairness criteria exist:

These are mutually incompatible. It is a mathematical result, not a limitation of current methods: except in degenerate cases, you cannot satisfy calibration and equalised odds simultaneously when base rates differ between groups.

That has a consequence worth stating plainly. There is no "fair" setting to switch on. You must choose which criterion matters for your application, and that choice is a judgement about consequences — exactly like module 03's precision/recall trade, which was also never a purely technical decision. The engineering contribution is making the trade explicit and measured rather than implicit and unexamined.

Privacy

Four points already established, collected:

• Training data is permanent in weights (module 19). No deletion without retraining, which makes right-to-erasure a design constraint rather than a feature request.
• Retrieval ignores permissions (module 13). Embedding a document preserves nothing about who could read it; the fix is filtering inside the search or partitioning the index.
• Prediction logs are personal data (module 16). "Log every prediction with its inputs" means retaining user content indefinitely in a debugging system. Sample, store derived features, set retention.
• Memorisation. Large models can reproduce training data verbatim, especially rare sequences — a credential, an address, a distinctive paragraph. Fine-tuning on a small corpus makes this more likely, not less.

The practical rule that follows: decide what data goes where before building, because the architecture decides your privacy properties (module 19), and retrofitting them means rebuilding.

Failure modes of generated text

Distinct from bias, and specific to systems that produce language:

• Hallucination — fluent, confident, wrong. Module 13's refusal instruction and module 12's faithfulness metric are the mitigations; neither eliminates it.
• Sycophancy — agreeing with the user's premise rather than correcting it. Actively harmful when the user is wrong about something that matters.
• Overconfidence — no calibrated uncertainty. Fluency reads as confidence to users, and there is no signal separating a well-grounded answer from a guess.
• Prompt injection — instructions embedded in retrieved or user content that the model follows. This one has no complete fix. The model cannot reliably distinguish instructions from data, because both are text in the same context. Treat model output as untrusted, and never grant it authority you would not grant the author of the input document.

That last point is why module 24's agents deserve caution: a system that only answers can be wrong, but a system that takes actions can be wrong and act on it.

What to actually do

Concrete practices, in the order they pay off:

1. Report metrics per group. The single highest-value practice here. If you adopt one thing, this.
2. Document the system. What it is for, what data trained it, what it should not be used for, known limitations, and how it performs across groups. Writing down what it should not do prevents most misuse, and the exercise surfaces assumptions nobody had stated.
3. Keep a human in the loop for consequential decisions. Credit, employment, healthcare, criminal justice. The model produces a recommendation with its evidence; a person decides.
4. Make it contestable. Anyone materially affected should be able to find out why and challenge it. RAG's citations (module 13) support this; a fine-tuned model's opacity does not — a real point in RAG's favour that module 19 raised.
5. Monitor for disparate drift. Module 16's drift monitoring, computed per group: performance can decay for one group while aggregate metrics stay flat.
6. Bound what automated systems can do. Step limits, cost ceilings, and no irreversible actions without confirmation (module 24).

Where this fits

Two failure modes to avoid in how you treat this module.

Treating it as a compliance checkbox — a review at the end, a form, a sign-off. By then the architecture has decided most of the outcomes: what data went where, which model shape, whether decisions are attributable. Module 19's data-permanence argument is the clearest case — by the time anyone reviews, the training run has happened.

Treating it as someone else's job. The measurements are per-group metrics and drift dashboards. The mitigations are index partitioning, retention policies, sampling, and step limits. These are all engineering, and the engineer is the person positioned to notice.

What breaks

• Aggregate metrics only. 90% overall can be 95% and 60% by group. One loop reveals it; almost nobody runs it.
• Removing the protected attribute and declaring the problem solved. Proxies reconstruct it (module 18).
• Expecting a fairness setting. The criteria are mutually incompatible; you choose, and the choice is about consequences.
• Fine-tuning on personal data without an erasure plan. No deletion without retraining (module 19).
• Trusting model output as instructions. Prompt injection has no complete fix — treat output as untrusted, and scope tool authority accordingly.
• Automating a consequential decision end to end because the accuracy looked adequate. Aggregate accuracy is not the relevant question for the person affected.
• Deferring all of this to a review at the end. The architecture has already decided most of it.

Feeds into: nothing further — this is the last module. Where you go next is best decided by what you missed in the question sets, not by a syllabus.

99 — Corrections

Claims that are commonly stated wrong, and the correction. Plus fixes made to this tutorial's own modules.

Why this module exists

Two lists, for two different reasons.

Commonly confused collects places where the standard explanation is routinely mangled — in course notes, blog posts, revision summaries, and generated explainers. These are not exotic edge cases; they are the specific sentences that get transcribed slightly wrong and then repeated. Several will contradict something you have read elsewhere, and that is the point.

Corrections records fixes made to modules in this tutorial after they were written. It is generated material about a moving field, so the same scrutiny applies to it as to anything else.

Reading with an eye for the error is a different activity from reading to absorb — you cannot check a claim you have not understood. It is worth doing deliberately rather than hoping it happens.

Commonly confused

C-01 — "Anomaly reduction"

The unsupervised task of finding outliers is anomaly detection. You are identifying the odd records, not reducing them. The mangled name usually arrives by contamination from the neighbouring topic, dimensionality reduction.

C-02 — Standard deviation

Frequently defined as something like "the mean of the squares of the differences from the mean" — which describes variance, badly.

Standard deviation is the square root of the variance. Variance is the mean squared deviation; taking the square root returns the measure to the original units, and that is the entire reason it exists. Module 01.

C-03 — What standard deviation controls

Often stated as controlling the height of a normal curve. It controls the spread. Height follows from it: total area under the curve is fixed at 1, so a wider curve is necessarily flatter. Stating it the other way round inverts the causation and makes larger SD hard to reason about.

C-04 — Semi-supervised label propagation

"Label one point and the rest label themselves" is the intent, not a guarantee. Propagation assumes points close in feature space share a label. Where that holds, it works; where it does not, you have confidently mislabelled the whole dataset with nothing raised. Face-photo clustering is the standard example precisely because face embeddings separate unusually cleanly — most data does not.

C-05 — ROC curve axes

The most damaging error on this list.

The x-axis of a ROC curve is the false positive rate, which is 1 − specificity — not specificity. Specificity is TN/(TN+FP); FPR is FP/(FP+TN). They are complements, so reading one as the other inverts every curve you interpret. Module 03.

C-06 — Vanishing vs exploding gradient remedies

The two remedy lists get swapped constantly — ReLU prescribed for exploding gradients, gradient clipping for vanishing.

• Vanishing → ReLU (gradient of 1 for positive inputs, so it does not shrink), LSTM/GRU gating, residual connections, better initialisation.
• Exploding → gradient clipping (caps the norm), truncated backpropagation through time, lower learning rate.

The logic is directional and worth holding onto: clipping only bounds growth, so it can only help explosion; ReLU only avoids shrinkage, so it can only help vanishing. Neither touches the other. RMSprop, being adaptive, is relevant to both. Module 05.

C-07 — Confusion matrix layout

Cells get transposed often enough to be worth stating plainly:

Read each term as two words: the second says what was predicted, the first says whether that prediction was right. Module 03.

Stale by design

A category rather than a list. Any code you find in a tutorial, a blog post, or a generated answer was written against library versions that have since moved, and the following have all changed within recent memory:

• LangChain split into provider packages — langchain_openai, langchain_huggingface, langchain_chroma, langchain_community. Older monolithic imports break or warn, and the .run() chain interface gave way to .invoke().
• Keras 3 removed Embedding(input_length=...), and preprocessing.text.Tokenizer is superseded by the TextVectorization layer.
• Chroma no longer needs an explicit .persist() when a persist_directory is set.
• Qdrant replaced recreate_collection(vector_size=...) with create_collection(vectors_config=VectorParams(...)), and points are PointStruct objects.
• MLflow deprecated the positional artifact-path argument to log_model in favour of name=.

The general rule matters more than the specifics, which will themselves be stale: treat any code block as a design to implement, not a snippet to paste. Check the current documentation for anything you are about to depend on. Module 11's argument for pinned dependencies is the same argument from the other direction.

Corrections to this tutorial

Fixes applied to modules after they were written. Numbered G-NN.

G-01 — Module 10 attributed vanishing gradients mainly to tanh

Was: "Each factor is typically below 1 (tanh saturates, and its derivative is at most 1)." Correct: The factor is Wᵀ · diag(tanh′). The tanh term is bounded above by 1 so it can only ever shrink the product — but the recurrent weight matrix W sets the direction. Largest singular value below 1 gives vanishing, above 1 gives explosion. That is why one architecture produces both pathologies depending on initialisation, which the original wording could not explain.

G-02 — Module 10 gave an incomplete account of the LSTM-variant literature

Was: "…with the forget gate the component that mattered most." Correct: The large-scale variant study identifies the forget gate and the output activation function as critical; removing either significantly impairs performance. The same work found that coupling the input and forget gates simplifies LSTM without significant loss — directly relevant, since that coupling is exactly what GRU's update gate does.

G-03 — Module 11's Dockerfile contradicted its own advice

Installed build-essential unconditionally, two sections above an image-size section arguing against precisely that kind of dead weight. Most ML packages ship binary wheels; now marked conditional.

G-04 — Module 11 showed depends_on without its caveat

depends_on controls start order only — it waits for the container to start, not for the service to accept queries. The dependent service comes up and fails its first requests unless it retries on connect or the dependency declares a healthcheck with condition: service_healthy.

G-05 — Module 12 omitted cost and latency

Thorough on output quality, silent on what a query costs or how long it takes — in a module about whether a system is viable. Both belong in the same evaluation run, because they trade directly against quality.

Extending this

Neither list above is complete.

Take any module and go through it asking which claims are wrong, imprecise, or true only under a condition the text does not state. Three notes from doing it repeatedly:

• Every claim that survives checking is worth more afterwards. The value is not only in the corrections found.
• Disagreements you cannot settle are gaps, not failures. Add them to the course roadmap and resolve them against a real source.
• Check the modules you scored 10/10 on first. A perfect score means your questions sat too close to the text, so the module went unexamined.

Question Bank

One file per module. Ten questions each, generated from the module they test.

How to use

Close the module first. Answering with the text open is re-reading, and re-reading feels like learning while producing almost none. The retrieval attempt is the part that works — including the failed attempts, which is why the scoring below is deliberately harsh.

Answers are collapsed behind <details> blocks. Write your answer down before expanding. Out loud is fine; in your head is not — "I knew that" is unfalsifiable and always feels true.

Scoring

Binary. You got it, or it's a gap.

• "I nearly had it" → gap
• "I knew it but couldn't phrase it" → gap
• "I got the what but not the why" → gap

Every miss is collected in Your gaps. This is the only step nothing else can check, so its value is exactly equal to your honesty about it.

Question types

The why questions matter most. Recall decays and is re-lookupable; knowing which failure a technique prevents is what lets you choose one under pressure.

Coverage

Module 00 has no set — it's orientation, and there's nothing in it to test that later modules don't test better. Module 99 has no set either; it's checked by using it.

Questions — 01 Statistics & Probability

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Define IQR and state which quartiles it uses.

Q2 — recall

At what class balance is entropy maximum for a binary split, and what is it when a set is pure?

Q3 — recall

Write Bayes' theorem and name each term's role.

Q4 — recall

Which hypothesis test for: comparing means of two groups; testing association between two categorical variables; comparing means across four groups?

Q5 — why

Why is standard deviation preferred to variance for reporting, given one is just the square root of the other?

Q6 — why

A 95% confidence interval is [10, 20]. Why is "there's a 95% chance the true value is between 10 and 20" wrong?

Q7 — why

A test is 99% accurate for a disease affecting 1 in 10,000 people. You test positive. Why is your probability of having it still low?

Q8 — why

Why does the central limit theorem matter for methods that assume normality?

Q9 — applied

You're reporting API response times: mean 200ms, median 50ms. What's the shape of the distribution, and which number should the SLA use?

Q10 — applied

A decision tree must pick between two features for its root split. Which measure decides, and what is it computing?

Score: ___/10 → every miss becomes a row in Your gaps.

Questions — 02 Classical ML

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Logistic regression is named for regression but does classification. What makes it a classifier, and what is linear in it?

Q2 — recall

Name the three Naive Bayes variants and what each assumes about features.

Q3 — recall

What are support vectors, and what does an SVM maximise?

Q4 — recall

Fill in: bagging trains models ___ and reduces ___; boosting trains models ___ and reduces ___.

Q5 — why

A single decision tree reaches 100% training accuracy and 62% on test. Why does a random forest fix this?

Q6 — why

Naive Bayes assumes features are independent, which is essentially never true. Why does it still work?

Q7 — why

Why must you scale features for KNN and SVM but not for a decision tree?

Q8 — why

Why is t-SNE unsuitable as a preprocessing step, despite being a dimensionality reduction method?

Q9 — applied

Tabular customer data, 50,000 rows, 30 features, predicting churn. You need a result today and must explain the model to a non-technical stakeholder. What do you run, in what order?

Q10 — applied

You run k-means with k=5 on customer data and get five clean-looking clusters. What have you not established, and how would you check?

Score: ___/10 → every miss becomes a row in Your gaps.

Questions — 03 Evaluation

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Write the formulas for precision and recall, and state what each is "of".

Q2 — recall

What are the three data splits and what is each for?

Q3 — recall

What do Type I and Type II errors correspond to in a confusion matrix?

Q4 — recall

What are the two axes of a ROC curve? Be precise about the x-axis.

Q5 — why

F1 is the harmonic mean, not the arithmetic mean. Why does that choice matter?

Q6 — why

A model scores 99.9% accuracy on fraud detection. Why might it be worthless, and what should you look at?

Q7 — why

Why is your reported test score fiction if you tuned hyperparameters against the test set?

Q8 — why

Plain R² never decreases when you add a feature — even a column of random numbers. Why, and what do you use instead?

Q9 — applied

You're building a cancer screening classifier. Which metric do you optimise, and what do you accept in exchange?

Q10 — applied

Your 4-class classifier reports macro F1 of 0.82. Two of the four classes are rare. What does the number hide, and what do you run instead?

Score: ___/10 → every miss becomes a row in Your gaps.

Questions — 04 Text → Vectors

Close the module before starting. Write answers down. Score binary.

Q1 — recall

List the five rungs of the representation ladder in order.

Q2 — recall

What do TF and IDF each measure, and what does a high product mean?

Q3 — recall

Name the four standard cleaning steps for classical text methods.

Q4 — recall

What are the two main limitations of static word embeddings like Word2Vec?

Q5 — why

Why does one-hot encoding make "cat" as similar to "dog" as to "bureaucracy"?

Q6 — why

TF-IDF discards word order entirely. Why is it still a strong baseline for text classification?

Q7 — why

Why is aggressively cleaning text before passing it to BERT a mistake?

Q8 — why

Why cosine similarity rather than Euclidean distance on embeddings, and what must you do to use FAISS's inner-product index for cosine?

Q9 — applied

You must classify 500 support tickets into 5 categories by end of day, on a laptop, with no GPU. What representation and model?

Q10 — applied

Your semantic search returns "I hate this product" as a top match for "I love this product". Explain and say what to change.

Score: ___/10 → every miss becomes a row in Your gaps.

Questions — 05 Deep Learning

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the four steps of one training iteration, in order.

Q2 — recall

What are the batch sizes for batch, stochastic, and mini-batch gradient descent?

Q3 — recall

List the four layer types in a CNN and what each does.

Q4 — recall

What is the target output of an autoencoder?

Q5 — why

Why does a network need a non-linear activation function? What happens without one?

Q6 — why

You increase a network from 5 layers to 50, and the early layers stop changing. What is happening and why does ReLU help?

Q7 — why

Gradient clipping fixes exploding gradients but not vanishing ones. Why can you derive that from what clipping does?

Q8 — why

Why can't a feedforward network predict the next word in a sentence?

Q9 — applied

You have 224×224 colour images and someone proposes a fully connected first layer of 1,000 neurons. Why is a CNN better, in terms of parameter count and the assumption each makes?

Q10 — applied

You need to detect faulty machines from sensor data. You have millions of normal readings and almost no labelled faults. What architecture fits, and why?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 06 Frameworks

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the five lines of a PyTorch training step, in order.

Q2 — recall

What is the TensorFlow equivalent of PyTorch's autograd, and how is it invoked?

Q3 — recall

What three methods must a PyTorch Dataset subclass implement?

Q4 — recall

Which loss for: multi-class with integer labels in PyTorch; binary classification in PyTorch; regression robust to outliers?

Q5 — why

Why does omitting optimizer.zero_grad() break training, and why does PyTorch accumulate gradients by default at all?

Q6 — why

Why should the final layer of a PyTorch classifier output raw logits with no softmax?

Q7 — why

What does model.eval() change, and what goes wrong if you forget it?

Q8 — why

PyTorch uses a dynamic graph, TF 1.x used a static one. What did each buy?

Q9 — applied

Your Keras model throws a shape error on the loss. Labels are one-hot; loss is SparseCategoricalCrossentropy. What's wrong?

Q10 — applied

You're prototyping a novel architecture with data-dependent control flow, then deploying to mobile. What do you use for each phase?

Score: ___/10 → every miss becomes a row in Your gaps.

Questions — 07 Transformers

Close the module before starting. Write answers down. Score binary.

Q1 — recall

What do Q, K and V stand for, and what is each token asking or offering?

Q2 — recall

Name the four sub-layers of an encoder block in order.

Q3 — recall

How does masked attention prevent a decoder seeing future tokens?

Q4 — recall

In cross-attention, where do Q, K and V come from?

Q5 — why

Naive dot-product attention has two defects that Q/K/V matrices fix. Name both.

Q6 — why

Why divide attention scores by √d_k? What breaks without it?

Q7 — why

Why is positional encoding necessary at all, and why sinusoids rather than integers 1, 2, 3…?

Q8 — why

Why does the encoder need a feed-forward network after attention, given attention already mixes information?

Q9 — applied

Transformer decoder training is parallel but generation is sequential. Explain why both are true, and what makes the parallel version legitimate.

Q10 — applied

You need to classify 50,000 support tickets and have 2,000 labelled examples. Why does the transformer's two-phase training make this feasible?

Score: ___/10 → every miss becomes a row in Your gaps.

Questions — 08 Projects

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Write the pipeline for the LSTM email classifier, from raw email to predicted class.

Q2 — recall

What does oov_token='<OOV>' do in a Keras Tokenizer, and why bother?

Q3 — recall

What two things must be true for FAISS IndexFlatIP to give cosine similarity?

Q4 — recall

What do chunk_size and chunk_overlap control, and what does overlap prevent?

Q5 — why

Why is Bidirectional LSTM fine for email classification but not for text generation?

Q6 — why

The Ticket Delay project retrieves similar examples and then classifies. Explain the bug and why it fails silently.

Q7 — why

Why must the vector index be built at application startup rather than inside the request handler?

Q8 — why

For phishing detection, why might stripping URLs be actively harmful?

Q9 — applied

Three-class ticket classification, 5,000 labelled rows, stable categories, high request volume. RAG-plus-LLM or a trained classifier? Justify on three axes.

Q10 — applied

Projects 1 and 2 report accuracy and classification reports. Project 3 returns a label from an LLM. How do you evaluate project 3, and why is it harder?

Score: ___/10 → every miss becomes a row in Your gaps.

Questions — 09 Scale & Ops

Close the module before starting. Write answers down. Score binary.

Q1 — recall

What is the entry point to a PySpark application, and which abstraction should you prefer for structured data?

Q2 — recall

Name three Spark transformations and three actions.

Q3 — recall

What must every MLlib pipeline do before fitting, and with what class?

Q4 — recall

Name MLflow's three logging primitives and what each records.

Q5 — why

Why does a filter over ten billion rows return instantly while the following show() takes minutes?

Q6 — why

Why is collect() on a large DataFrame the classic Spark failure?

Q7 — why

Why is "we might need to scale later" a bad reason to start with Spark?

Q8 — why

mlflow.log_model ran successfully. Why does that not mean you have a registered model?

Q9 — applied

Your fraud model scored 0.94 F1 at deployment. Six months on, no errors in the logs and no alerts. Why might it be failing, and what would you have needed to detect it?

Q10 — applied

You've run twenty variants of a model over two weeks and one scored 0.91. You didn't track anything. What have you lost, and what's the minimum you'd log next time?

Score: ___/10 → every miss becomes a row in Your gaps.

Modules 10 onward extend this material; each names its prerequisites at the top.

Questions — 10 GRU & the RNN Family

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name GRU's two gates and what each controls.

Q2 — recall

Write GRU's final state update line, and say what happens when z ≈ 0.

Q3 — recall

Fill the row: gates, states carried, and weight matrices for vanilla RNN, LSTM, GRU.

Q4 — recall

What does nn.GRU return in PyTorch, and how does that differ from nn.LSTM?

Q5 — why

Why does a vanilla RNN lose gradient information over 50 steps, in terms of what backpropagation actually computes?

Q6 — why

Explain how GRU's update rule creates a gradient path, and name the transformer mechanism that uses the same trick.

Q7 — why

GRU couples the forget and input decisions that LSTM keeps separate. What can LSTM do that GRU structurally cannot?

Q8 — why

Transformers beat RNNs on training speed. Why doesn't that argument settle the question at inference time?

Q9 — applied

Keyword spotting on a microcontroller: continuous audio, no defined end, a few hundred KB of memory. Why is a GRU a better fit than a transformer here?

Q10 — applied

You swap LSTM(64) for GRU(64) in Keras. Accuracy is unchanged but an epoch now takes 15× longer. What do you check?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 11 Docker for ML

Close the module before starting. Write answers down. Score binary.

Q1 — recall

What is the difference between an image and a container?

Q2 — recall

In a Dockerfile, which is copied first — requirements.txt or your application code? Why that order?

Q3 — recall

In Compose, what does depends_on guarantee — and what does it not?

Q4 — recall

What flag gives a container GPU access, and what host-side component does it require?

Q5 — why

Your container's logs show uvicorn started cleanly, but every request from the host is refused. What is almost certainly wrong, and why is it silent?

Q6 — why

Deleting a secret file in a later Dockerfile layer doesn't protect it. Why not?

Q7 — why

Baking model weights into the image makes it large and forces a rebuild per model update. Why is it still the right default for production serving?

Q8 — why

Why does an unpinned requirements.txt defeat the purpose of containerising?

Q9 — applied

Your CPU-only inference image is uncomfortably large. What is likely the single biggest contributor, and what is the fix?

Q10 — applied

Your training container dies with exit code 137 and no Python traceback. What happened, and why is there no stack trace?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 12 Evaluating LLM & Gen-AI Systems

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the four ranking metrics used for retrieval, and what each asks.

Q2 — recall

Name the three families of generated-text metrics, in ascending order of cost.

Q3 — recall

List the four RAG evaluation dimensions and the failure each catches.

Q4 — recall

Beyond output quality, name the two dimensions that decide whether a gen-AI system is viable, and say what they trade against.

Q5 — why

Why does ROUGE score a correct answer badly when it's worded differently from the reference?

Q6 — why

Why must a RAG system be measured at both stages rather than end-to-end only?

Q7 — why

Faithfulness can be scored without a reference answer. Why does that matter practically?

Q8 — why

Why is "is every claim supported by the context: yes/no" a better judge prompt than "rate quality 1–10"?

Q9 — applied

Your RAG assistant returns confident answers containing facts absent from your documents. Which dimension has failed, and what do you check first?

Q10 — applied

You're asked to evaluate a system that classifies tickets into three fixed labels using an LLM. What do you build, and which metrics apply?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 13 RAG, Properly

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the four chunking strategies and when each fits.

Q2 — recall

What does each of dense and sparse retrieval find, and what does each miss?

Q3 — recall

Write the Reciprocal Rank Fusion formula and name its inputs.

Q4 — recall

What is the standard two-stage retrieval pattern, with rough numbers?

Q5 — why

Why can a cross-encoder not replace the bi-encoder for the retrieval stage?

Q6 — why

Your RAG assistant surfaces a confidential document to a user who shouldn't see it. Why do generation-layer safeguards not help here?

Q7 — why

Retrieved examples arrived without their labels. Why couldn't a better prompt have fixed that?

Q8 — why

Why does omitting a refusal instruction produce hallucination specifically in RAG?

Q9 — applied

Your RAG support bot answers conceptual questions well but fails whenever a user pastes an error code. Diagnose and fix.

Q10 — applied

You have a week to improve a working-but-mediocre RAG system. What order do you work in, and why that order?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 14 Vector Databases

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the four index families and the trade each makes.

Q2 — recall

Which two knobs tune the recall/latency trade at query time, and for which index?

Q3 — recall

Write the formula for raw vector memory, and give the rough figure for 1M 768-dimension vectors.

Q4 — recall

Name the three levers for reducing vector memory cost.

Q5 — why

An ANN index is described as a speed optimisation. What is the more honest description, and how would you quantify what you gave up?

Q6 — why

Why can post-filtering return zero results from a healthy index, and when is it worst?

Q7 — why

Why must chunk IDs be derived deterministically from the original document?

Q8 — why

Why does swapping the embedding model make an existing index meaningless rather than merely worse?

Q9 — applied

Your team runs Postgres and needs semantic search over ~200k documents. Make the case for pgvector over a dedicated vector store.

Q10 — applied

Retrieval quality "suddenly collapsed" last week. Nothing in the RAG code changed. What do you investigate, in order?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 15 Kubernetes for ML

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the three probe types, the question each asks, and what failing each does.

Q2 — recall

What is the difference between requests and limits?

Q3 — recall

What happens when a container exceeds its memory limit? Its CPU limit?

Q4 — recall

What are the four ML-specific facts that generate nearly all Kubernetes pain?

Q5 — why

A pod restarts every ninety seconds with no error in its logs. Diagnose it and explain why the logs are clean.

Q6 — why

Why should liveness "test almost nothing"?

Q7 — why

Why is CPU-based autoscaling close to useless for GPU inference?

Q8 — why

Why does autoscaling save less money for model serving than the graphs suggest?

Q9 — applied

You have four models, each on its own GPU pod, all averaging under 10% GPU utilisation. What is wrong and what are your options?

Q10 — applied

You're deploying a self-hosted vector store to the cluster. What are the three things to get right, and which one do people discover too late?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 16 Monitoring, Drift & Retraining

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the four monitoring layers and say which need labels.

Q2 — recall

Distinguish data drift from concept drift.

Q3 — recall

Name three measures for quantifying distribution shift.

Q4 — recall

What must be logged with every prediction for outcome monitoring to be possible later?

Q5 — why

Why is "software fails loudly, models fail silently" the framing that makes this its own discipline?

Q6 — why

Why do production labels flatter the model, and what's the fix?

Q7 — why

Why must the evaluation set stay fixed when retraining on a rolling window?

Q8 — why

Why can you sample prediction logs without losing drift detection?

Q9 — applied

Predicted-fraud rate jumped from 2% to 11% overnight. Input distributions are unchanged. What do you suspect, and why not "more fraud"?

Q10 — applied

You're asked to set up automatic retraining that deploys without human review. What do you insist on before agreeing?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 17 Tokenization

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Write the four steps of BPE training, and say what the resulting tokenizer actually is.

Q2 — recall

What does byte-level BPE operate on, how large is its base vocabulary, and what does that guarantee?

Q3 — recall

How does WordPiece's merge criterion differ from BPE's, and what notation does it use?

Q4 — recall

What is SentencePiece, and what does it contribute beyond the merge algorithm?

Q5 — why

Character-level tokenization eliminates the OOV problem entirely. Why isn't it used?

Q6 — why

Why can't a model reliably count the letters in "strawberry"?

Q7 — why

Why does the same message cost more in some languages than others, and why is that more than a billing detail?

Q8 — why

Why is replacing a pre-trained model's tokenizer almost never worth it?

Q9 — applied

A colleague loads GPT-2 weights with BERT's tokenizer. There's no error and the output is fluent nonsense. Explain, and name the analogous failure elsewhere in this tutorial.

Q10 — applied

Your RAG system retrieves good context, but answers ignore material you can see in the retrieved chunks. Tokenization is the cause. What happened?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 18 Feature Engineering & Data Leakage

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Define leakage in one sentence, and say why the offline score isn't "wrong".

Q2 — recall

Name the five kinds of leakage.

Q3 — recall

Write the timing test applied to every candidate feature — both clauses.

Q4 — recall

How do you encode "hour of day" so 23 and 0 are adjacent, and why two components?

Q5 — why

Why is target encoding uniquely dangerous among categorical encodings?

Q6 — why

Why is a random train/test split wrong for time-ordered data, even though it's statistically unbiased?

Q7 — why

Why is a Pipeline a better answer to contamination than remembering to fit on train only?

Q8 — why

Why doesn't dropping a protected attribute remove it from the model?

Q9 — applied

Your churn model hits AUC 0.98 — far above anything the team has managed. What do you do first, and what do you look for?

Q10 — applied

A model validated cleanly, deployed, and degraded within a week. Monitoring shows no input drift. Name two causes from this module and how you'd distinguish them.

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 19 Prompting vs RAG vs Fine-tuning

Close the module before starting. Write answers down. Score binary.

Q1 — recall

What part of the system does each of the three techniques change?

Q2 — recall

Give the one-sentence rule that separates RAG from fine-tuning.

Q3 — recall

Name four situations where fine-tuning is genuinely the right call.

Q4 — recall

What does LoRA do differently from full fine-tuning, and why does it matter?

Q5 — why

Why does fine-tuning to add facts appear to work and then fail?

Q6 — why

Why is "do not skip to fine-tuning" a rule rather than a preference?

Q7 — why

Why can the data-permanence question decide the architecture before any capability argument?

Q8 — why

Why does fine-tuning win at high request volume and lose at low volume?

Q9 — applied

An internal assistant must answer from company policy documents that change monthly, and cite its sources. Which technique, and why are the other two wrong?

Q10 — applied

Three-class ticket classification, 5,000 labelled rows, stable categories, high volume. Walk the decision — and name the option the framing tends to hide.

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 20 Hyperparameter Tuning & Validation

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Distinguish parameters from hyperparameters, and say why the distinction changes the machinery you need.

Q2 — recall

Name the four cross-validation variants and when each applies.

Q3 — recall

What do the inner and outer loops of nested CV each do?

Q4 — recall

For a neural network and for gradient-boosted trees, what do you tune first?

Q5 — why

Why is the best CV score from a 200-configuration search biased upward?

Q6 — why

Explain why random search beats grid search on the same budget.

Q7 — why

When does Hyperband make sense, and what property must the task have?

Q8 — why

Why does leakage make tuning actively harmful rather than merely unhelpful?

Q9 — applied

Compute the training runs for 50 configurations, 5-fold CV, nested with 5 outer folds — and say what you'd do instead.

Q10 — applied

Your imbalanced binary classifier shows CV scores swinging from 0.71 to 0.94 across folds. What's the likely cause and the fix?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 21 Attention Efficiency & Context Length

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Doubling the context multiplies attention compute by how much, and why?

Q2 — recall

What does the KV cache hold, what does it scale with, and why does it exist?

Q3 — recall

What is FlashAttention, and what is it not?

Q4 — recall

Distinguish MHA, MQA and GQA.

Q5 — why

Why is attention described as memory-bandwidth bound rather than compute bound, and what follows from that?

Q6 — why

Why does the KV cache, not the weights, usually cap inference batch size?

Q7 — why

Why doesn't a long context window substitute for good retrieval?

Q8 — why

Why does continuous batching help more than a uniform benchmark suggests?

Q9 — applied

Your inference server OOMs only under long-context traffic; the model fits fine otherwise. Name three levers, cheapest first.

Q10 — applied

A 500-page manual, queried thousands of times a day. Long context or retrieval — and what changes the answer?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 22 Transformer Variants

Close the module before starting. Write answers down. Score binary.

Q1 — recall

For each of the three shapes, say what a token can see.

Q2 — recall

What pre-training objective goes with each shape?

Q3 — recall

Name the three axes conflated under the word "variant".

Q4 — recall

Which shape is the retrieval half of a RAG system, and which is the generation half?

Q5 — why

Why does masked language modelling require the absence of a causal mask?

Q6 — why

Why is an encoder usually the better classifier at a fixed model size?

Q7 — why

Why can't an encoder-only model generate text?

Q8 — why

Why is comparing parameter counts across shapes misleading?

Q9 — applied

A team prompts a large decoder-only model to sort support tickets into five fixed categories, at high volume. Diagnose the cost, and say what you'd propose.

Q10 — applied

You plan to fine-tune on a narrow task. Do you start from the base or the instruct model, and why?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 23 Serving Models

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Give three reasons FastAPI is the better default over Flask for a model service.

Q2 — recall

State the rule for async def versus plain def in a FastAPI handler.

Q3 — recall

Name the three components of a request's cost.

Q4 — recall

What does dynamic batching do, and what is the trade?

Q5 — why

Why is async def around a blocking model call worse than plain synchronous code?

Q6 — why

Why isn't worker count a free throughput dial?

Q7 — why

Why does GPU batching raise throughput by nearly the batch size rather than a little?

Q8 — why

Why does the batching window need to come from your p99 target rather than intuition?

Q9 — applied

Under load, your service's latency collapses and Kubernetes starts restarting the pod, though nothing errors. What do you check first?

Q10 — applied

When would you move from a hand-written FastAPI service to a dedicated model server?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 24 Orchestration Frameworks

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the five things an orchestration framework actually gives you.

Q2 — recall

What distinguishes a chain from a graph?

Q3 — recall

What does explicit state buy you in a graph framework?

Q4 — recall

Name two guards every agent loop needs.

Q5 — why

Why is observability the most under-appreciated feature and hardest to retrofit?

Q6 — why

Why does expressing dynamic control flow as a chain produce worse code than using a graph?

Q7 — why

Why are agents hard to evaluate with module 12's methods?

Q8 — why

Why does state in process memory work in the prototype and fail in production?

Q9 — applied

Retrieve → format prompt → call model, one provider, no plans to change. Framework or plain code? Justify.

Q10 — applied

You're comparing two frameworks. What's the single most important question, and why that one?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Questions — 25 Responsible AI

Close the module before starting. Write answers down. Score binary.

Q1 — recall

Name the four ways bias enters through data.

Q2 — recall

Name the four formal fairness criteria and what each requires.

Q3 — recall

List the four privacy points collected from earlier modules.

Q4 — recall

What is the single highest-value practice in this module?

Q5 — why

Why is "the model works exactly as specified" the problem with historical bias rather than a defence?

Q6 — why

Why can't you simply enable a "fair" setting?

Q7 — why

Why does prompt injection have no complete fix?

Q8 — why

Why does treating this as an end-of-project compliance review fail?

Q9 — applied

Your classifier reports 90% accuracy and the team is satisfied. What do you run, and what might you find?

Q10 — applied

A loan model must be explainable to rejected applicants and support erasure requests. How do those two constraints shape the architecture?

Score: ___/10 → every miss becomes a row in Your gaps.

Scored 10/10? Go back to the module and hunt for a claim you cannot justify — a perfect score usually means the questions sat too close to the text, not that the module is exhausted.

Your gaps