Week 02 · Page 1 of 4 · Lesson
Training error tells you how well a model memorized what you showed it. Only validation tells you whether it learned anything that will hold up on a compound it has never seen.
1. Why training error alone is a trap
In Week 1, you fitted a straight line through four points by hand and used it to predict a fifth. That worked because the underlying relationship really was close to linear. But nothing forced the relationship to be linear — you could, in principle, fit a curve so flexible that it passes through every single training point exactly, leaving zero training error. Would that be a better model?
Almost always, no. A curve flexible enough to hit every training point exactly is usually also fitting the small, meaningless wiggles and noise specific to those particular four or five compounds — noise that will not repeat for a new, sixth compound. The model's training error would be perfect and its real-world usefulness could be terrible. This single observation is the reason this entire week exists.
Generalization
Generalization is a model's ability to make accurate predictions on new examples it was not trained on. Generalization, not training error, is what makes a model scientifically useful, since the entire point of building the model was to predict properties for compounds you have not yet computed or measured.
2. Two ways a model can fail to generalize
A model that generalizes poorly fails in one of two characteristic ways. Physicists will recognize the shape of this trade-off from other contexts in which a single free parameter cannot simultaneously minimize two competing error sources.
Bias (underfitting)
Bias is the error that comes from a model being too simple to capture the real relationship between features and label. A linear model forced onto a clearly curved relationship has high bias: it will fit both the training data and new data about equally poorly, because it is structurally incapable of representing the true pattern.
Variance (overfitting)
Variance is the error that comes from a model being so flexible that it fits the specific noise in one particular training set, rather than the underlying relationship. A high-variance model fits its own training data extremely well but performs much worse on new data, because what it learned was partly an artifact of which particular compounds happened to be in the training set.
| Symptom | High bias (underfitting) | High variance (overfitting) |
|---|---|---|
| Training error | High | Very low |
| Error on new data | High, similar to training error | Much higher than training error |
| Typical cause | Model too simple (e.g. linear fit to a curved relationship) | Model too flexible relative to the amount of training data |
| Materials-science example | Fitting a straight line to a property that depends nonlinearly on composition | A deep neural network trained on 30 compounds, with enough parameters to memorize all 30 |
You cannot read bias or variance off a single training-error number; the same low training error is consistent with a good model or a badly overfit one. Distinguishing the two requires checking performance on data the model did not see during training — which motivates everything in the rest of this lesson.
3. Worked example — building an overfit model on purpose
Take the four-compound dataset from Week 1's Worked Example A: Δχ values 0.5, 1.0, 1.5, 2.0 with formation energies −0.20, −0.55, −0.85, −1.15 eV/atom.
Week 1 fit a line through these four points: E ≈ −0.633·Δχ + 0.12. This line does not pass through any of the four points exactly — there is a small residual error at each one.
With four data points, a cubic polynomial (four free parameters: E = a·Δχ³ + b·Δχ² + c·Δχ + d) has exactly enough flexibility to pass through all four points exactly, with zero training error. Such a curve exists and can be constructed for any four points.
Recall that a fifth compound at Δχ = 2.5 was predicted by the linear model to have formation energy ≈ −1.46 eV/atom. A cubic fit to only four points, by contrast, is essentially unconstrained beyond Δχ = 2.0 — it could swing sharply upward or downward just past the last training point, since nothing in the four points constrains its behavior there. Two researchers fitting cubics to the same four points by slightly different numerical methods could get visibly different predictions at Δχ = 2.5, while any two reasonable linear fits to the same four points would agree closely.
The point of this example: the cubic model has zero training error and is the worse model. Its apparent perfection on the training data is exactly the warning sign Section 2 described — high flexibility relative to a small dataset, with no way to know whether it is right except by checking it against data it did not train on.
4. Train/test splitting
The fix is direct: do not let a model train on every example you have. Hold some examples back, untouched during training, and use them purely to check how well the model performs on data it has genuinely never seen.
Train/test split
A train/test split divides a dataset into two disjoint parts: a training set, used to fit the model's parameters, and a test set, held out and used only to evaluate the trained model's error on examples it never saw during training. The test error, not the training error, is the honest estimate of generalization.
A common split is 80% training, 20% test, though the right proportions depend on how much data you have. With materials datasets often numbering in the dozens to low hundreds, even this simple split has a real cost: every compound moved into the test set is one fewer compound available for training, which matters when data is already scarce.
5. K-fold cross-validation
A single train/test split has a weakness of its own: the particular compounds that happen to land in the test set can make a model look better or worse than it really is, purely by chance, especially with small datasets. K-fold cross-validation addresses this by reusing the data more efficiently.
K-fold cross-validation
K-fold cross-validation divides the dataset into k equally sized parts ("folds"). The model is trained k separate times, each time using k−1 folds for training and the remaining one fold as the test set, rotating which fold is held out each time. The final reported error is the average of the k individual test errors. Every example is used for testing exactly once and for training k−1 times.
Suppose you have 10 compounds, labeled 1 through 10, and choose k = 5. Each fold contains 2 compounds.
| Round | Test fold (held out) | Training set |
|---|---|---|
| 1 | Compounds 1, 2 | Compounds 3–10 |
| 2 | Compounds 3, 4 | Compounds 1, 2, 5–10 |
| 3 | Compounds 5, 6 | Compounds 1–4, 7–10 |
| 4 | Compounds 7, 8 | Compounds 1–6, 9, 10 |
| 5 | Compounds 9, 10 | Compounds 1–8 |
Train five separate models, one per round, each on the 8 training compounds for that round.
Record the prediction error on the 2 held-out compounds for each round: suppose the five round errors (mean absolute error, in eV/atom) are 0.04, 0.07, 0.05, 0.09, 0.06.
Average the five errors: (0.04 + 0.07 + 0.05 + 0.09 + 0.06) / 5 = 0.062 eV/atom. This is the cross-validated error estimate — a more stable number than any single round, because it does not depend on which 2 compounds happened to be held out.
The spread across the five round errors (0.04 to 0.09) is itself informative: a wide spread suggests the model's performance depends heavily on which compounds it happened to see, which is itself a mild symptom of variance.
6. The data-leakage trap specific to materials datasets
Cross-validation assumes the folds are genuinely independent of each other. Materials datasets frequently violate this assumption in a way that is easy to miss.
Data leakage
Data leakage occurs when information from the test set indirectly influences training, making validation error look better than it really is. In materials datasets, a common cause is having multiple near-duplicate entries derived from the same parent compound — different strain states, different dopant levels, different polymorphs — which can be split so that near-identical entries land in both the training and test folds.
If a model effectively memorizes "compounds that look like X have formation energy near −0.85," and a near-duplicate of X is in the test fold purely by chance, the model will appear to generalize well — when in fact it has only memorized X. The fix is group-aware splitting: assign every fold by parent compound or composition group, never by individual entry, so that no near-duplicate of a training example ever appears in the test set.
What this means for the rest of the course
From this week forward, every reported model performance in this course will be a cross-validated or held-out test error, never a training error. When you see an accuracy or error number reported anywhere — in a paper, in this course, or in your own future work — the first question to ask is always: was this measured on data the model was trained on, or held out from it?
Next on this week's pages
Proceed to the Directed Work page, where you will compute a cross-validation estimate by hand on a new dataset and diagnose a data-leakage scenario.
My name is Dr Abderrahmane Reggad. I'm a 54 year old and I am a researcher in materials' sciences.
0 Comments