Week 02 · Page 2 of 4 · Directed Work (Travaux Dirigés)
Anyone can compute a training error. The skill this week trains is computing an honest one.
Attempt each problem yourself before expanding the solution.
Two models were trained on the same dataset of 60 compounds to predict bulk modulus. Their errors are reported below.
| Model | Training error (MAE, GPa) | Cross-validated error (MAE, GPa) |
|---|---|---|
| Model 1 (linear) | 18.2 | 19.1 |
| Model 2 (deep neural network) | 0.3 | 24.6 |
(a) Which model shows symptoms of high bias, and which shows symptoms of high variance? (b) Which model would you trust more for predicting a new, sixth compound, and why? (c) Is Model 2's training error of 0.3 GPa, by itself, evidence that Model 2 is the better model?
Show solution
(a) Model 1 shows the signature of high bias: its training error (18.2) and cross-validated error (19.1) are close together, meaning it performs almost as poorly on data it has seen as on data it has not — consistent with a model too simple to capture the real relationship. Model 2 shows the signature of high variance: a near-perfect training error (0.3) collapses to a much worse cross-validated error (24.6), the classic sign of a model that memorized its training compounds rather than learning a generalizable relationship.
(b) Model 1 is more trustworthy here, despite its higher training error, because its cross-validated error (the honest estimate of performance on new compounds) is lower than Model 2's: 19.1 versus 24.6.
(c) No. Training error alone says nothing about generalization, which is the entire point of this week's lesson. A training error of 0.3 GPa is consistent with both an excellent model and a badly overfit one; only the cross-validated number distinguishes the two, and here it reveals Model 2 to be the worse choice.
You have 8 compounds (labeled 1–8) and choose k = 4, so each fold contains 2 compounds. The resulting per-round errors (MAE, eV/atom) are: Round 1 (fold = compounds 1,2): 0.05. Round 2 (fold = compounds 3,4): 0.11. Round 3 (fold = compounds 5,6): 0.06. Round 4 (fold = compounds 7,8): 0.08.
(a) Compute the cross-validated error estimate. (b) Round 2's error is noticeably higher than the other three. Name two possible explanations consistent with what this week's lesson covered. (c) Would simply discarding Round 2's result and averaging only the other three rounds be good practice? Why or why not?
Show solution
(a) (0.05 + 0.11 + 0.06 + 0.08) / 4 = 0.075 eV/atom.
(b) Two consistent explanations: (i) compounds 3 and 4 may be genuinely harder to predict — perhaps they lie in a region of feature space poorly represented by the other six training compounds, which is a real signal worth investigating; (ii) pure chance variation from having only 2 compounds in this fold, where a single unusual compound can swing the average sharply, since the fold size is so small.
(c) No. Discarding an inconvenient result without a principled reason — sometimes called "cherry-picking" — would bias the reported error to look better than the model's real performance. If compounds 3 and 4 are suspected of being unusual, the correct response is to investigate why (perhaps they belong to a different structural family the features do not capture well), not to remove them from the average after seeing that they hurt the score.
A student has a dataset of 40 entries: 10 parent compounds, each represented at 4 different strain levels (0%, 1%, 2%, 3% lattice strain), giving 40 total rows. The student performs a standard random 5-fold split across all 40 rows and reports an excellent cross-validated error of 0.02 eV/atom for predicting formation energy from strain level and composition.
(a) Explain what is likely wrong with this validation procedure. (b) Describe the correct way to split this specific dataset. (c) Would you expect the corrected validation error to be higher, lower, or the same as 0.02 eV/atom, and why?
Show solution
(a) A random split across all 40 rows will very likely place different strain levels of the same parent compound into both the training and test folds. Since the four strain variants of one parent compound are extremely similar to each other, the model can effectively memorize "this parent compound has formation energy near Y" from the training rows and apply that memorized value almost unchanged to the test rows of the same parent — without having learned anything that would generalize to a genuinely new, unseen compound. This is the data leakage problem described in the Lesson.
(b) The split must be performed by parent compound, not by row: all 4 strain variants of a given parent compound must be placed entirely in either the training set or the test set, never split between them. With 10 parent compounds and a 5-fold split, each fold would hold out 2 complete parent compounds (all 4 strain rows each).
(c) Higher (worse). The reported 0.02 eV/atom almost certainly reflects leaked information from near-duplicate strain variants, not genuine generalization to new compounds. A correctly grouped split removes this shortcut, and the resulting honest error is very likely to be larger — though only the corrected experiment would reveal by how much.
If Problem 3 felt obvious in hindsight but you would not have caught it in your own data before this lesson, that is a normal and useful reaction — data leakage from grouped or near-duplicate samples is one of the most common silent errors in published materials-ML work, precisely because it makes results look better, not worse, so it often goes unquestioned.
My name is Dr Abderrahmane Reggad. I'm a 54 year old and I am a researcher in materials' sciences.
0 Comments