Week 02 · Page 3 of 4 · Practical Work (Travaux Pratiques)

The goal of this practical is not to write clever code. It is to see, with your own numbers, the gap between an optimistic validation score and an honest one.

Practical Work First Code scikit-learn

Setup

You will need Python 3 with the following packages: numpy, pandas, and scikit-learn. If you do not have these installed, run:

pip install numpy pandas scikit-learn

Save the dataset below as week2_data.csv in the same folder as your script. This is a synthetic dataset constructed for this exercise: 12 parent compounds, each appearing at 2 strain levels (24 rows total). Each parent has its own small composition-independent shift in formation energy — a stand-in for chemistry the single feature does not fully capture — while the two strain rows of a given parent are nearly identical to each other.

parent_id,strain_pct,feature_dchi,formation_energy 1,0,0.501,-0.1946 1,2,0.511,-0.1751 2,0,0.631,-0.3494 2,2,0.652,-0.3607 3,0,0.78,-0.3187 3,2,0.793,-0.3247 4,0,0.906,-0.2659 4,2,0.923,-0.2572 5,0,1.042,-0.6153 5,2,1.062,-0.5879 6,0,1.182,-0.7048 6,2,1.206,-0.7004 7,0,1.319,-0.5236 7,2,1.332,-0.4961 8,0,1.458,-0.7059 8,2,1.474,-0.7046 9,0,1.584,-0.9822 9,2,1.609,-0.9584 10,0,1.729,-0.9226 10,2,1.749,-0.9028 11,0,1.86,-1.1316 11,2,1.889,-1.1224 12,0,1.996,-1.223 12,2,2.022,-1.2041
1 Load the data and evaluate a 1-nearest-neighbor model with a naive random split

We deliberately use a 1-nearest-neighbor model for this exercise rather than linear regression. A 1-nearest-neighbor model predicts a new point's label by copying the label of the single closest training point — which makes it the easiest possible model to "cheat" with if a near-duplicate happens to leak into the training set. Create a file week2_practical.py:

import pandas as pd import numpy as np from sklearn.neighbors import KNeighborsRegressor from sklearn.model_selection import KFold, GroupKFold, cross_val_score data = pd.read_csv("week2_data.csv") X = data[["feature_dchi"]].values y = data["formation_energy"].values groups = data["parent_id"].values model = KNeighborsRegressor(n_neighbors=1) # --- Naive random 4-fold split (ignores parent_id grouping) --- naive_cv = KFold(n_splits=4, shuffle=True, random_state=0) naive_scores = cross_val_score(model, X, y, cv=naive_cv, scoring="neg_mean_absolute_error") print("Naive random split, fold MAEs:", -naive_scores) print("Naive random split, mean MAE:", -naive_scores.mean())

Run this script (python week2_practical.py) and record the mean MAE it prints. You should see a small number — roughly 0.04 eV/atom.

2 Repeat with a group-aware split

Add the following to the same script, below what you already wrote:

# --- Group-aware 4-fold split (respects parent_id grouping) --- group_cv = GroupKFold(n_splits=4) group_scores = cross_val_score(model, X, y, cv=group_cv, groups=groups, scoring="neg_mean_absolute_error") print("Group-aware split, fold MAEs:", -group_scores) print("Group-aware split, mean MAE:", -group_scores.mean())

Run the script again and record this second mean MAE. You should see a substantially larger number than in Step 1 — roughly 0.16 eV/atom, around four times worse.

3 Interpret the gap

You have just reproduced, with real code and real numbers, the leakage scenario from Directed Work Problem 3. With the naive split, a strain variant of a parent compound frequently ends up in the test fold while its near-identical sibling sits in the training fold; the 1-nearest-neighbor model then "predicts" the test point by copying its sibling's almost-identical label, producing an artificially low error. The group-aware split removes this shortcut entirely — every parent compound is fully on one side of the split or the other — and the reported error rises sharply to reflect genuine generalization to unseen parent compounds.

Now repeat both evaluations using from sklearn.linear_model import LinearRegression in place of KNeighborsRegressor(n_neighbors=1), keeping everything else the same. You should find the naive-vs-grouped gap is much smaller for linear regression. Write one sentence explaining why a model's susceptibility to this kind of leakage depends on the model, not only on the data.

What to take away

The size of a leakage effect depends on how easily a model can "memorize" near-duplicates. Highly flexible, memorization-prone models (nearest-neighbor methods, deep networks, large ensembles) are far more exposed to grouped data leakage than simple models like linear regression — which is one more reason the bias/variance framing from the Lesson and the validation strategy you choose are not independent decisions.

Optional extension

If you want to go further: increase n_neighbors from 1 to 3 and rerun both splits. Does the gap between the naive and group-aware error shrink, grow, or stay the same? Try to predict the answer before running the code, based on what averaging over more neighbors should do to a model's ability to "copy" a single leaked near-duplicate.

← Directed Work Practical Work · Page 3 of 4 Quiz →

INTELLIGENT PHYSICS: MACHINE LEARNING METHODS FOR MATERIALS SCIENCE — WEEK 2, PRACTICAL WORK