Week 02 · Page 4 of 4 · Self-Check Quiz

Ten questions. If you miss more than two, revisit the Lesson before Week 3 — regression, the first method you will build there, is only meaningful if you can already judge whether it generalizes.

Quiz Self-Check 10 Questions

Answer each question yourself before revealing the answer.

Question 1

A model achieves near-zero error on its training data but a much larger error under cross-validation. This is the signature of:

  1. High bias (underfitting)
  2. High variance (overfitting)
  3. Perfect generalization
  4. Data leakage, always
Show answer
Correct answer: B. A large gap between excellent training error and much worse validation error is the defining signature of high variance (overfitting): the model fit the specific training examples, including their noise, rather than a relationship that holds on new data. Data leakage (D) could in principle make even validation error look artificially good, which is the opposite direction from what's described here.
Question 2

A linear model has training error 12.0 and cross-validated error 12.8 (same units). What does this pattern most directly suggest?

  1. The model is severely overfit
  2. The model may be too simple to capture the true relationship (high bias)
  3. The model has perfectly memorized the training data
  4. Nothing can be concluded from these two numbers
Show answer
Correct answer: B. Training and validation errors that are close together, but both relatively high, are consistent with high bias: the model is not flexible enough to fit even its own training data well, so performance on new data is similarly limited rather than dramatically worse.
Question 3

What is the defining feature of a train/test split?

  1. The test set is used to adjust the model's parameters during training
  2. The test set is held out and used only to evaluate a model already trained on the training set
  3. The training and test sets must always be the same size
  4. Train/test splitting is only used for unsupervised learning
Show answer
Correct answer: B. The essential property of a test set is that it is never used to fit the model's parameters — it exists solely to provide an honest estimate of performance on unseen data, after training is complete.
Question 4

In 5-fold cross-validation, how many times is each individual data point used for testing, and how many times for training?

  1. 5 times for testing, 5 times for training
  2. 1 time for testing, 4 times for training
  3. 1 time for testing, 1 time for training
  4. 5 times for testing, 0 times for training
Show answer
Correct answer: B. With k=5, each point falls into exactly one fold, which is used as the test set in exactly one of the five rounds, and as part of the training set in the other four rounds.
Question 5

Why is the average of the k fold errors generally preferred over a single train/test split?

  1. It is always faster to compute
  2. It reduces sensitivity to which particular examples happened to land in the test set
  3. It guarantees the model has no bias
  4. It eliminates the need for a test set entirely
Show answer
Correct answer: B. A single split's reported error depends heavily on which specific examples were held out, especially with small datasets. Averaging over k rotations reduces this sensitivity and gives a more stable estimate of true generalization performance.
Question 6

A materials dataset contains 5 strain variants for each of 20 parent compounds (100 rows total). A student performs a standard random 5-fold split across all 100 rows. What is the main risk?

  1. The model will definitely underfit
  2. Strain variants of the same parent compound may be split across training and test folds, causing data leakage
  3. The model cannot be trained at all with this much data
  4. There is no risk; random splitting is always correct
Show answer
Correct answer: B. Since strain variants of the same parent are very similar, a random row-level split is likely to place some variants of a given parent in training and others in the test fold, letting the model effectively 'see' near-duplicates of test examples during training — inflating the apparent validation performance without genuine generalization.
Question 7

What does GroupKFold (or an equivalent group-aware split) guarantee that a standard KFold split does not?

  1. Every fold will have exactly the same number of examples
  2. All examples sharing the same group label are kept entirely within one fold, never split across folds
  3. The model will achieve lower error
  4. Training will run faster
Show answer
Correct answer: B. Group-aware splitting guarantees that no group (e.g. all strain variants of one parent compound) is divided between training and test — directly preventing the leakage scenario described in the Lesson and reproduced in the Practical Work.
Question 8

In the Week 2 Practical Work, why did the 1-nearest-neighbor model show a much larger gap between the naive and group-aware validation errors than linear regression did?

  1. KNN is always a worse model than linear regression
  2. KNN with k=1 can directly copy the label of a near-duplicate training point, so it benefits much more from leaked near-duplicates than a linear model can
  3. Linear regression does not use cross-validation
  4. The two models were evaluated on different datasets
Show answer
Correct answer: B. A 1-nearest-neighbor model predicts by copying its closest training point's label, which makes it especially able to exploit a leaked near-duplicate sitting in the training set. Linear regression fits a single global trend and cannot 'copy' an individual point's label in the same way, so it benefits much less from the same leakage.
Question 9

If you increase k (the number of neighbors) in a k-nearest-neighbors model from 1 to a larger value, what should happen to its sensitivity to the kind of leakage described in this week's lesson, and why?

  1. It should increase, because more neighbors means more memorization
  2. It should decrease, because the prediction is now an average over several points rather than a copy of a single nearest one
  3. It should stay exactly the same regardless of k
  4. k has no relationship to data leakage
Show answer
Correct answer: B. Averaging over more neighbors dilutes the influence of any single leaked near-duplicate, since the prediction becomes a blend of several training points rather than a direct copy of the closest one — consistent with the Practical Work's observed shrinkage of the naive/grouped gap when k increases from 1 to 3.
Question 10

A colleague reports a machine learning model for predicting Curie temperature with a cross-validated error far better than anything previously published, using a dataset where many compounds are closely related solid-solution variants of a few parent systems. What should you check first?

  1. Whether the model used a neural network or a simpler method
  2. Whether the cross-validation splitting was done by parent system/group rather than by individual row, to rule out leakage inflating the reported performance
  3. Whether the paper used Python or another programming language
  4. Whether the training set was large enough in absolute terms
Show answer
Correct answer: B. An unusually strong result on a dataset with many near-duplicate or closely related compounds is exactly the situation in which to suspect data leakage from an inadequate (non-grouped) splitting strategy before accepting the reported performance at face value — the central caution of this week's lesson.
Scoring guide

9–10 correct: you are ready for Week 3. 6–8 correct: revisit the definition boxes for bias, variance, and data leakage on the Lesson page. 5 or fewer: re-read the full Lesson page and redo the Practical Work before starting Week 3, since the regression methods introduced next week will be evaluated using exactly the tools from this week.

← Practical Work Quiz · Page 4 of 4 Week 3 →

INTELLIGENT PHYSICS: MACHINE LEARNING METHODS FOR MATERIALS SCIENCE — WEEK 2, QUIZ