MY202 · Lab 3
Jonas Weinert
Department of Methodology | London School of Economics and Political Science
29 October 2026
| Part | Topic | Functions | Mode | Time |
|---|---|---|---|---|
| Setup: notebook lines 24–72 | together | 2 min | ||
| Concept: warm-up, then our first algorithm | ~, linear_reg(), fit() |
slides | 15 min | |
| 1 | Exploring the data | ggplot, geom_histogram, geom_boxplot |
your turn | 12 min |
| 2 | Hypothesis testing | extract_fit_engine, summary, ~ . |
together | 8 min |
| 3 | The machine learning approach | initial_split, training, testing, augment, metric_set |
together | 18 min |
| 4 | Where do we make errors? | .pred, .resid, geom_hline |
together | 8 min |
| 5 | Lasso | penalty, mixture, set_engine, tidy |
step 1 together, then your turn | 22 min |
Together: we type the code line by line, the slide explains each piece, and a notebook line number says where to type. Your turn: a timer, tasks and a syntax sheet, then the solutions.
Concept
Three warm-up questions · what a model is · how a line is chosen · how we judge it
A rule predicts a grade as 10 + 0.5 × hours of study. A student studies 4 hours a week. What grade does the rule predict?
How does linear regression choose which line to draw through the points?
A house-price model has R² = 0.65. What does that mean?
A model is a rule that turns inputs into a prediction. Made up for now: a grade from study hours and past failures.
Where do 10, 0.5 and −2 come from? The numbers in the rule are its coefficients. Someone has to choose them.
An algorithm is a recipe for choosing them from data. Today’s: ordinary least squares, built from scratch in the next five slides.
200 markets: what was spent on TV ads, and how much was sold. We want a line that predicts sales from a budget.
Vote: A, B or C?
The harder question Most pick B. But why? A computer can’t choose a line without a number that says how wrong a line is.
Residual = actual − predicted Positive: the line guessed too low. Negative: too high.
200 markets, 200 residuals How do we turn 200 numbers into one score for the line?
Just add the residuals
A flat line at the average is clearly bad, yet its residuals add up to exactly 0: positives and negatives cancel.
Square first, then add
Flat line: 5,417. Line B: 2,103. Lower is better, and one miss of 4 now costs more than two misses of 2.
Ordinary least squares (OLS) The line with the smallest sum of squared residuals. That’s the whole algorithm.
You don’t have to try every line For linear models a formula jumps straight to the bottom of the bowl. Later models really do walk downhill.
ŷ = β₀ + β₁ · x
ŷ = β₀ + β₁ x₁ + β₂ x₂ + … + βₚ xₚ
Why 67 coefficients from 30 features? Text columns become 0/1 columns: higher yes/no becomes higheryes (1 for yes). A feature with 4 categories gets 3 such columns, one for each category except the first, the reference.
R² = 1 − RSS / TSS
RMSE = √ mean( residual² )
linear_reg() |> fit(final_grade ~ higher, data = students)
higher + failures. Everything else in the table: .In words: “fit a linear model of final_grade explained by higher, using the students table.”
Part 1
649 students · 30 features · one grade
final_grade, the outcome. Mean 11.9, median 12.Who they are school · sex · age · address · famsize · pstatus · medu · fedu · mjob · fjob · guardian
How they study studytime · failures · schoolsup · famsup · paid · higher · absences · reason · traveltime
How they live goout · freetime · romantic · activities · famrel · dalc · walc · health · internet · nursery
Most features are text labels, not numbers: studytime is "<2hrs", "2-5hrs", "5-10hrs" or ">10hrs"; health runs from "VeryBad" to "VeryGood". failures (0–3), age and absences are numeric.
Part 1 · EDA · notebook line 131
final_grade: is it skewed?final_grade by higherSyntax you need
students |>
ggplot(aes(x = final_grade)) +
geom_histogram(bins = 20) +
theme_histogram()
students |>
ggplot(aes(x = higher, y = final_grade)) +
geom_boxplot() +
theme_boxplot()Stuck?
+ at the endread_csv() chunk near the top of the notebookResult Grades cluster between 10 and 14. A handful of students scored 0.
Result Students who plan higher education score higher. How much higher, and could it be chance? That’s Part 2.
Part 2
Is there an effect? How sure are we? · summary() · ~ .
summary()Notebook lines 146–152 the hypothesis-testing approachrun together
| Estimate | Std. Error | t value | Pr(>|t|) | |
| (Intercept) | 8.80 | 0.367 | 24.0 | <2e-16 |
| higheryes | 3.48 | 0.388 | 8.96 | <2e-16 |
Also printed: Multiple R-squared 0.110. extract_fit_engine() pulls the lm object out of the tidymodels wrapper so summary() can print it.
Estimate The coefficient: students who want higher education score 3.48 points higher. The intercept, 8.80, is the mean grade of the “no” students.
Std. Error How uncertain the estimate is. Another sample would give a slightly different 3.48.
t value and p-value t = estimate ÷ standard error: 9 standard errors from zero. The p-value: if the true effect were 0, how surprising would this be? Extremely.
~ .Notebook lines 160–165 the multivariable modelrun together
# A tibble: 7 × 3
term estimate p.value
<chr> <dbl> <dbl>
1 schoolMS -1.2 1.3e- 5
2 sexM -0.7 5.7e- 3
3 studytime5-10hrs 0.97 9 e- 3
4 failures -1.5 3.4e-12
5 schoolsupyes -1.1 1.9e- 3
6 higheryes 1.6 3.5e- 5
7 dalcHigh -2.7 5.4e- 4
The 7 features with p < 0.01, out of 67 coefficients.
higheryes halves From 3.48 to 1.62, still significant. Part of the raw difference was study time and past failures travelling with the wish for higher education.
R² rises to 0.41 Sixty-seven coefficients explain 41% of the variation. Each past failure costs about 1.5 points.
Caveat p-values say which features matter for explaining these 649 students. They are no guarantee the model predicts well for new students.
Part 3
Split · fit · evaluate · initial_split() · augment() · metric_set()
Hypothesis testing Does higher matter? How sure are we?
Coefficients, standard errors, p-values. Uses all the data. Part 2.
Machine learning How well can we predict a grade we haven’t seen?
Predictions and their error on new data. Splits the data. Parts 3–5.
Same model. Different question. Different workflow.
Notebook lines 191–212 the notebook's version of "what is a model?"code together
Lines 192–195: the rule as an R function Three inputs, three coefficients, one output. Nobody fitted it: the numbers are made up, like the grade rule earlier.
sim_model_preds(10, 20, 30)150 - sim_model_preds(10, 20, 30)You revise with last year’s exam paper, then sit… last year’s paper again. Is your mark a fair measure of what you learned?
Overfitting Scored on its own training rows, a model flatters itself: it has partly memorised their quirks. The test set is a fresh exam. A big gap between the two scores is the warning sign.
Notebook lines 218–225 type the three steps under the three commentscode together
set.seed(123) students_split <- initial_split(students, prop = 0.75) students_train <- training(students_split) students_test <- testing(students_split)
Notebook lines 233–235 the "Code here" chunk: fit on the training setcode together
mv_model <- linear_reg() |> fit(final_grade ~ ., data = students_train) mv_model |> augment(new_data = students_test)
augment() takes a table and gives it back with two new columns. It appears in the evaluation chunk next.| final_grade | school | … | .pred | .resid |
| 11 | GP | … | 13.4 | −2.36 |
| 12 | GP | … | 11.0 | 0.99 |
.pred the prediction for each row · .resid actual − prediction
Notebook lines 246–256 given in the notebook: run it, training set then test setrun together
reg_metrics <- metric_set(rsq, rmse) mv_model |> augment(new_data = students_test) |> reg_metrics(truth = final_grade, estimate = .pred)
rsq is R², rmse is RMSE..pred column that augment() just added.The notebook runs it twice: once with new_data = students_train, once with students_test. Compare the two tables.
set.seed(123)
students_split <- initial_split(students, prop = 0.75)
students_train <- training(students_split)
students_test <- testing(students_split)
mv_model <- linear_reg() |>
fit(final_grade ~ ., data = students_train)
reg_metrics <- metric_set(rsq, rmse)
mv_model |>
augment(new_data = students_test) |>
reg_metrics(truth = final_grade, estimate = .pred)Notebook lines 218–256 as one pipeline: split the students, fit the full linear model on the training rows, then score it on the test rows. Each click shows the output of the next step.
initial_split() records which rows are which. Printing the split object shows the sizes: 486 training, 163 test, 649 in total. training() and testing() turn it into two tibbles.
<Training/Testing/Total>
<486/163/649>
fit() runs least squares on the 486 training rows. tidy() shows the coefficients it found: one row each, 68 in all.
# A tibble: 68 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) 7.45 2.38 3.13 0.00190
2 schoolMS -0.998 0.323 -3.09 0.00216
3 sexM -0.549 0.294 -1.87 0.0626
4 age 0.154 0.120 1.29 0.198
# ℹ 64 more rows
metric_set() makes a new function that computes both metrics at once.
A metric set, consisting of:
- `rsq()`, a numeric metric | direction:
maximize
- `rmse()`, a numeric metric | direction:
minimize
augment() returns the test table with two new columns: .pred, the predicted grade, and .resid, how far off it was. (Only those and the grade are shown here.)
# A tibble: 163 × 3
final_grade .pred .resid
<dbl> <dbl> <dbl>
1 11 13.4 -2.36
2 11 13.0 -1.98
3 12 11.0 0.988
4 17 15.6 1.39
# ℹ 159 more rows
reg_metrics() compares the two columns. On students the model has never seen: R² 0.21, RMSE 3.04 grade points.
# A tibble: 2 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 rsq standard 0.211
2 rmse standard 3.04
On the training set the same three lines give R² 0.446 and RMSE 2.37.
Notebook lines 258–260 classroom discussion: how can we interpret these results?run together
R² share of variation explained
RMSE typical miss, in grade points
Is this model good? Better than guessing the mean. But on new students it explains 21% of the variation, not 45%, and a typical prediction is 3 grade points off. The gap is overfitting.
Why? 67 coefficients fitted on 486 rows. Many describe noise. Could we keep only the useful ones? Part 5.
Part 4
Residual plots · .pred against .resid
Notebook lines 268–324 run the four chunks: simulated data, its scatter, its residual plot, then oursrun together
A shapeless cloud around zero Nothing left to explain. This is what a well-fitting model leaves behind.
Wide, and with a pattern A diagonal streak at the bottom left. Any pattern is something the model missed.
Notebook lines 302–324 the same plot twice: simulated data, then our model on the test setcode together
mv_model |> augment(new_data = students_test) |> ggplot(aes(.pred, .resid)) + geom_hline(yintercept = 0, linetype = "dashed") + geom_point()
augment().augment() first, then ggplot(): the two columns only exist once the model has been applied. Line 326, the action point: why does the simulated data look so much better?
Part 5
Regularisation · λ · hyperparameters · penalty · mixture = 1 · set_engine("glmnet")
67 coefficients, 486 training rows The model is free to use every coefficient, so it does, including the ones that only describe noise in the training rows. R² 0.45 in training, 0.21 on new students.
How would you stop a model from using coefficients it doesn’t need?
Option 1: by hand Drop the features with large p-values and refit. Slow, subjective, and the p-values came from the model you’re about to change.
Option 2: inside the algorithm Change the score the algorithm minimises so that every coefficient has a cost. That’s the lasso.
Lasso minimises RSS + λ · Σ |βⱼ|
penalty.λ = 0 No rent. Back to OLS: all 67 coefficients in use, and the overfitting from Part 3.
λ = 0.1 Coefficients shrink, and 27 of them hit exactly zero. Feature selection built into the fitting.
λ = 1 Only 2 survive. Dial it up further and everything goes to zero: the model predicts the mean for everyone.
Two forces on each coefficient The data pull it away from zero, with a force that weakens as the fit improves. The penalty pushes it back toward zero.
The lasso’s push is constant |β| has the same slope everywhere, so the push is λ, whatever the coefficient’s size. If the data’s pull is weaker than λ, the coefficient is pinned at exactly 0.
Ridge, for contrast β² goes flat near zero, so its push fades as the coefficient shrinks: coefficients get tiny but never reach zero. mixture = 1 picks the lasso, 0 the ridge.
Parameters The βs, 67 of them. Chosen by the algorithm, from the data, by minimising the score. Inside the machine.
Hyperparameters λ (penalty) and mixture. Chosen by you, before fitting. A knob on the outside of the machine.
Why can’t the data choose λ? Any λ > 0 makes the training fit worse, so the algorithm would always pick 0. Instead: try several values, score each on data the model hasn’t seen, keep the best. Today’s bonus does it by hand; Lab 5 automates it.
Notebook lines 336–341 Step 1: run it together. Steps 2–5 are yoursrun together
lasso_model <- linear_reg(penalty = 0.1, mixture = 1) |> set_engine("glmnet") |> fit(final_grade ~ ., data = students_train) tidy(lasso_model)
lm, can’t penalise; glmnet can.tidy() gives one row per coefficient. Filter out the zeros.Everything after fit() is unchanged: augment() and reg_metrics() work exactly as in Part 3, so Step 4 is copy and paste.
Part 5 · lasso · notebook lines 347–377
tidy() the model and keep the non-zero coefficients. How many survived? Which is the largest?reg_metrics. Better or worse than Part 3?Syntax you need
tidy(lasso_model) |>
filter(estimate != 0)
tidy(lasso_model) |>
filter(term != "(Intercept)") |>
ggplot(aes(abs(estimate), fct_reorder(term, abs(estimate)),
fill = estimate > 0)) +
geom_col() + theme_bar() + scale_fill_jama()
lasso_model |>
augment(new_data = students_test) |>
reg_metrics(truth = final_grade, estimate = .pred)Stuck?
penalty, then Step 4 again. The solutions have a map() versionlasso_model <-
linear_reg(penalty = 0.1, mixture = 1) |>
set_engine("glmnet") |>
fit(final_grade ~ ., data = students_train)
tidy(lasso_model) |>
filter(estimate != 0)
lasso_model |>
augment(new_data = students_test) |>
reg_metrics(truth = final_grade, estimate = .pred)Fit a lasso with penalty 0.1 on the training rows, look at which coefficients survived, then score it on the test rows.
Same fit() as before, with a price on coefficients and glmnet doing the work. Counting the coefficients: 27 were set to exactly zero, 41 (including the intercept) survived.
# A tibble: 2 × 2
zero n
<lgl> <int>
1 FALSE 41
2 TRUE 27
tidy() lists every coefficient; filter() keeps the non-zero ones. Sorted by size here: wanting higher education, past failures and school support matter most.
# A tibble: 40 × 3
term estimate penalty
<chr> <dbl> <dbl>
1 higheryes 1.81 0.1
2 failures -1.37 0.1
3 schoolsupyes -1.05 0.1
4 feduNone 1.01 0.1
5 schoolMS -0.966 0.1
6 traveltime>1hr -0.568 0.1
# ℹ 34 more rows
Exactly the evaluation code from Part 3. On the test set: R² 0.24 and RMSE 2.95, against 0.21 and 3.04 for the model with all 67 coefficients.
# A tibble: 2 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 rsq standard 0.240
2 rmse standard 2.95
Step 2 shows the 6 largest coefficients, sorted with arrange(desc(abs(estimate))), to fit the slide. The notebook prints all 41 in the original order.
Notebook lines 347–365 Steps 2–4your turn
Test set 163 students the models never saw
| Coefficients | R² | RMSE | |
Linear, ~ . |
67 | 0.211 | 3.04 |
| Lasso, penalty 0.1 | 40 | 0.240 | 2.95 |
Result A simpler model that predicts new students slightly better. 27 coefficients gone, nothing lost. That’s regularisation.
The notebook’s grid 0.0001, 0.001 and 0.01 are all about 3.03, hardly different from plain OLS. 0.1 is best at 2.95.
Two more, past the grid 0.3 is worse again (3.04, with only 14 coefficients) and 1 is bad (3.29, with 2). A bowl again, this time scored on test data.
That’s tuning Turn the knob, score on data the model hasn’t seen, keep the best setting. Lab 5 automates it.
linear_reg() |> fit() · augment() · metric_set(rsq, rmse) is the workflow for every model from now onQuestion: which of the 30 features would you have bet on before today? Did the lasso agree?
parsnip and yardstick chapters.Week 4 Classification: predicting categories instead of numbers
MY202 · Lab 3 · Linear regression