Linear Regression: our first algorithm

MY202 · Lab 3

Jonas Weinert

Department of Methodology  |  London School of Economics and Political Science

29 October 2026

Lab overview

Part Topic Functions Mode Time
Setup: notebook lines 24–72 together 2 min
Concept: warm-up, then our first algorithm ~, linear_reg(), fit() slides 15 min
1 Exploring the data ggplot, geom_histogram, geom_boxplot your turn 12 min
2 Hypothesis testing extract_fit_engine, summary, ~ . together 8 min
3 The machine learning approach initial_split, training, testing, augment, metric_set together 18 min
4 Where do we make errors? .pred, .resid, geom_hline together 8 min
5 Lasso penalty, mixture, set_engine, tidy step 1 together, then your turn 22 min

Together: we type the code line by line, the slide explains each piece, and a notebook line number says where to type. Your turn: a timer, tasks and a syntax sheet, then the solutions.

Our first algorithm

Concept

Three warm-up questions · what a model is · how a line is chosen · how we judge it

Warm-up 1 of 3

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

A rule predicts a grade as 10 + 0.5 × hours of study. A student studies 4 hours a week. What grade does the rule predict?

A10.5
B12
C14
DI'm not sure how to read the rule
B. 10 + 0.5 × 4 = 10 + 2 = 12. Reading a rule like this is all the arithmetic we need today.

Warm-up 2 of 3

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

How does linear regression choose which line to draw through the points?

AThe line that passes through the most points
BThe line whose gaps to the points add up to zero
CThe line whose squared gaps to the points are as small as possible
DI've never thought about it
C. Ordinary least squares: the smallest sum of squared residuals. B is a trap: many bad lines have gaps that add up to zero.

Warm-up 3 of 3

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

A house-price model has R² = 0.65. What does that mean?

A65% of the model's predictions are correct
BThe features (such as floor area) explain 65% of the variation in house prices
CThe model's error is 65% smaller than a perfect model's
DI haven't met R² before
B. R² is about variation, not about how many predictions are "right": a regression prediction is almost never exactly right. The remaining 35% is left in the residuals.

What is a model?

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

A model is a rule that turns inputs into a prediction. Made up for now: a grade from study hours and past failures.

The rulegrade = 10 + 0.5 × hours − 2 × failures
One student6 hours a week, 1 past failure
Plug in10 + 0.5 × 6 − 2 × 1 = 10 + 3 − 2
Prediction= 11
Actual grade13
Residual13 − 11 = +2 the rule guessed 2 points too low

Where do 10, 0.5 and −2 come from? The numbers in the rule are its coefficients. Someone has to choose them.

An algorithm is a recipe for choosing them from data. Today’s: ordinary least squares, built from scratch in the next five slides.

Which line is best?

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

200 markets: what was spent on TV ads, and how much was sold. We want a line that predicts sales from a budget.

Vote: A, B or C?

The harder question Most pick B. But why? A computer can’t choose a line without a number that says how wrong a line is.

How wrong is a line? Residuals

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

One marketbudget 230.1, sold 22.1
The line sayssales ≈ 18.0
Residual22.1 − 18.0 = +4.1

Residual = actual − predicted Positive: the line guessed too low. Negative: too high.

200 markets, 200 residuals How do we turn 200 numbers into one score for the line?

Add them up?

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Just add the residuals

A flat line at the average is clearly bad, yet its residuals add up to exactly 0: positives and negatives cancel.

Square first, then add

Market 1+4.1 ² = 17.1
Market 2+1.3 ² = 1.6
Market 5-2.7 ² = 7.4 negative, but its square is positive
All 200RSS = 2,103 residual sum of squares

Flat line: 5,417. Line B: 2,103. Lower is better, and one miss of 4 now costs more than two misses of 2.

Try every line: the bowl

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Ordinary least squares (OLS) The line with the smallest sum of squared residuals. That’s the whole algorithm.

You don’t have to try every line For linear models a formula jumps straight to the bottom of the bowl. Later models really do walk downhill.

The equation, piece by piece

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

ŷ = β₀ + β₁ · x

  1. The prediction. The hat means “our estimate”, as opposed to the real y. Spoken “y-hat”.
  2. The intercept. The prediction when x is 0.
  3. The slope. The change in the prediction when x goes up by one unit.
  4. The feature. The input we know. Also called predictor or independent variable.
Fitted linesales = 7.03 + 0.0475 × TV every extra $1,000 on TV adds about 48 units
Market 1TV = 230.1
Plug in7.03 + 0.0475 × 230.1 = 7.03 + 10.94
Predictionŷ = 18.0 the "line says" number from three slides ago

More than one feature

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

ŷ = β₀ + β₁ x₁ + β₂ x₂ + … + βₚ xₚ

  1. One coefficient per feature. Each β: the change in the prediction when that feature rises by 1, with the others held fixed.
  2. Still a line, just in more dimensions. Same squared residuals, same bowl, same bottom.
  3. p features, p slopes. Today: 30 columns, 67 coefficients.

Why 67 coefficients from 30 features? Text columns become 0/1 columns: higher yes/no becomes higheryes (1 for yes). A feature with 4 categories gets 3 such columns, one for each category except the first, the reference.

How good is the line? R²

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

R² = 1 − RSS / TSS

  1. Share of the variation explained: 0 is no better than the mean, 1 is a perfect fit.
  2. Squared residuals of our line.
  3. Squared residuals of the flat line at the mean.

Flat lineTSS = 5,417
Line BRSS = 2,103
Divide1 − 2,103 / 5,417 = 1 − 0.39
R²= 0.61 TV explains 61% of the variation

How good is the line? RMSE

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

RMSE = √ mean( residual² )

  1. Root mean squared error: the typical miss, in the units of the outcome. An RMSE of 3 on a 0–20 grade means a typical prediction is about 3 points off.
  2. Average the squared residuals, then take the square root to undo the squaring.
AdvertisingRSS = 2,103 over 200 markets
Mean2,103 / 200 = 10.5 the average squared residual
Root√10.5 = 3.2
RMSE= 3.2 thousand units a typical prediction is about 3200 units off

The same thing in code

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

linear_reg() |> fit(final_grade ~ higher, data = students)

  1. Which kind of model. A linear one. On its own this fits nothing: it’s an empty recipe.
  2. Run the algorithm. Find the coefficients with the smallest RSS.
  3. The outcome, y: what we want to predict. Left of the tilde.
  4. The tilde reads as “explained by” or “as a function of”. It turns the two sides into a formula.
  5. The feature(s), x: right of the tilde. Several: higher + failures. Everything else in the table: .
  6. Which table the columns live in.

In words: “fit a linear model of final_grade explained by higher, using the students table.”

Exploring the data

Part 1

649 students · 30 features · one grade

Meet the data

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up
649students from two Portuguese secondary schools. One row each.
30features: who they are, how they study, how they live
0–20final_grade, the outcome. Mean 11.9, median 12.

Who they are school · sex · age · address · famsize · pstatus · medu · fedu · mjob · fjob · guardian

How they study studytime · failures · schoolsup · famsup · paid · higher · absences · reason · traveltime

How they live goout · freetime · romantic · activities · famrel · dalc · walc · health · internet · nursery

Most features are text labels, not numbers: studytime is "<2hrs", "2-5hrs", "5-10hrs" or ">10hrs"; health runs from "VeryBad" to "VeryGood". failures (0–3), age and absences are numeric.

Exercises: Part 1

Part 1 · EDA · notebook line 131

  1. A histogram of final_grade: is it skewed?
  2. A boxplot of final_grade by higher
  3. One plot of your own from the list in the notebook: study time, failures, school, going out, health, support…
  4. One sentence: what seems to matter for grades?
  5. Share your most interesting plot on Moodle (notebook line 129)

Syntax you need

students |>
  ggplot(aes(x = final_grade)) +
  geom_histogram(bins = 20) +
  theme_histogram()

students |>
  ggplot(aes(x = higher, y = final_grade)) +
  geom_boxplot() +
  theme_boxplot()

Stuck?

  • could not find function “theme_histogram”: run the setup chunk
  • Empty plot? Every line except the last needs a + at the end
  • object ‘students’ not found: run the read_csv() chunk near the top of the notebook

Part 1: what did we see?

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Result Grades cluster between 10 and 14. A handful of students scored 0.

Result Students who plan higher education score higher. How much higher, and could it be chance? That’s Part 2.

The hypothesis-testing approach

Part 2

Is there an effect? How sure are we? · summary() · ~ .

Reading summary()

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 146–152 the hypothesis-testing approachrun together

linear_reg() |>
  fit(final_grade ~ higher, data = students) |>
  extract_fit_engine() |>
  summary()
Estimate Std. Error t value Pr(>|t|)
(Intercept) 8.80 0.367 24.0 <2e-16
higheryes 3.48 0.388 8.96 <2e-16

Also printed: Multiple R-squared 0.110. extract_fit_engine() pulls the lm object out of the tidymodels wrapper so summary() can print it.

Estimate The coefficient: students who want higher education score 3.48 points higher. The intercept, 8.80, is the mean grade of the “no” students.

Std. Error How uncertain the estimate is. Another sample would give a slightly different 3.48.

t value and p-value t = estimate ÷ standard error: 9 standard errors from zero. The p-value: if the true effect were 0, how surprising would this be? Extremely.

Controlling for everything: ~ .

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 160–165 the multivariable modelrun together

linear_reg() |>
  fit(final_grade ~ ., data = students) |>
  extract_fit_engine() |>
  summary()
# A tibble: 7 × 3
  term             estimate p.value
  <chr>               <dbl>   <dbl>
1 schoolMS            -1.2  1.3e- 5
2 sexM                -0.7  5.7e- 3
3 studytime5-10hrs     0.97 9  e- 3
4 failures            -1.5  3.4e-12
5 schoolsupyes        -1.1  1.9e- 3
6 higheryes            1.6  3.5e- 5
7 dalcHigh            -2.7  5.4e- 4

The 7 features with p < 0.01, out of 67 coefficients.

higheryes halves From 3.48 to 1.62, still significant. Part of the raw difference was study time and past failures travelling with the wish for higher education.

R² rises to 0.41 Sixty-seven coefficients explain 41% of the variation. Each past failure costs about 1.5 points.

Caveat p-values say which features matter for explaining these 649 students. They are no guarantee the model predicts well for new students.

The machine learning approach

Part 3

Split · fit · evaluate · initial_split() · augment() · metric_set()

Two questions, two approaches

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Hypothesis testing Does higher matter? How sure are we?

Coefficients, standard errors, p-values. Uses all the data. Part 2.

Machine learning How well can we predict a grade we haven’t seen?

Predictions and their error on new data. Splits the data. Parts 3–5.

Same model. Different question. Different workflow.

A made-up model, in the notebook

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 191–212 the notebook's version of "what is a model?"code together

sim_model_preds <- function(x1, x2, x3) {
  y <- 1.1 * x1 + 2.2 * x2 + 3.3 * x3
  y
}

Lines 192–195: the rule as an R function Three inputs, three coefficients, one output. Nobody fitted it: the numbers are made up, like the grade rule earlier.

Line 201sim_model_preds(10, 20, 30)
Plug in1.1 × 10 + 2.2 × 20 + 3.3 × 30 = 11 + 44 + 99
Prediction= 154
Line 209150 - sim_model_preds(10, 20, 30)
Residual150 − 154 = −4 actual − predicted: the rule guessed 4 too high

Why split?

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

You revise with last year’s exam paper, then sit… last year’s paper again. Is your mark a fair measure of what you learned?

649 studentsall the rows
→
Training set · 486 rows (75%)the model is fitted here
Test set · 163 rows (25%)locked away; opened only to score the finished model

Overfitting Scored on its own training rows, a model flatters itself: it has partly memorised their quirks. The test set is a fresh exam. A big gap between the two scores is the warning sign.

The split, piece by piece

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 218–225 type the three steps under the three commentscode together

set.seed(123) students_split <- initial_split(students, prop = 0.75) students_train <- training(students_split) students_test <- testing(students_split)

  1. Make the random draw repeatable, so your numbers match the slides.
  2. Decide which rows go where. A split object: a note of which rows are which, not yet two tables.
  3. The share for training. 75% here; 70–80% is typical.
  4. Pull out the training table: 486 rows.
  5. Pull out the test table: 163 rows. Don’t touch it until the model is finished.

Fit and predict

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 233–235 the "Code here" chunk: fit on the training setcode together

mv_model <- linear_reg() |> fit(final_grade ~ ., data = students_train) mv_model |> augment(new_data = students_test)

  1. Fit on the training rows only, and save the model. This is the line to type at line 234.
  2. Predict. augment() takes a table and gives it back with two new columns. It appears in the evaluation chunk next.
final_grade school … .pred .resid
11 GP … 13.4 −2.36
12 GP … 11.0 0.99

.pred the prediction for each row · .resid actual − prediction

Evaluate

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 246–256 given in the notebook: run it, training set then test setrun together

reg_metrics <- metric_set(rsq, rmse) mv_model |> augment(new_data = students_test) |> reg_metrics(truth = final_grade, estimate = .pred)

  1. Bundle the metrics into one function. rsq is R², rmse is RMSE.
  2. Which column is the truth: the real grade.
  3. Which column is the guess: the .pred column that augment() just added.

The notebook runs it twice: once with new_data = students_train, once with students_test. Compare the two tables.

Part 3 together: what you should see

set.seed(123)
students_split <- initial_split(students, prop = 0.75)
students_train <- training(students_split)
students_test  <- testing(students_split)

mv_model <- linear_reg() |>
  fit(final_grade ~ ., data = students_train)

reg_metrics <- metric_set(rsq, rmse)

mv_model |>
  augment(new_data = students_test) |>
  reg_metrics(truth = final_grade, estimate = .pred)

Notebook lines 218–256 as one pipeline: split the students, fit the full linear model on the training rows, then score it on the test rows. Each click shows the output of the next step.

initial_split() records which rows are which. Printing the split object shows the sizes: 486 training, 163 test, 649 in total. training() and testing() turn it into two tibbles.

<Training/Testing/Total>
<486/163/649>

fit() runs least squares on the 486 training rows. tidy() shows the coefficients it found: one row each, 68 in all.

# A tibble: 68 × 5
  term        estimate std.error statistic p.value
  <chr>          <dbl>     <dbl>     <dbl>   <dbl>
1 (Intercept)    7.45      2.38       3.13 0.00190
2 schoolMS      -0.998     0.323     -3.09 0.00216
3 sexM          -0.549     0.294     -1.87 0.0626 
4 age            0.154     0.120      1.29 0.198  
# ℹ 64 more rows

metric_set() makes a new function that computes both metrics at once.

A metric set, consisting of:
- `rsq()`, a numeric metric  | direction:
maximize
- `rmse()`, a numeric metric | direction:
minimize

augment() returns the test table with two new columns: .pred, the predicted grade, and .resid, how far off it was. (Only those and the grade are shown here.)

# A tibble: 163 × 3
  final_grade .pred .resid
        <dbl> <dbl>  <dbl>
1          11  13.4 -2.36 
2          11  13.0 -1.98 
3          12  11.0  0.988
4          17  15.6  1.39 
# ℹ 159 more rows

reg_metrics() compares the two columns. On students the model has never seen: R² 0.21, RMSE 3.04 grade points.

# A tibble: 2 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rsq     standard       0.211
2 rmse    standard       3.04 

On the training set the same three lines give R² 0.446 and RMSE 2.37.

Part 3: training versus test

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 258–260 classroom discussion: how can we interpret these results?run together

R² share of variation explained

Training0.446
Test0.211

RMSE typical miss, in grade points

Training2.37
Test3.04

Is this model good? Better than guessing the mean. But on new students it explains 21% of the variation, not 45%, and a typical prediction is 3 grade points off. The gap is overfitting.

Why? 67 coefficients fitted on 486 rows. Many describe noise. Could we keep only the useful ones? Part 5.

Where do we make errors?

Part 4

Residual plots · .pred against .resid

Residual plots: what good looks like

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 268–324 run the four chunks: simulated data, its scatter, its residual plot, then oursrun together

A shapeless cloud around zero Nothing left to explain. This is what a well-fitting model leaves behind.

Wide, and with a pattern A diagonal streak at the bottom left. Any pattern is something the model missed.

The residual plot, piece by piece

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 302–324 the same plot twice: simulated data, then our model on the test setcode together

mv_model |> augment(new_data = students_test) |> ggplot(aes(.pred, .resid)) + geom_hline(yintercept = 0, linetype = "dashed") + geom_point()

  1. x: the prediction for each student, from augment().
  2. y: the residual, actual − predicted. Above the line the model guessed too low, below it too high.
  3. The “no error” line. A good model leaves a shapeless cloud around it.

augment() first, then ggplot(): the two columns only exist once the model has been applied. Line 326, the action point: why does the simulated data look so much better?

Lasso: a penalty for every coefficient

Part 5

Regularisation · λ · hyperparameters · penalty · mixture = 1 · set_engine("glmnet")

The problem with “everything”

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

67 coefficients, 486 training rows The model is free to use every coefficient, so it does, including the ones that only describe noise in the training rows. R² 0.45 in training, 0.21 on new students.

How would you stop a model from using coefficients it doesn’t need?

Option 1: by hand Drop the features with large p-values and refit. Slow, subjective, and the p-values came from the model you’re about to change.

Option 2: inside the algorithm Change the score the algorithm minimises so that every coefficient has a cost. That’s the lasso.

Charge rent for every coefficient

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Lasso minimises   RSS + λ · Σ |βⱼ|

  1. Fit the data, as before. OLS minimised this and nothing else.
  2. Lambda: the price per unit of coefficient. You choose it. The notebook calls it penalty.
  3. The total size of all coefficients, signs ignored. The bill.
Say λ = 0.1and three coefficients: 1.8, −1.4, 0.05
Total size|1.8| + |−1.4| + |0.05| = 3.25
The bill0.1 × 3.25 = 0.325 added to the RSS
Drop the 0.05bill falls by 0.005 if the RSS rises by less than that, the model is better off without it

Turning the dial

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

λ = 0 No rent. Back to OLS: all 67 coefficients in use, and the overfitting from Part 3.

λ = 0.1 Coefficients shrink, and 27 of them hit exactly zero. Feature selection built into the fitting.

λ = 1 Only 2 survive. Dial it up further and everything goes to zero: the model predicts the mean for everyone.

Why exactly zero?

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Two forces on each coefficient The data pull it away from zero, with a force that weakens as the fit improves. The penalty pushes it back toward zero.

The lasso’s push is constant |β| has the same slope everywhere, so the push is λ, whatever the coefficient’s size. If the data’s pull is weaker than λ, the coefficient is pinned at exactly 0.

Ridge, for contrast β² goes flat near zero, so its push fades as the coefficient shrinks: coefficients get tiny but never reach zero. mixture = 1 picks the lasso, 0 the ridge.

Parameters and hyperparameters

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Parameters The βs, 67 of them. Chosen by the algorithm, from the data, by minimising the score. Inside the machine.

Hyperparameters λ (penalty) and mixture. Chosen by you, before fitting. A knob on the outside of the machine.

Why can’t the data choose λ? Any λ > 0 makes the training fit worse, so the algorithm would always pick 0. Instead: try several values, score each on data the model hasn’t seen, keep the best. Today’s bonus does it by hand; Lab 5 automates it.

Lasso in code

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 336–341 Step 1: run it together. Steps 2–5 are yoursrun together

lasso_model <- linear_reg(penalty = 0.1, mixture = 1) |> set_engine("glmnet") |> fit(final_grade ~ ., data = students_train) tidy(lasso_model)

  1. λ. The rent per unit of coefficient. 0.1 to start.
  2. Which penalty. 1 is the lasso, 0 is the ridge, in between mixes them.
  3. Which package computes. The default, lm, can’t penalise; glmnet can.
  4. Step 2, yours: tidy() gives one row per coefficient. Filter out the zeros.

Everything after fit() is unchanged: augment() and reg_metrics() work exactly as in Part 3, so Step 4 is copy and paste.

Exercises: Part 5

Part 5 · lasso · notebook lines 347–377

  1. Line 348: tidy() the model and keep the non-zero coefficients. How many survived? Which is the largest?
  2. Line 356: the bar plot of coefficient sizes
  3. Line 364: test-set R² and RMSE with reg_metrics. Better or worse than Part 3?
  4. Line 376, bonus: try other penalties (0.01, 0.3, 1). Which gives the lowest test RMSE?

Syntax you need

tidy(lasso_model) |>
  filter(estimate != 0)

tidy(lasso_model) |>
  filter(term != "(Intercept)") |>
  ggplot(aes(abs(estimate), fct_reorder(term, abs(estimate)),
             fill = estimate > 0)) +
  geom_col() + theme_bar() + scale_fill_jama()

lasso_model |>
  augment(new_data = students_test) |>
  reg_metrics(truth = final_grade, estimate = .pred)

Stuck?

  • could not find function “fct_reorder”: run the setup chunk (it’s in the tidyverse)
  • Bonus: copy Step 1 with a new penalty, then Step 4 again. The solutions have a map() version

Solutions: Part 5

lasso_model <-
  linear_reg(penalty = 0.1, mixture = 1) |>
  set_engine("glmnet") |>
  fit(final_grade ~ ., data = students_train)

tidy(lasso_model) |>
  filter(estimate != 0)

lasso_model |>
  augment(new_data = students_test) |>
  reg_metrics(truth = final_grade, estimate = .pred)

Fit a lasso with penalty 0.1 on the training rows, look at which coefficients survived, then score it on the test rows.

Same fit() as before, with a price on coefficients and glmnet doing the work. Counting the coefficients: 27 were set to exactly zero, 41 (including the intercept) survived.

# A tibble: 2 × 2
  zero      n
  <lgl> <int>
1 FALSE    41
2 TRUE     27

tidy() lists every coefficient; filter() keeps the non-zero ones. Sorted by size here: wanting higher education, past failures and school support matter most.

# A tibble: 40 × 3
  term           estimate penalty
  <chr>             <dbl>   <dbl>
1 higheryes         1.81      0.1
2 failures         -1.37      0.1
3 schoolsupyes     -1.05      0.1
4 feduNone          1.01      0.1
5 schoolMS         -0.966     0.1
6 traveltime>1hr   -0.568     0.1
# ℹ 34 more rows

Exactly the evaluation code from Part 3. On the test set: R² 0.24 and RMSE 2.95, against 0.21 and 3.04 for the model with all 67 coefficients.

# A tibble: 2 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rsq     standard       0.240
2 rmse    standard       2.95 

Step 2 shows the 6 largest coefficients, sorted with arrange(desc(abs(estimate))), to fit the slide. The notebook prints all 41 in the original order.

Part 5: did the penalty help?

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

Notebook lines 347–365 Steps 2–4your turn

Test set 163 students the models never saw

Coefficients R² RMSE
Linear, ~ . 67 0.211 3.04
Lasso, penalty 0.1 40 0.240 2.95

Result A simpler model that predicts new students slightly better. 27 coefficients gone, nothing lost. That’s regularisation.

Bonus: turning the dial by hand

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up

The notebook’s grid 0.0001, 0.001 and 0.01 are all about 3.03, hardly different from plain OLS. 0.1 is best at 2.95.

Two more, past the grid 0.3 is worse again (3.04, with only 14 coefficients) and 1 is bad (3.29, with 2). A bowl again, this time scored on test data.

That’s tuning Turn the knob, score on data the model hasn’t seen, keep the best setting. Lab 5 automates it.

Summary

Concept1 · EDA2 · Hypothesis3 · ML4 · Residuals5 · LassoWrap-up
1Ordinary least squares picks the line with the smallest sum of squared residuals. A coefficient is the change in the prediction when its feature rises by one unit, other things equal
2Split first. Fit on the training set, score on the test set. The gap between the two scores is overfitting
3linear_reg() |> fit() · augment() · metric_set(rsq, rmse) is the workflow for every model from now on
4The lasso adds a price per coefficient. λ is a hyperparameter you set; coefficients that aren't worth the rent go to exactly zero

Question: which of the 30 features would you have bet on before today? Did the lasso agree?

Further reading

Week 4 Classification: predicting categories instead of numbers