Linear Regression: our first algorithm
MY202 · Lab 3 · Self-paced guide
Before you start
This page walks you through Lab 3 one small step at a time. It assumes you have done Labs 1 and 2, so you know what a notebook, a chunk, a pipe |> and a ggplot are. Everything new is explained the first time it appears, and there is a glossary at the end.
What’s new this week. Today you meet your first algorithm: a recipe a computer follows to learn something from data. The algorithm is ordinary least squares, the engine behind linear regression. Before any code, the page builds it from scratch with nothing harder than school arithmetic. Then you use it on real data, twice: once the way a social scientist tests a hypothesis, once the way a machine learning scientist builds a prediction.
Your turn Something to do in the notebook. The box says which notebook lines to go to.
What you should see The real result R gives back. Compare it with yours.
Big idea A concept worth remembering beyond today.
Check yourself A quick question. Click an answer to see if you were right.
At the end of each step there is a Mark as done button. Your ticks are saved in this browser, so you can close the page and pick up where you left off.
The big idea at the start takes about 30 minutes and is worth doing slowly: everything after it builds on it. Parts 3 and 5 are the most important coding parts. Part 5’s bonus is optional.
Open the notebook and run the setup chunk
- Open the Lab 3 student notebook on Nuvolos. The packages and the data are already installed there.
- Run the chunk at lines 24–62. It loads five packages and creates the four
theme_...()functions you wrote in Lab 2. - Run the chunk at lines 70–72. It reads the data into an object called
students.
The notebook loads five packages. Two are new:
| Package | What it gives you |
|---|---|
tidymodels |
A family of packages for modelling, the way the tidyverse is a family for wrangling. Today: making models (linear_reg(), fit()), splitting data (initial_split()), and scoring predictions (metric_set()). |
MASS |
Used once, in Part 4, to simulate data. |
tidyverse, ggsci, scales |
Old friends from Labs 1 and 2: wrangling, plots, colour schemes and number formats. |
The notebook says “Please create a data folder called data”. On Nuvolos this is already done: the file sits in data/students-data-cleaned.csv. If the chunk at line 71 runs without an error, you’re set.
tidymodels loads about a dozen packages and lists them, plus some “conflicts” where two packages have a function with the same name. That’s normal, as in Lab 1.
The big idea: our first algorithm
About 30 minutes, no code. The notebook doesn’t explain how a linear model is fitted. This part does, because it is the foundation of almost every model in the course. You’ll use a second, simpler dataset here: how much 200 markets spent on TV advertising, and how many units they sold. One input and one outcome make it easy to draw.
What is a model?
A model is a rule that turns inputs into a prediction. Here is one, made up for now, that predicts a student’s grade from how many hours a week they study and how many times they have failed a course before:
The gap between what really happened and what the rule predicted is the residual: actual − predicted. A positive residual means the rule guessed too low; a negative one, too high.
The numbers inside the rule, 10, 0.5 and −2, are its coefficients (also called parameters). Someone has to choose them. An algorithm is a recipe for choosing them from data, so that the rule’s predictions are as good as possible. The next four steps build that recipe.
The same rule, a different student: 4 hours a week, no past failures, actual grade 11. What is the residual?
+1
Check the prediction first: 10 + 0.5 × 4 − 2 × 0 = 12. Then actual − predicted.
−1
The rule predicts 10 + 2 − 0 = 12. Actual − predicted = 11 − 12 = −1: the rule guessed 1 point too high.
12
12 is the prediction. The residual is the gap between the actual grade and the prediction.
Which line is best?
Here are the 200 markets. Each dot is one market: across, how much it spent on TV ads; up, how much it sold. We want a straight line through the cloud that predicts sales from a budget. Here are three candidates.
Most people pick B, and they are right. But think about why. “It looks like it goes through the middle” isn’t something a computer can use. To choose a line, a computer needs a number that says how wrong a line is. Lower number, better line.
How wrong is a line? Residuals
Take line B and one market. The line makes a guess for that market; the market’s real sales differ from the guess by its residual.
The pink segments are the residuals of all 200 markets. That’s 200 numbers. We need to turn them into one score for the line.
Add them up? Square them first
The obvious idea is to add the residuals up. Here is why that fails. This flat line at the average sales is clearly a bad line: it ignores the budget completely.
Yet its residuals add up to exactly zero: the misses above the line and the misses below it cancel out. A score that rates a terrible line as perfect is no use.
The fix is to square each residual first, then add. A square is never negative, so nothing cancels.
This total is the residual sum of squares, RSS. It is our score for a line: lower is better. The flat line scores 5,417; line B scores 2,103.
Squaring makes big misses count much more than small ones. One miss of 4 costs 4² = 16; two misses of 2 cost 2² + 2² = 8. So the best line will try hard to avoid large errors, which is usually what we want.
Try every line: the bowl
Now we can rank lines. Try many slopes, compute each one’s RSS, and plot RSS against slope:
Too flat: big RSS. Too steep: big RSS again. In between, a bowl with one lowest point. The line at the bottom of the bowl is line B.
Ordinary least squares (OLS) picks the line with the smallest sum of squared residuals. That’s the whole algorithm: “least squares” is a description, not a name. For straight-line models there is a formula that jumps directly to the bottom of the bowl, so R finds it in a split second without trying any lines.
For a linear model, yes. Each residual changes in a straight line as the slope changes, so its square is a parabola, and adding 200 parabolas gives one parabola that opens upwards. There is always exactly one bottom. With more features the bowl has more dimensions, but still one bottom. Later in the course you’ll meet models whose error surface is bumpy, with many valleys; those really do have to search.
Why not just add up the residuals to score a line?
Adding is slower for the computer.
Speed isn’t the issue. Think about what happens to positive and negative residuals.
Positive and negative residuals cancel, so a bad line can score zero.
Exactly: the flat line at the average scores zero. Squaring makes every miss count.
Residuals can’t be added because they are in different units.
All residuals are in the same units as the outcome (here, thousands of units sold), so they can be added. The problem is cancelling.
The equation, piece by piece
The line at the bottom of the bowl, written as an equation:
ŷ = β₀ + β₁ · x
- The prediction. The hat on y means “our estimate”, as opposed to the real y. Say “y-hat”.
- The intercept. The prediction when x is 0: where the line crosses the vertical axis.
- The slope. How much the prediction changes when x goes up by one unit. This is the coefficient the algorithm chooses.
- The feature. The input we know. Also called the predictor or independent variable.
The Greek letters are just names for the numbers the algorithm chooses (β is “beta”). For the advertising data, OLS chooses:
In words: with no TV advertising, a market sells about 7 thousand units, and every extra $1,000 spent on TV adds about 48 units.
More than one feature
With several features, the rule just gets more terms:
ŷ = β₀ + β₁ x₁ + β₂ x₂ + … + βₚ xₚ
- One coefficient per feature. Each β is the change in the prediction when that feature rises by 1, with all the other features held fixed.
- Still the same algorithm. Same squared residuals, same bowl (in more dimensions), same bottom.
- p features, p slopes. Your grade rule from the first step had p = 2.
Most of today’s features are text, like higher (“yes” or “no”). R turns them into 0/1 columns automatically: higheryes is 1 for students who answered yes and 0 otherwise. Its coefficient is the difference between yes and no students, other things equal. A feature with 4 categories, like studytime, becomes 3 such columns: one for each category except the first, the reference category the others are compared with. That’s why today’s 30 features turn into 67 coefficients.
How good is the line? R² and RMSE
OLS gives us the best line. But best can still be bad. Two numbers say how good it is, and you’ll report them for every model in this course.
R² = 1 − RSS / TSS
- R-squared: the share of the variation explained. 0 means no better than guessing the average; 1 means every point lies on the line.
- The RSS of our line.
- The RSS of the dumbest model: the flat line at the average, which predicts the mean for everyone. Its RSS is called the total sum of squares, TSS: all the variation there is to explain.
RMSE = √ mean( residual² )
- Root mean squared error: the typical miss, in the units of the outcome. For grades out of 20, an RMSE of 3 means a typical prediction is about 3 points off.
- Average the squared residuals, then take the square root to undo the squaring.
A house-price model has R² = 0.65. What does that mean?
65% of the model’s predictions are correct.
A very common misreading. Predictions from a regression are almost never exactly right, so “correct” isn’t what R² counts.
The features explain 65% of the variation in house prices.
Yes. The other 35% is left in the residuals: variation the model doesn’t explain.
The model’s error is 65% smaller than a perfect model’s.
A perfect model has no error at all, so this can’t be right. R² compares the model with the flat line at the average.
The same thing in code
In R, with tidymodels, fitting a linear model takes one line:
linear_reg() |> fit(final_grade ~ higher, data = students)
- Which kind of model. A linear one. On its own this fits nothing: it’s an empty recipe.
- Run the algorithm. Find the coefficients with the smallest RSS.
- The outcome, y: what we want to predict. It goes left of the tilde.
- The tilde
~reads as “explained by”. It turns the two sides into a formula. - The feature(s), x. Several are joined with
+:higher + failures. A dot.means “every other column”. - Which table the columns come from.
In words: “fit a linear model of final_grade explained by higher, using the students table.” You’ll run exactly this in Part 2.
On a UK keyboard it’s next to Enter, with Shift and the # key. On a Mac with a UK layout, try Option + N then Space, or the key left of 1 or Z. On a US keyboard it’s top left, with Shift.
The big idea done. Everything from here on uses these words: residual, RSS, coefficient, R², RMSE.
Part 1: Explore the data
About 20 minutes. Before modelling, look at the data. You’ll make a few plots with the ggplot skills from Lab 2.
Meet the students
students has one row per student, 649 students from two Portuguese secondary schools. The outcome is final_grade, from 0 to 20. The other 30 columns are features: things that might help predict the grade. They come in three groups:
| Group | Features |
|---|---|
| Who they are | school, sex, age, address, famsize, pstatus, medu, fedu, mjob, fjob, guardian |
| How they study | studytime, failures, schoolsup, famsup, paid, higher, absences, reason, traveltime |
| How they live | goout, freetime, romantic, activities, famrel, dalc, walc, health, internet, nursery |
Rows: 649
Columns: 31
$ school <chr> "GP", "GP", "GP", "GP", "GP", "GP", "GP", "GP", "G…
$ sex <chr> "F", "F", "F", "F", "F", "M", "M", "F", "M", "M", …
$ age <dbl> 18, 17, 15, 15, 16, 16, 16, 17, 15, 15, 15, 15, 15…
$ address <chr> "U", "U", "U", "U", "U", "U", "U", "U", "U", "U", …
$ famsize <chr> "GT3", "GT3", "LE3", "GT3", "GT3", "LE3", "LE3", "…
$ pstatus <chr> "A", "T", "T", "T", "T", "T", "T", "A", "A", "T", …
$ medu <chr> "Higher", "Primary", "Primary", "Higher", "Seconda…
$ fedu <chr> "Higher", "Primary", "Primary", "5th-9th", "Second…
$ mjob <chr> "at_home", "at_home", "at_home", "health", "other"…
$ fjob <chr> "teacher", "other", "other", "services", "other", …
$ reason <chr> "course", "course", "other", "home", "home", "repu…
$ guardian <chr> "mother", "father", "mother", "mother", "father", …
$ traveltime <chr> "15-30min", "<15min", "<15min", "<15min", "<15min"…
$ studytime <chr> "2-5hrs", "2-5hrs", "2-5hrs", "5-10hrs", "2-5hrs",…
$ failures <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ schoolsup <chr> "yes", "no", "yes", "no", "no", "no", "no", "yes",…
$ famsup <chr> "no", "yes", "no", "yes", "yes", "yes", "no", "yes…
$ paid <chr> "no", "no", "no", "no", "no", "no", "no", "no", "n…
$ activities <chr> "no", "no", "no", "yes", "no", "yes", "no", "no", …
$ nursery <chr> "yes", "no", "yes", "yes", "yes", "yes", "yes", "y…
$ higher <chr> "yes", "yes", "yes", "yes", "yes", "yes", "yes", "…
$ internet <chr> "no", "yes", "yes", "yes", "no", "yes", "yes", "no…
$ romantic <chr> "no", "no", "no", "yes", "no", "no", "no", "no", "…
$ famrel <chr> "Good", "Excellent", "Good", "Average", "Good", "E…
$ freetime <chr> "Average", "Average", "Average", "Low", "Average",…
$ goout <chr> "High", "Average", "Low", "Low", "Low", "Low", "Hi…
$ dalc <chr> "VeryLow", "VeryLow", "Low", "VeryLow", "VeryLow",…
$ walc <chr> "VeryLow", "VeryLow", "Average", "VeryLow", "Low",…
$ health <chr> "Average", "Average", "Average", "VeryGood", "Very…
$ absences <dbl> 4, 2, 6, 0, 0, 6, 0, 2, 0, 0, 2, 0, 0, 0, 0, 6, 10…
$ final_grade <dbl> 11, 11, 12, 14, 13, 13, 13, 13, 17, 13, 14, 13, 12…
Lines 76–106 describe features such as studytime, medu or health as numbers (1 to 4, 0 to 4, 1 to 5). In the file they are text labels: studytime is "<2hrs", "2-5hrs", "5-10hrs" or ">10hrs", and health runs from "VeryBad" to "VeryGood". Look at <chr> in the output above. Only age, failures, absences and final_grade are numbers.
Make some plots
The notebook lists ideas at lines 112–127. Make at least three plots in the chunk at line 131 (or add more chunks below it). Start with these two, then pick one idea of your own.
students |>
ggplot(aes(x = final_grade)) +
geom_histogram(bins = 20, alpha = 0.5, colour = "black") +
theme_histogram()
students |>
ggplot(aes(x = higher, y = final_grade)) +
geom_boxplot() +
theme_boxplot()The notebook asks you to share your most interesting finding on Moodle: post one plot and one sentence saying what it shows.
Grades cluster between 10 and 14, and a handful of students scored 0: remember them, they come back in Part 4. Students who plan to go on to higher education (higher = yes) score higher. Plots suggest; models measure. Part 2 puts a number on that difference.
Study time against grade (a boxplot per category):
students |>
ggplot(aes(x = studytime, y = final_grade)) +
geom_boxplot() +
theme_boxplot()Past failures (a number, so a jittered scatter):
students |>
ggplot(aes(x = failures, y = final_grade)) +
geom_jitter(width = 0.2, alpha = 0.4) +
theme_scatter()The theme functions are created in the setup chunk (lines 24–62). Run it again.
Part 2: The hypothesis-testing approach
About 20 minutes. This is how a social scientist uses a linear model: to ask whether an effect exists and how sure we can be.
One feature: does higher matter?
The hypothesis: students who want to go on to higher education get better grades. Fit a linear model of final_grade explained by higher, then ask for a summary.
linear_reg() |> fit(final_grade ~ higher, data = students) |> extract_fit_engine() |> summary()
- Unwrap the model. tidymodels wraps the fitted model in its own packaging. This pulls out the plain R model inside (called an
lm). - Print the classic regression table: coefficients, standard errors, p-values and R². It only works on the unwrapped model, hence step 1.
The code is already in the notebook. Run the chunk.
Call:
stats::lm(formula = final_grade ~ higher, data = data)
Residuals:
Min 1Q Median 3Q Max
-12.2759 -1.2759 -0.2759 1.7241 6.7241
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 8.7971 0.3671 23.962 <2e-16 ***
higheryes 3.4788 0.3883 8.958 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 3.05 on 647 degrees of freedom
Multiple R-squared: 0.1103, Adjusted R-squared: 0.109
F-statistic: 80.24 on 1 and 647 DF, p-value: < 2.2e-16
Reading the output
Focus on the Coefficients table. Each row is one coefficient; each column says something about it.
| Column | higheryes |
What it means |
|---|---|---|
| Estimate | 3.48 | The coefficient. Students who want higher education score 3.48 points higher. |
| (Intercept) Estimate | 8.80 | The prediction when higheryes is 0: the average grade of the “no” students. |
| Std. Error | 0.39 | How uncertain the estimate is. A different sample of students would give a slightly different 3.48; this is roughly how much it would wobble. |
| t value | 8.96 | Estimate ÷ standard error: the effect is about 9 standard errors away from zero. |
| Pr(>|t|) | < 2e-16 | The p-value: if the true effect were zero, how surprising would a result like this be? Here, extremely surprising, so we reject “no effect”. |
Then the line near the bottom: Multiple R-squared: 0.1103. The feature explains 11% of the variation in grades.
Significant is not the same as predictive. The effect of higher is about as certain as effects get (p < 0.001), yet it explains only 11% of the variation. A model can be great for testing a hypothesis and poor for predicting individual grades.
All features at once: ~ .
A single feature might be standing in for others. Students who want higher education may also study more and fail less. To control for that, put every feature in the model. The dot means “every other column”.
Run the chunk. The output is long: 67 coefficients. Scroll to the row for higheryes and to the R-squared line at the bottom.
The coefficient for higheryes halves, from 3.48 to about 1.6, but stays significant. Part of the raw difference was study time and past failures travelling together with the wish for higher education. R² rises to 0.41: together, the features explain 41% of the variation. Each past failure is associated with about 1.5 points less.
p-values tell you which features matter for explaining these 649 students. They are no guarantee that the model predicts well for new students. And they can be hunted for: trying many models and reporting only the significant one is called p-hacking. The notebook links the Data Falsificada series, about a famous case of faked data.
After adding all the other features, the coefficient for higheryes drops from 3.48 to 1.6. What’s the best reading?
Wanting higher education doesn’t matter after all.
It still matters: 1.6 points, and still significant. It’s smaller, not gone.
Part of the 3.48 was really other things, like study time and past failures, that go together with wanting higher education.
Yes. Holding those fixed, the remaining difference linked to higher is about 1.6 points.
The first model was calculated wrongly.
Both models are calculated correctly. They answer different questions: the raw difference, and the difference with other features held fixed.
Part 3: The machine learning approach
About 40 minutes. The most important part of the lab. Same model, different question: not “is there an effect?” but “how well can we predict a grade we haven’t seen?”
A made-up model, in R
The notebook starts with a made-up rule, like the grade rule from the big idea, written as an R function: a recipe with three inputs.
sim_model_preds <- function(x1, x2, x3) { y <- 1.1 * x1 + 2.2 * x2 + 3.3 * x3 y }
- Make a function with three inputs, called
x1,x2andx3. - The rule: three made-up coefficients, 1.1, 2.2 and 3.3. No data was used to choose them.
- The result the function gives back: the last line inside the curly brackets.
Run the chunk at lines 191–196 to create the function. Nothing is printed; it appears in the Environment pane.
In the empty chunk at line 201, make a prediction for a new observation with
x1 = 10,x2 = 20,x3 = 30:sim_model_preds(10, 20, 30)The real value was
y = 150. In the chunk at line 209, calculate the residual:150 - sim_model_preds(10, 20, 30)
[1] 154
[1] -4
The two discussion questions at lines 204 and 212 ask for exactly these numbers: the prediction is 154, and we are 4 points off.
Why split the data?
Imagine revising for an exam with last year’s paper, then being tested on… last year’s paper again. You’d score well, but the mark would say little about what you learned. You might just have memorised the answers.
A model fitted to some data has the same problem. Scored on the very rows it learned from, it flatters itself: part of what it learned is the quirks of those particular rows. So we split the data before fitting anything:
| Set | Share | Used for |
|---|---|---|
| Training set | 75%, 486 students | Fitting the model: the coefficients come from these rows only |
| Test set | 25%, 163 students | Scoring the finished model. Locked away until then |
A model that does well on the training set and much worse on the test set has learned noise rather than pattern. That is overfitting, and the gap between the two scores is its first warning sign.
Split the data
The chunk at lines 218–225 has three comments and no code. Under each comment goes one step.
set.seed(123)
- Make the random split repeatable. Which students end up in which set is decided at random. A seed fixes the random draw, so you get exactly the same split, and the same numbers, as this page.
students_split <- initial_split(students, prop = 0.75)
- Decide which rows go where. The result is a split object: a note of which rows are which, not yet two tables.
- The share for training: 75%. Between 70% and 80% is typical.
students_train <- training(students_split)
students_test <- testing(students_split)
- Pull out the training table from the split object.
- Pull out the test table.
Type the code under the three comments, so the chunk reads:
# Set a seed for reproducibility
set.seed(123)
# Split the data with 75% being used to train the model
students_split <- initial_split(students, prop = 0.75)
# Create tibbles of the training and test set
students_train <- training(students_split)
students_test <- testing(students_split)Run the whole chunk, then type students_split in the Console to print the split object.
<Training/Testing/Total>
<486/163/649>
486 rows for training, 163 for testing, 649 in total.
The seed must run immediately before the split, in the same chunk, and the chunk should be run once from the top. Running initial_split() again without the seed gives a different random split.
Fit the model on the training set
The same code as in Part 2, with two changes: it uses only students_train, and the result is saved under a name, mv_model (short for “multivariable model”), so we can use it later.
In the chunk at line 234, type:
mv_model <- linear_reg() |>
fit(final_grade ~ ., data = students_train)Run it. Nothing is printed, but mv_model appears in the Environment pane.
To see what was fitted, tidy() turns the coefficients into a table, one row per coefficient:
Evaluate on the training and test sets
The chunk at lines 246–256 is already filled in. Three new functions appear in it.
reg_metrics <- metric_set(rsq, rmse)
- Bundle two metrics into one new function.
rsqis R²,rmseis RMSE. The result,reg_metrics, computes both at once.
mv_model |> augment(new_data = students_test) |> reg_metrics(truth = final_grade, estimate = .pred)
- Predict.
augment()takes a table, runs every row through the model, and gives the table back with two new columns:.pred, the prediction, and.resid, actual − prediction. - Which column holds the real values.
- Which column holds the predictions: the
.predcolumnaugment()just added.
Step through the pipeline to see what each part does:
mv_model |>
augment(new_data = students_test) |>
reg_metrics(truth = final_grade, estimate = .pred)augment() adds .pred and .resid to the 163 test students. Here only those two and the real grade are shown; the other 31 columns are still there.
reg_metrics() compares the real grades with the predictions and returns one row per metric.
Run the chunk. It prints two tables: first for the training set (data the model has seen), then for the test set (data it hasn’t).
How can we interpret these results?
This is the discussion question at line 260.
| R² | RMSE | |
|---|---|---|
| Training set, 486 students the model has seen | 0.446 | 2.37 |
| Test set, 163 students it hasn’t | 0.211 | 3.04 |
On students it has seen, the model explains 45% of the variation in grades. On new students, only 21%, and a typical prediction is 3 grade points off. The model is better than guessing the average, but it flatters itself on its own training data. That gap is overfitting.
Why? The model has 67 coefficients fitted on 486 students. With that many, some will happen to describe noise in those particular students: patterns that aren’t there in new students. Part 5 is about keeping only the useful ones.
Which number would you report as “how well the model predicts grades”?
R² on the training set, 0.446
The training set is the exam the model revised from. Its score is too optimistic.
R² and RMSE on the test set: 0.211 and 3.04
Yes. Only the test set shows how the model does on students it has never seen.
The R² of 0.41 from Part 2
That model used all 649 students to fit and to score, so it has the same problem as the training-set score.
Part 4: Where do we make errors?
About 20 minutes. R² and RMSE summarise the errors in one number. A residual plot shows them all, so you can see where the model goes wrong.
A near-perfect relationship, simulated
To know what a good residual plot looks like, start with data where a straight line fits almost perfectly. Such data rarely exist in the social world, so the notebook simulates them: it asks the computer to make up 1,000 pairs of numbers with a correlation of 0.99.
- Run the chunk at lines 268–287. It creates
sim_datawith two columns,V1andV2. You don’t need to understandmvrnorm(); it’s the function that draws correlated random numbers. - Run the chunk at lines 291–298 to plot them.
Residual plots
A residual plot puts each observation’s prediction across and its residual up.
... |> augment(new_data = sim_data) |> ggplot(aes(.pred, .resid)) + geom_hline(yintercept = 0, linetype = "dashed") + geom_point()
- Across: the prediction for each observation, from
augment(). - Up: the residual, actual − prediction. Above the dashed line, the model guessed too low; below it, too high.
- The “no error” line, dashed, at zero.
- Run the chunk at lines 302–312: a model of
V2explained byV1, then its residual plot. - Run the chunk at lines 316–324: the same plot for
mv_modelon the test set.
A good model leaves a shapeless cloud around zero: nothing left to explain. Any pattern is something the model missed.
Why does the simulated model look better?
This is the action point at line 326. Compare the two plots: look at the scale of the vertical axis, and at the bottom left of ours.
The simulated residuals are tiny (about ±0.25) and shapeless. Ours spread from about −15 to +5, so the model often misses by several grade points. And there is a diagonal streak of points at the bottom left.
What is the diagonal streak at the bottom left of our residual plot?
Students whose real grade was 0.
Yes. If the real grade is 0, the residual is 0 − prediction, so the higher the prediction, the lower the residual: a straight diagonal line. You saw these students in the Part 1 histogram. No linear rule from these features predicts a 0.
A bug in the code.
The code is fine. Think about which real grade would give residual = −prediction exactly.
Students in the training set that leaked into the test set.
The split keeps the two sets apart. The streak comes from the grades themselves.
Part 5: The lasso, a penalty for every coefficient
About 45 minutes. Part 3 ended with a problem: 67 coefficients, 486 students, and a model that does much worse on new students. The lasso is a version of linear regression that keeps only the coefficients that earn their place.
The idea: charge rent for every coefficient
How could you stop a model from using coefficients it doesn’t need? One way is by hand: drop the features with large p-values and refit. That’s slow, subjective, and the p-values come from the model you’re about to change. The lasso does it inside the algorithm, by changing the score it minimises.
Lasso minimises RSS + λ · Σ |βⱼ|
- Fit the data, exactly as before. OLS minimises this and nothing else.
- Lambda: the price per unit of coefficient. A number you choose, 0 or more. The notebook calls it
penalty. - The total size of all coefficients, signs ignored. |·| is the absolute value: |−1.4| = 1.4. Σ means “add up”. This is the bill.
Every coefficient now has to earn its place. It is only kept at a size where the improvement in fit is worth the rent. A coefficient that barely helps shrinks, often all the way to zero. One that helps a lot stays large.
Turning the dial
Here is the lasso fitted to the training set at three values of λ. Each bar is one of the 67 coefficients, sized by how big it is.
- λ = 0: no rent. That’s ordinary least squares: all 67 coefficients in use, with the overfitting from Part 3.
- λ = 0.1: coefficients shrink, and 27 of them are set to exactly zero. The lasso has chosen features for us.
- λ = 1: only 2 survive. Turn the dial further and every coefficient goes to zero: the model predicts the average for everyone.
Too little penalty and we’re back to overfitting; too much and the model knows nothing. The best λ is somewhere in between.
Why exactly zero?
Each coefficient feels two forces. The data pull it away from zero; that pull gets weaker as the coefficient approaches the size that fits best. The penalty pushes it back toward zero.
The lasso’s rent, |β|, is a V with a sharp kink at zero. Its slope is the same size everywhere, so the push toward zero is always λ, however small the coefficient already is. If the data’s pull on a coefficient is weaker than λ, the push wins all the way to exactly 0.
For contrast, the ridge penalty uses β². It flattens out near zero, so its push fades as the coefficient shrinks: coefficients get tiny but never reach zero. In tidymodels, mixture = 1 means lasso and mixture = 0 means ridge.
Parameters and hyperparameters
| Parameters | Hyperparameters | |
|---|---|---|
| Today | The coefficients, the βs: 67 of them | λ (penalty) and mixture |
| Chosen by | The algorithm, from the data | You, before fitting |
| Picture | Inside the machine | A knob on the outside of the machine |
Why can’t the algorithm choose λ itself? Because any λ above zero makes the fit to the training data worse, so it would always pick 0. To choose λ, try several values and score each one on data the model hasn’t seen. You’ll do that by hand in the bonus; Lab 5 automates it.
Which of these is a hyperparameter?
The coefficient of higheryes
That’s a parameter: the algorithm finds it from the data.
The penalty, λ
Yes. You set it before fitting; the algorithm takes it as given.
The test-set RMSE
That’s a metric: a score for the finished model, not a setting of the model.
Step 1: Fit a lasso
lasso_model <- linear_reg(penalty = 0.1, mixture = 1) |> set_engine("glmnet") |> fit(final_grade ~ ., data = students_train)
- λ, the rent per unit of coefficient.
- Which penalty: 1 is the lasso, 0 is the ridge, in between mixes the two.
- Which package does the computing. tidymodels hands the work to an “engine”. The default,
lm, can’t penalise;glmnetcan.
The rest is the same fit() as before, on the training set.
The code is already there. Run the chunk.
Step 2: Which coefficients survived?
tidy() gives one row per coefficient. Keep only those the lasso didn’t set to zero with filter(). != means “is not equal to”.
tidy(lasso_model) |>
filter(estimate != 0)The action point at line 351 asks three questions. 41 rows survive: the intercept plus 40 coefficients, so 27 were shrunk to zero. To find the most important feature, sort by size: add |> arrange(desc(abs(estimate))). The largest is higheryes, about 1.8, followed by failures (−1.37 per past failure) and schoolsupyes (−1.05).
Flip the filter: filter(estimate == 0) lists the 27 coefficients the lasso dropped.
Step 3: A bar plot of the coefficients
The bar plot shows every coefficient’s size, sorted, coloured by whether it is positive. Step through it:
tidy(lasso_model) |>
filter(term != "(Intercept)") |>
ggplot(aes(
abs(estimate),
fct_reorder(term, abs(estimate)),
fill = estimate > 0
)) +
geom_col() +
theme_bar() +
scale_fill_jama() +
labs(x = "Lasso coefficient", y = "Feature", fill = "Positive?")Start from the coefficient table and drop the intercept, which isn’t a feature.
abs(estimate) makes the bar length the size, ignoring the sign. fct_reorder(term, abs(estimate)) sorts the features by that size, so the biggest is on top. fill = estimate > 0 colours each bar by whether the coefficient is positive: TRUE or FALSE. geom_col() draws one bar per row.
Then the finishing touches: the bar theme from Lab 2, a colour scheme from ggsci, and labels.
Type the code into the chunk at line 356 and run it. The zero coefficients appear at the bottom with no bar.
Step 4: Evaluate on the test set
The evaluation code is exactly the same as in Part 3, with lasso_model instead of mv_model.
lasso_model |>
augment(new_data = students_test) |>
reg_metrics(truth = final_grade, estimate = .pred)The discussion question at line 369: is this an improvement?
| Test set, 163 students | Coefficients | R² | RMSE |
|---|---|---|---|
| Linear model, Part 3 | 67 | 0.211 | 3.04 |
| Lasso, penalty 0.1 | 40 | 0.240 | 2.95 |
Yes, a modest one. The lasso predicts new students slightly better with 27 fewer coefficients. A simpler model that generalises a little better: that’s the point of regularisation.
Bonus, Step 5: try other penalties
Copy Step 1 into the chunk at line 376 with a different penalty, then add the evaluation from Step 4. Try 0.01, 0.3 and 1. Which gives the lowest test RMSE?
linear_reg(penalty = 0.3, mixture = 1) |>
set_engine("glmnet") |>
fit(final_grade ~ ., data = students_train) |>
augment(new_data = students_test) |>
reg_metrics(truth = final_grade, estimate = .pred)Here are the results for six penalties:
Tiny penalties (0.0001 to 0.01) behave almost like plain OLS. 0.1 is best, at 2.95. Beyond it, the model drops too much: 3.04 with 14 coefficients at 0.3, and 3.29 with only 2 at 1. Another bowl, but this time scored on test data. Choosing a hyperparameter this way is called tuning.
The solutions build every model at once with nested tibbles and map2(), the tools from Lab 1 Part 3. The idea is the same as copying Step 1 four times: one model per penalty, one RMSE per model. Lab 5 introduces a cleaner way, cross-validation.
Wrap-up
What you learned
The algorithm Ordinary least squares picks the line with the smallest sum of squared residuals. A coefficient is the change in the prediction when its feature rises by one.
Two approaches Hypothesis testing asks whether an effect exists and uses all the data. Machine learning asks how well we predict new cases, so it splits the data.
The workflow initial_split(), then linear_reg() |> fit() on training, then augment() and metric_set(rsq, rmse) on test. The same steps for every model from now on.
The lasso Adds a price per coefficient. λ is a hyperparameter you choose; coefficients that aren’t worth the rent go to exactly zero.
One question to take away: before today, which of the 30 features would you have bet on to predict grades? Did the lasso agree?
Further reading
- An Introduction to Statistical Learning, free online: chapter 3 on linear regression and section 6.2 on the lasso. The standard textbook for this course’s ideas.
- Tidy Modeling with R, free online: the chapters on fitting models with
parsnipand judging them withyardstick.
Glossary
Underlined words on this page are explained in this list. Click or tap one anywhere on the page to see what it means and where it was first explained.
| Word | Meaning |
|---|---|
| algorithm | A recipe a computer follows to learn something from data, like choosing a model’s coefficients |
| coefficient | A number in a model’s rule that the algorithm chooses. Also called a parameter |
| feature | An input used to make a prediction: a column other than the outcome |
| formula | How R writes a model: outcome ~ features, e.g. final_grade ~ higher |
| hyperparameter | A setting you choose before fitting, such as the lasso’s penalty λ |
| intercept | The prediction when every feature is 0 |
| lasso | Linear regression with a price per unit of coefficient. Sets unhelpful coefficients to exactly zero |
| ordinary least squares | The algorithm that picks the line with the smallest sum of squared residuals |
| outcome | What the model predicts. Today: final_grade |
| overfitting | Learning noise in the training data, so the model does much worse on new data |
| p-value | If the true effect were zero, how surprising a result like this would be. Small means surprising |
| penalty | The lasso’s price per unit of coefficient, λ. Called penalty in tidymodels |
| R² | The share of the variation in the outcome the model explains, from 0 to 1 |
| regularisation | Adding a penalty to keep a model simple, so it generalises better |
| residual | Actual − predicted, for one observation |
| RMSE | Root mean squared error: the typical size of a miss, in the units of the outcome |
| RSS | Residual sum of squares: square every residual, add them up. OLS makes it as small as possible |
| seed | A starting number that makes random steps, like the split, give the same result every time |
| slope | How much the prediction changes when a feature rises by one unit |
| standard error | How uncertain an estimate is: roughly how much it would vary with another sample |
| test set | The data held back to score the finished model |
| tidymodels | A family of R packages for modelling, loaded with library("tidymodels") |
| training set | The data the model is fitted on |