MY202 · Lab 5
Jonas Weinert
Department of Methodology | London School of Economics and Political Science
29 October 2026
| Part | Topic | Functions | Mode | Time |
|---|---|---|---|---|
| Setup | together | 3 min | ||
| I | k nearest neighbours and cross-validation | vfold_cv(), tune(), recipe(), workflow(), tune_grid(), select_best() |
concept → together, twice, then your turn | 37 min |
| II | A tree, then a forest, then boosting: each a game, code together, your turn | decision_tree(), rpart.plot(), rand_forest(), boost_tree() |
game → together → your turn, three times | 46 min |
| Wrap-up (+ 1 min spare) | 3 min |
Every block runs the same way: a concept with an everyday example, then code together (we type it into the notebook; the line number says where), then your turn (a timer, tasks and a syntax sheet, then the solutions step by step).
Part I
33,234 survey respondents · two 0–10 questions · PTSD yes or no
Notebook lines 44–70 read_csv(), then the plot of the feature spacerun together
ptsd, the outcome: 1 = yes, 0 = noTwo features, both 0–10 anxiety and acute_stress. Only 121 combinations: each dot is a cell of about 275 people, coloured by the share with PTSD.
Where would you draw the line between “yes” and “no”?
| Flat | Size (m²) | Walk to Tube | Sold within a month? |
| A | 58 | 5 min | yes |
| B | 52 | 8 min | yes |
| C | 61 | 7 min | no |
| D | 50 | 4 min | yes |
| E | 57 | 9 min | no |
| F | 95 | 6 min | no |
| G | 55 | 20 min | no |
| H | 40 | 15 min | yes |
| New flat | 55 | 6 min | ? |
55 m², six minutes from the Tube: will it sell within a month? You have last year’s sales.
Find the most similar flats The agent pulls the comparables, the sales most like this one: A to E. F is far bigger, G and H far from the Tube.
That is the whole algorithm k nearest neighbours: the k most similar rows vote. Here k = 5.
160 flats: each a dot; the diamond is the new one.
k = 1: the single most similar One flat decides. Fast, and at the mercy of one odd sale.
k = 5, k = 15: a wider circle Steadier. Here the answer flips between k = 1 and k = 15: same flat, different prediction.
k is a hyperparameter The data can’t say what k should be. Like λ in Lab 3, we choose it.
Colour every point by its vote: the decision boundary.
k = 1: an island for every flat Every training flat is classified right, and every odd sale becomes a rule.
k = 5, k = 50: smoother More voters average the oddities away; at k = 160 the whole picture is the majority colour.
Which k is right? Not the picture’s call: we need a score per k.
Solutions notebook lines 61–79 "for tutors to run": four kNN models fitted in advance; not in the student workbookdemo
Same picture, real data Anxiety and acute stress instead of size and minutes; shaded = predicted PTSD. Every dot is a cell of about 275 people.
Small k, jagged; large k, smooth k = 100 can’t see past its own cell and barely finds the “yes” region. k = 1,000 draws a broad diagonal band: high anxiety and high stress.
Which k, then? The picture can’t say. We need a fair score for each k: next.
Obvious plan: try every k, score each on the 160 flats, keep the best. What goes wrong?
Accuracy on the 160 training flats share of flats classified right
Accuracy on flats held out of the fit 5-fold cross-validation, next slide
k = 1 always wins, and it’s a lie Each flat’s nearest neighbour is itself, so k = 1 gets every training flat right. The training score rewards memorising.
The test set can’t choose either Last week’s rule: the test set is the final exam, opened once. Use it to pick k and its score is no longer honest.
Score on flats the fit never saw Hold some training flats back, fit on the rest, score on the held-out ones. Now k = 1 is exposed and k = 50 wins.
Five past papers Cut the training set into 5 folds, vfold_cv(ptsd_train, v = 5). The test set is not in the picture: it’s the real exam, sealed until the end.
Revise on four, sit the fifth as a mock Each round fits on 4 folds and scores the 5th. Rotate, so every row is a mock question once. Five honest scores, none from the real exam.
Average, then pick One mean score per k. Repeat for every k on the list, keep the best, refit once on all five folds. Then, once, the test set.
Notebook lines 107–111 an empty chunk: type the split, then the foldscode together
set.seed(123) ptsd_split <- initial_split(ptsd, prop = 0.75, strata = ptsd) ptsd_train <- training(ptsd_split) ptsd_test <- testing(ptsd_split) ptsd_cv <- vfold_cv(ptsd_train, v = 5) ptsd_cv
v, not k: k is taken by the neighbours today.# 5-fold cross-validation
# A tibble: 5 × 2
splits id
<list> <chr>
1 <split [19939/4985]> Fold1
2 <split [19939/4985]> Fold2
3 <split [19939/4985]> Fold3
4 <split [19939/4985]> Fold4
# ℹ 1 more row
Read it Five rows, one per round: 19,939 rows to fit on, 4,985 held out.
Notebook lines 113–119 the empty chunk under "Instantiate k-NN model"code together
knn_model <- nearest_neighbor(neighbors = tune()) |> set_mode("classification") |> set_engine("kknn")
linear_reg() and logistic_reg().kknn package finds the neighbours. It loads itself when the model is fitted.Nothing is fitted yet knn_model is a description, not a model. Print it: “neighbors = tune()”.
Next: one more idea before the recipe kNN measures distances. Distances in what units?
Notebook line 123 "You will need to normalise both predictors as k-NN uses distances." Why?concept
Flat F is the same distance from the Tube but 40 m² bigger. Flat G is the same size but 14 minutes further. Which is more similar to the new flat?
The answer depends on the units Distance adds the two gaps together, so whichever feature has the bigger numbers wins. That can’t be right.
So measure every feature in SDs Subtract the mean, divide by the standard deviation: “how unusual is this gap?” That’s step_normalize() in the recipe, and it’s why kNN needs one.
Notebook lines 121–129 the note at line 123 says why: "k-NN uses distances"code together
knn_rec <- recipe(ptsd ~ ., data = ptsd_train) |> step_normalize(all_numeric_predictors()) wf_knn <- workflow() |> add_model(knn_model) |> add_recipe(knn_rec)
fit() on a workflow runs both, and new data gets exactly the same treatment.Notebook lines 133–152 metrics and grid in the chunk at 133; tune_grid() is sketched at 143–150: fill the four brackets, remove the #code together
class_metrics <- metric_set(recall, precision, f_meas) grid_knn <- tibble(neighbors = c(100, 500, 1000, 3000)) doParallel::registerDoParallel() knn_model_cv <- tune_grid(wf_knn, resamples = ptsd_cv, metrics = class_metrics, grid = grid_knn, control = control_resamples(event_level = "second"))
control.tune_grid() is running. How many kNN models is it fitting, and which rows does each one see?
finalize_workflow() |> fit() in the exercise), and that one meets the test set.Part I · lines 156–198
knn_model_cv |> collect_metrics() on its own: one row per k and metric. Then run the chunk: the mean metric against k, one colour per metric.select_best() by F1, then finalise the workflow and fit on the whole training set. Half a minute.Syntax you need
knn_model_cv |> collect_metrics() # mean of the 5 rounds, per k
knn_model_cv |> collect_metrics() |>
mutate(neighbors = as.factor(neighbors)) |> # evenly spaced
ggplot(aes(neighbors, mean, colour = .metric,
group = .metric)) + # one line per metric
geom_point() + geom_line(linetype = "dashed")
highest_f1 <- select_best(knn_model_cv, metric = "f_meas")
knn_model_final <- wf_knn |>
finalize_workflow(highest_f1) |> # fill in tune()
fit(ptsd_train) # the one model we keep
knn_model_final |>
augment(new_data = ptsd_test) |> # as Lab 4
f_meas(truth = ptsd, estimate = .pred_class,
event_level = "second")Stuck?
group = .metric is what joins the dots: without it, no linesevent_level = "second" is missingknn_model_cv |> collect_metrics()
knn_model_cv |> collect_metrics() |>
mutate(neighbors = as.factor(neighbors)) |>
ggplot(aes(neighbors, mean, colour = .metric,
group = .metric)) +
geom_point(size = 2.5) + geom_line(linetype = "dashed")
highest_f1 <- select_best(knn_model_cv, metric = "f_meas")
knn_model_final <- wf_knn |>
finalize_workflow(highest_f1) |>
fit(ptsd_train)
knn_model_final |>
augment(new_data = ptsd_train) |>
f_meas(truth = ptsd, estimate = .pred_class, event_level = "second")
knn_model_final |>
augment(new_data = ptsd_test) |>
f_meas(truth = ptsd, estimate = .pred_class, event_level = "second")Read the cross-validated scores, pick the best k, refit once, then the test set.
12 rows: 4 values of k × 3 metrics. mean is the average of the 5 rounds, n says 5, std_err how much the rounds disagreed. Shown wide here to fit.
# A tibble: 4 × 4
neighbors f_meas precision recall
<dbl> <dbl> <dbl> <dbl>
1 100 0.085 0.67 0.046
2 500 0.625 0.7 0.565
3 1000 0.733 0.705 0.763
4 3000 0.736 0.704 0.771
Recall climbs with k; precision is flat; F1 is best at 3,000 and 1,000 is almost as good. k = 100 is useless here (recall 0.05).
select_best() returns the winning row of the grid: neighbors = 3,000.
# A tibble: 1 × 2
neighbors .config
<dbl> <chr>
1 3000 pre0_mod4_post0
finalize_workflow() swaps tune() for 3,000; fit() then fits that model once, on all 24,924 training rows. The 20 tuning fits are gone.
== Workflow [trained] ==================
Preprocessor: Recipe
Model: nearest_neighbor()
neighbors = 3000
kernel = optimal
...
Training F1 0.736, test F1 0.737, cross-validated 0.736: all the same. No overfitting, and the cross-validated score predicted the test score well.
# A tibble: 2 × 4
set .metric .estimator .estimate
<chr> <chr> <chr> <dbl>
1 train f_meas binary 0.736
2 test f_meas binary 0.737
F1 for the chosen k = 3,000 three sets of rows, three jobs
Cross-validation did its job It ranked the candidates on training rows only, and its score for the winner matched the test set. The test set was opened once.
Why large k won Every dot is a cell of 275 people. k = 100 can’t see past its own cell; k = 3,000 averages a neighbourhood of cells. With two 0–10 features, the cell majority is already as good as it gets: F1 0.739.
kNN has no coefficients Nothing to read off, no “per unit of anxiety”. Flexible, and slow to predict. Trees, next, are flexible and readable.
Part II
2,904 US counties · 2008–2018 · the weekly cost of childcare
Notebook lines 296–318 read_csv(), the split by year and the recipe: all given, run them nowrun together
mcsaBack to regression A dollar outcome: Lab 3’s RMSE and R², and set_mode("regression") on every model.
Time is in the data Each county appears year after year. Train on 2008–2016, test on 2017–2018: predict the future.
Notebook lines 303–309 already run with read_csv(): what the two filter() lines didread together
childcare_train <- childcare |> filter(study_year <= 2016) childcare_test <- childcare |> filter(study_year > 2016)
initial_split(), no seed: the calendar decides.Folds would need the same care vfold_cv() shuffles rows. With years: fit on the past, score the next (sliding_period()). Not needed today.
Why not a random split? Each county appears every year. A random split puts a county’s 2017 row in training and its 2018 row in the test set: the model has nearly seen the answer, and the score flatters it.
The honest question Next year’s prices from this year’s data: predict the future. The test years are a little harder, because prices rose.
| Person | Age | Likes pop | Plays guitar | Enjoys lyrics | Follows trends | Swiftie? |
| A | 19 | yes | no | yes | yes | Swiftie |
| B | 27 | yes | yes | yes | no | Swiftie |
| C | 20 | no | yes | yes | no | not |
| D | 24 | yes | no | yes | yes | Swiftie |
| E | 22 | no | yes | yes | no | not |
| F | 29 | no | yes | no | yes | not |
| G | 18 | yes | no | yes | yes | Swiftie |
| H | 21 | no | no | yes | no | not |
| I | 26 | yes | yes | yes | yes | Swiftie |
| J | 25 | no | no | no | yes | not |
| K | 23 | yes | yes | no | no | not |
| L | 31 | no | no | yes | yes | Swiftie |
Ask one yes/no question, then guess “Swiftie” or “not” for each group. Which question gets the most right?
The first split of a decision tree Try every question (every cut-off, for age). Keep the one that sorts cleanest.
Pop fans: one more question Among the six pop fans, “enjoys lyrics?” sorts all six. K is the pop fan who doesn’t care about lyrics.
Each group gets its own question The second question is asked inside a group, not of everyone. Non-pop fans: no question tidies them up, so the guess stays “not”.
A decision tree A flow chart of yes/no questions, a guess in every leaf. Readable by anyone: that’s its big selling point.
The six non-pop fans
| Person | Age | Plays guitar | Enjoys lyrics | Follows trends | Swiftie? |
| C | 20 | yes | yes | no | not |
| E | 22 | yes | yes | no | not |
| F | 29 | yes | no | yes | not |
| H | 21 | no | yes | no | not |
| J | 25 | no | no | yes | not |
| L | 31 | no | yes | yes | Swiftie |
Can a third question find L too?
Yes: one does L is 31, the oldest. “Age 30 or over?” makes the tree perfect on the table.
Would you bet that non-pop fans over 30 are Swifties?
No: a question invented for one person Ask until every leaf is pure and the tree memorises the table: overfitting. Real trees stop early (min_n, cost_complexity).
The question is a cut-off Try every income as a cut-off, like age in the game. The cleanest: income below $59.2k?
The guess is the mean Each side predicts its average price: $93 and $125. “Cleanest” = the smallest squared error, Lab 3’s RMSE.
Ask again inside each side A staircase. decision_tree() plays this on all 52 columns.
Notebook lines 314–327 the recipe (already run), then the Task 1 chunk: type the spec and the fitcode together
childcare_rec <- recipe(mcsa ~ ., data = childcare_train) |> update_role(starts_with(c("county", "state")), new_role = "id") dt_model <- decision_tree() |> set_mode("regression") |> set_engine("rpart") dt_fit <- workflow() |> add_recipe(childcare_rec) |> add_model(dt_model) |> fit(data = childcare_train)
id labels, out of the model. The year stays in.rpart’s built-in brakes.tune(), so fit straight away: Part I’s pattern without the folds.No normalising this time A tree asks “income below $47.7k?”. A cut-off doesn’t care about units, so no step_normalize(). kNN measured distances; trees don’t.
One recipe, three models The forest and the boosted trees reuse childcare_rec. Only the model line changes.
Part II · Task 1 · lines 324–338
dt_fit with rpart.plot(). Find the first question.metric_set(rsq, rmse).Syntax you need
rpart.plot(extract_fit_engine(dt_fit), # the rpart tree inside
roundint = FALSE)
regress_metrics <- metric_set(rsq, rmse) # Lab 3's scores
dt_fit |>
augment(new_data = childcare_train) |> # adds .pred
regress_metrics(mcsa, .pred) # truth, estimate
dt_fit |>
augment(new_data = childcare_test) |>
regress_metrics(mcsa, .pred)Stuck?
rpart.plot(extract_fit_engine(dt_fit), roundint = FALSE)
regress_metrics <- metric_set(rsq, rmse)
dt_fit |>
augment(new_data = childcare_train) |>
regress_metrics(mcsa, .pred)
dt_fit |>
augment(new_data = childcare_test) |>
regress_metrics(mcsa, .pred)Draw it, then score: training set, then test set.
The Swiftie game on 52 columns. First question one_race_a < 2.2, then income. 6 columns used, 9 leaves from $80 to $224.
Training set: RMSE 27.5 dollars a week, R² 0.32. 9 numbers can only explain so much.
# A tibble: 2 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 rsq standard 0.320
2 rmse standard 27.5
Test set: RMSE 31.5, R² 0.32. A little worse than training; predicting the mean would give 39.1. The baseline the next two models must beat.
# A tibble: 2 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 rsq standard 0.323
2 rmse standard 31.5
Friend 1saw A B C D E G H I
may ask: plays guitar or follows trends
Mia: Swiftie
Friend 2saw A B E F I J K L
may ask: follows trends or age
Mia: not
Friend 3saw A B D E F I K L
may ask: age or enjoys lyrics
Mia: Swiftie
Each friend sees less 8 of the 12 people (a random resample) and 2 of the 5 questions (random columns). Different evidence, a different first question.
Mia: 22, not a pop fan, plays guitar, enjoys lyrics, follows trends Votes: Swiftie, not, Swiftie. Swiftie, 2 to 1. Our one tree said “not”: not a pop fan.
That is a random forest 1,000 such friends. Each tree on a resample of the rows, each split from a few random columns (mtry). They vote; for a price, they average.
One friend, one resample Draw the counties at random, with replacement; grow a tree. One staircase.
Three friends, three staircases Each resample has different counties, so the steps land in different places.
Average them: smooth Each tree’s odd steps cancel out in the average of 50. The forest overfits far less than any one deep tree.
Notebook lines 348–351 the Task 2 sketch: fill in the brackets, remove the #code together
rf_model <- rand_forest(mtry = round(sqrt(52), 0), trees = 1e3) |> set_mode("regression") |> set_engine("ranger", num.threads = parallel::detectCores() - 1)
mtry: columns per split. Each split may look at 7 random columns of the 52, the friends’ “2 of 5”. √52 ≈ 7 is a rule of thumb.trees: how many friends. 1,000; more only costs time.ranger grows the trees; num.threads uses all cores but one, about 20 seconds instead of a minute.Part II · Task 2 · lines 347–358
rf_model in a workflow with childcare_rec; call it rf_fit. Takes about 20 seconds.Task 1’s pattern, new model
rf_fit <- workflow() |>
add_recipe(childcare_rec) |>
add_model(rf_model) |> # was dt_model
fit(data = childcare_train)
rf_fit |>
augment(new_data = childcare_test) |>
regress_metrics(mcsa, .pred)
# and the same with childcare_trainStuck? Training RMSE far below the test one? That’s normal for a forest: see the solution.
rf_fit <- workflow() |>
add_recipe(childcare_rec) |>
add_model(rf_model) |>
fit(data = childcare_train)
rf_fit |> augment(new_data = childcare_train) |>
regress_metrics(mcsa, .pred)
rf_fit |> augment(new_data = childcare_test) |>
regress_metrics(mcsa, .pred)Fit, then score both sets.
1,000 trees, each on a resample, each split choosing among 7 random columns. About 20 seconds.
== Workflow [trained] ==================
Preprocessor: Recipe
Model: rand_forest()
Ranger result
Type: Regression
Number of trees: 1000
Mtry: 7
...
Test RMSE 23.9 against the tree’s 31.5: much better. Training 6.6: tiny, and that’s normal. Every row was in most trees’ resamples, so the forest has seen it. Judge it on the test set.
# A tibble: 4 × 4
set .metric .estimator .estimate
<chr> <chr> <chr> <dbl>
1 train rsq standard 0.976
2 train rmse standard 6.61
3 test rsq standard 0.64
4 test rmse standard 23.9
| Person | Swiftie? | Friend 1 likes pop? |
Friend 2 enjoys lyrics? |
Friend 3 follows trends? |
Together 2 of 3 |
| A | Swiftie | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ |
| B | Swiftie | Swiftie ✓ | Swiftie ✓ | not ✗ | Swiftie ✓ |
| C | not | not ✓ | Swiftie ✗ | not ✓ | not ✓ |
| D | Swiftie | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ |
| E | not | not ✓ | Swiftie ✗ | not ✓ | not ✓ |
| F | not | not ✓ | not ✓ | Swiftie ✗ | not ✓ |
| G | Swiftie | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ |
| H | not | not ✓ | Swiftie ✗ | not ✓ | not ✓ |
| I | Swiftie | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ |
| J | not | not ✓ | not ✓ | Swiftie ✗ | not ✓ |
| K | not | Swiftie ✗ | not ✓ | not ✓ | not ✓ |
| L | Swiftie | not ✗ | Swiftie ✓ | Swiftie ✓ | Swiftie ✓ |
Friend 1 plays the game “Likes pop?”: 10 of 12, wrong about K and L.
Friend 2: “K and L count 4×” The best question now gets K and L right: lyrics? Wrong about C, E, H.
Friend 3: “C, E, H, K, L count 4×” Trends? gets those five right. Wrong about B, F, J.
Together: 12 of 12 Nobody is wrong twice. Boosting: small trees in turn, each aimed at the last ones’ mistakes, then combined.
With a number, “the mistakes” are the leftovers Each small tree predicts how far off the trees so far still are, county by county.
Add only a tenth Each tree’s answer is added with a small say: learn_rate = 0.1. One step at a time, so no single tree can wreck it.
After 100 trees The relay has found the shape. Keep going for ever and it starts to chase single counties, like the question for L.
Notebook lines 365–372 the Task 3 sketch: fill in the brackets, remove the #code together
xgb_model <- boost_tree( mtry = round(sqrt(52), 0), trees = 1000, learn_rate = 0.1 ) |> set_mode("regression") |> set_engine("xgboost")
mtry as in the forest.trees: the length of the relay. 1,000 legs.learn_rate: each tree’s say. A tenth of its answer is added.Part II · Tasks 3–4 · lines 364–390
xgb_model as xgb_fit; score both sets (376). A few seconds.Task 4: three fits, one table
model_results <-
list("Decision Tree" = dt_fit,
"Random Forest" = rf_fit,
"XGBoost" = xgb_fit) |>
map_dfr(~ augment(.x, new_data = childcare_test),
.id = "model") |> # list names → a column
group_by(model) |>
regress_metrics(mcsa, .pred) |> # one score per model
filter(.metric == "rmse")
ggplot(model_results,
aes(reorder(model, .estimate), .estimate)) +
geom_col() + coord_flip()Stuck? map_dfr() is Lab 4’s threshold sweep: a list in, one stacked table out.
xgb_fit <- workflow() |>
add_recipe(childcare_rec) |>
add_model(xgb_model) |>
fit(data = childcare_train)
xgb_fit |> augment(new_data = childcare_train) |>
regress_metrics(mcsa, .pred)
xgb_fit |> augment(new_data = childcare_test) |>
regress_metrics(mcsa, .pred)Fit, then score both sets.
1,000 small trees in sequence, each adding a tenth of its correction. A few seconds.
== Workflow [trained] ==================
Preprocessor: Recipe
Model: boost_tree()
Main Arguments:
mtry = 7
trees = 1000
learn_rate = 0.1
Computational engine: xgboost
...
Test RMSE 22.3, the best of the three (forest 23.9). Training 3.6: the same big gap as the forest, for the same reason.
# A tibble: 4 × 4
set .metric .estimator .estimate
<chr> <chr> <chr> <dbl>
1 train rsq standard 0.989
2 train rmse standard 3.59
3 test rsq standard 0.655
4 test rmse standard 22.3
model_results <-
list(
"Decision Tree" = dt_fit,
"Random Forest" = rf_fit,
"XGBoost" = xgb_fit
) |>
map_dfr(~ .x |> augment(new_data = childcare_test),
.id = "model") |>
group_by(model) |>
regress_metrics(mcsa, .pred) |>
filter(.metric == "rmse")
ggplot(model_results,
aes(x = reorder(model, .estimate),
y = .estimate, fill = model)) +
geom_col(show.legend = FALSE) +
geom_text(aes(label = round(.estimate, 1)),
hjust = -0.1) +
coord_flip() +
labs(x = NULL, y = "RMSE (lower is better)")Score the three fits on the test set in one go, then draw the result.
map_dfr() runs augment() on each fit and stacks the three tables; .id keeps the list names. group_by(model) then gives one RMSE per model.
# A tibble: 3 × 4
model .metric .estimator .estimate
<chr> <chr> <chr> <dbl>
1 Decision Tree rmse standard 31.5
2 Random Forest rmse standard 23.9
3 XGBoost rmse standard 22.4
reorder() sorts the bars by RMSE; coord_flip() turns them sideways so the names fit. The notebook adds a title, theme and colours.
| One tree | Random forest | Boosting | |
| In the game | "Pop?", then "lyrics?" | Three friends who each saw part of the table, voting | A relay: each friend aimed at the last one's mistakes |
| How the trees are grown | One, until the brakes stop it | 1,000 at once, independently: each on a resample, random columns | 1,000 small ones in turn, each on the leftovers |
| How they're combined | No combining | Average (or vote) | Sum, each with a small say |
| The knobs | tree_depth, min_n, cost_complexity |
trees, mtry |
trees, learn_rate, mtry |
| Can you read it? | Yes: a flow chart | No | No |
| Test RMSE (train) | 31.5 (27.5) | 23.9 (6.6) | 22.3 (3.6) |
Mind the gap An ensemble’s training error is always tiny: it has seen every row many times. The test set is the honest number. Part of the gap is time: prices rose in 2017–2018.
Nothing was tuned 1,000 trees, mtry 7, learning rate 0.1: rules of thumb. Part I’s tune() and cross-validation are how you’d do better.
fit() estimates: the number of neighbours, the depth of a tree, the number of trees, the learning rate. Choose it by cross-validation: cut the training set into folds, score every candidate on held-out folds, average, pick. The test set stays sealed until the end. In code: vfold_cv(), tune(), tune_grid(), select_best(), finalize_workflow()Question: you have the 2019 prices. Which model do you deploy, and how would you check whether the forest’s training RMSE of $7 means anything?
How was today’s lab? Scan the code, or open forms.gle/i3uobCAaGmkzf1ns7. It takes 30 seconds and changes what next week looks like.
MY202 · Lab 5 · Neighbours and trees