Neighbours and trees: tuning models with cross-validation

MY202 · Lab 5

Jonas Weinert

Department of Methodology  |  London School of Economics and Political Science

29 October 2026

Lab overview

Part Topic Functions Mode Time
Setup together 3 min
I k nearest neighbours and cross-validation vfold_cv(), tune(), recipe(), workflow(), tune_grid(), select_best() concept → together, twice, then your turn 37 min
II A tree, then a forest, then boosting: each a game, code together, your turn decision_tree(), rpart.plot(), rand_forest(), boost_tree() game → together → your turn, three times 46 min
Wrap-up (+ 1 min spare) 3 min

Every block runs the same way: a concept with an everyday example, then code together (we type it into the notebook; the line number says where), then your turn (a timer, tasks and a syntax sheet, then the solutions step by step).

k nearest neighbours

Part I

33,234 survey respondents · two 0–10 questions · PTSD yes or no

Meet the data: a feature space

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 44–70 read_csv(), then the plot of the feature spacerun together

33,234people, from an online health questionnaire in Mexico
34%have ptsd, the outcome: 1 = yes, 0 = no

Two features, both 0–10 anxiety and acute_stress. Only 121 combinations: each dot is a cell of about 275 people, coloured by the share with PTSD.

Where would you draw the line between “yes” and “no”?

How does an estate agent price a flat?

I · kNN + CVII · TreesForestsBoostingWrap-up
Flat Size (m²) Walk to Tube Sold within a month?
A 58 5 min yes
B 52 8 min yes
C 61 7 min no
D 50 4 min yes
E 57 9 min no
F 95 6 min no
G 55 20 min no
H 40 15 min yes
New flat 55 6 min ?

55 m², six minutes from the Tube: will it sell within a month? You have last year’s sales.

Find the most similar flats The agent pulls the comparables, the sales most like this one: A to E. F is far bigger, G and H far from the Tube.

5 most similarA, B, C, D, E
Their vote3 sold, 2 didn't → sells

That is the whole algorithm k nearest neighbours: the k most similar rows vote. Here k = 5.

k nearest neighbours: the neighbours vote

I · kNN + CVII · TreesForestsBoostingWrap-up

160 flats: each a dot; the diamond is the new one.

k = 1: the single most similar One flat decides. Fast, and at the mercy of one odd sale.

k = 5, k = 15: a wider circle Steadier. Here the answer flips between k = 1 and k = 15: same flat, different prediction.

k is a hyperparameter The data can’t say what k should be. Like λ in Lab 3, we choose it.

The boundary depends on k

I · kNN + CVII · TreesForestsBoostingWrap-up

Colour every point by its vote: the decision boundary.

k = 1: an island for every flat Every training flat is classified right, and every odd sale becomes a rule.

k = 5, k = 50: smoother More voters average the oddities away; at k = 160 the whole picture is the majority colour.

Which k is right? Not the picture’s call: we need a score per k.

Back to the PTSD data: the boundary depends on k

I · kNN + CVII · TreesForestsBoostingWrap-up

Solutions notebook lines 61–79 "for tutors to run": four kNN models fitted in advance; not in the student workbookdemo

Same picture, real data Anxiety and acute stress instead of size and minutes; shaded = predicted PTSD. Every dot is a cell of about 275 people.

Small k, jagged; large k, smooth k = 100 can’t see past its own cell and barely finds the “yes” region. k = 1,000 draws a broad diagonal band: high anxiety and high stress.

Which k, then? The picture can’t say. We need a fair score for each k: next.

Choosing k: the trap

I · kNN + CVII · TreesForestsBoostingWrap-up

Obvious plan: try every k, score each on the 160 flats, keep the best. What goes wrong?

Accuracy on the 160 training flats share of flats classified right

k = 1100%
k = 582%
k = 5082%

Accuracy on flats held out of the fit 5-fold cross-validation, next slide

k = 171%
k = 578%
k = 5080%

k = 1 always wins, and it’s a lie Each flat’s nearest neighbour is itself, so k = 1 gets every training flat right. The training score rewards memorising.

The test set can’t choose either Last week’s rule: the test set is the final exam, opened once. Use it to pick k and its score is no longer honest.

Score on flats the fit never saw Hold some training flats back, fit on the rest, score on the held-out ones. Now k = 1 is exposed and k = 50 wins.

Cross-validation: mock exams

I · kNN + CVII · TreesForestsBoostingWrap-up
fold 1fold 2fold 3fold 4fold 5score for this k
Round 1mockfitfitfitfitscore 1
Round 2fitmockfitfitfitscore 2
Round 3fitfitmockfitfitscore 3
Round 4fitfitfitmockfitscore 4
Round 5fitfitfitfitmockscore 5
mean of the 5

Five past papers Cut the training set into 5 folds, vfold_cv(ptsd_train, v = 5). The test set is not in the picture: it’s the real exam, sealed until the end.

Revise on four, sit the fifth as a mock Each round fits on 4 folds and scores the 5th. Rotate, so every row is a mock question once. Five honest scores, none from the real exam.

Average, then pick One mean score per k. Repeat for every k on the list, keep the best, refit once on all five folds. Then, once, the test set.

Split and folds

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 107–111 an empty chunk: type the split, then the foldscode together

set.seed(123) ptsd_split <- initial_split(ptsd, prop = 0.75, strata = ptsd) ptsd_train <- training(ptsd_split) ptsd_test <- testing(ptsd_split) ptsd_cv <- vfold_cv(ptsd_train, v = 5) ptsd_cv

  1. Last week’s split, stratified on the outcome so both sets have 34% with PTSD.
  2. Cut the training set into folds: the five past papers. The test set is not involved.
  3. Five of them. v, not k: k is taken by the neighbours today.
#  5-fold cross-validation 
# A tibble: 5 × 2
  splits               id   
  <list>               <chr>
1 <split [19939/4985]> Fold1
2 <split [19939/4985]> Fold2
3 <split [19939/4985]> Fold3
4 <split [19939/4985]> Fold4
# ℹ 1 more row

Read it Five rows, one per round: 19,939 rows to fit on, 4,985 held out.

The model: leave k open

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 113–119 the empty chunk under "Instantiate k-NN model"code together

knn_model <- nearest_neighbor(neighbors = tune()) |> set_mode("classification") |> set_engine("kknn")

  1. Which kind of model, like linear_reg() and logistic_reg().
  2. A placeholder for k: “cross-validation will fill this in”. The knob from the boundary slide.
  3. Say the job. kNN can classify (a vote) or predict a number (an average).
  4. The engine: the kknn package finds the neighbours. It loads itself when the model is fitted.

Nothing is fitted yet knn_model is a description, not a model. Print it: “neighbors = tune()”.

Next: one more idea before the recipe kNN measures distances. Distances in what units?

Why normalise? Units decide who is close

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook line 123 "You will need to normalise both predictors as k-NN uses distances." Why?concept

Flat F is the same distance from the Tube but 40 m² bigger. Flat G is the same size but 14 minutes further. Which is more similar to the new flat?

m² and minutesF: 40 · G: 14 G is "closer"
ft² and minutesF: 431 · G: 14 G, by a mile
m² and secondsF: 40 · G: 840 now F is closer
In standard deviationsF: 40 / 26 = 1.5 · G: 14 / 7.4 = 1.9 F is closer

The answer depends on the units Distance adds the two gaps together, so whichever feature has the bigger numbers wins. That can’t be right.

So measure every feature in SDs Subtract the mean, divide by the standard deviation: “how unusual is this gap?” That’s step_normalize() in the recipe, and it’s why kNN needs one.

Recipe and workflow

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 121–129 the note at line 123 says why: "k-NN uses distances"code together

knn_rec <- recipe(ptsd ~ ., data = ptsd_train) |> step_normalize(all_numeric_predictors()) wf_knn <- workflow() |> add_model(knn_model) |> add_recipe(knn_rec)

  1. A recipe: the formula plus the pre-processing to run before fitting.
  2. Every feature in standard deviations, the previous slide. Mean 0, SD 1, so distance is fair.
  3. Bundle model and recipe. fit() on a workflow runs both, and new data gets exactly the same treatment.

Metrics, candidates, and the tuning run

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 133–152 metrics and grid in the chunk at 133; tune_grid() is sketched at 143–150: fill the four brackets, remove the #code together

class_metrics <- metric_set(recall, precision, f_meas) grid_knn <- tibble(neighbors = c(100, 500, 1000, 3000)) doParallel::registerDoParallel() knn_model_cv <- tune_grid(wf_knn, resamples = ptsd_cv, metrics = class_metrics, grid = grid_knn, control = control_resamples(event_level = "second"))

  1. Three scores, as in Lab 4. We pick by F1.
  2. The candidates: one column per knob, one row per value to try.
  3. Use all the cores: the fits can run side by side.
  4. Every candidate on every fold: fit, score. 4 × 5 = 20 fits.
  5. The five rounds from the folds slide.
  6. Score the 1s, as last week, passed through control.

While it runs

I · kNN + CVII · TreesForestsBoostingWrap-up

tune_grid() is running. How many kNN models is it fitting, and which rows does each one see?

A4 models, one per k, each on the whole training set
B20 models: each k on each of the 5 rounds, fitted on 4 folds and scored on the 5th
C5 models, one per fold
D1 model, the best one
B. 4 candidates × 5 rounds. None of them is the final model: once the best k is known, the workflow is fitted once more on the whole training set (that's finalize_workflow() |> fit() in the exercise), and that one meets the test set.

Exercises: Part I

Part I · lines 156–198

  1. Line 158, given: first print knn_model_cv |> collect_metrics() on its own: one row per k and metric. Then run the chunk: the mean metric against k, one colour per metric.
  2. Lines 178–185, given: select_best() by F1, then finalise the workflow and fit on the whole training set. Half a minute.
  3. Line 192, yours: F1 on the test set, and on the training set for comparison.
  4. Line 198: which k won? Any overfitting?

Syntax you need

knn_model_cv |> collect_metrics()   # mean of the 5 rounds, per k

knn_model_cv |> collect_metrics() |>
  mutate(neighbors = as.factor(neighbors)) |>  # evenly spaced
  ggplot(aes(neighbors, mean, colour = .metric,
             group = .metric)) +                # one line per metric
  geom_point() + geom_line(linetype = "dashed")

highest_f1 <- select_best(knn_model_cv, metric = "f_meas")

knn_model_final <- wf_knn |>
  finalize_workflow(highest_f1) |>   # fill in tune()
  fit(ptsd_train)                    # the one model we keep

knn_model_final |>
  augment(new_data = ptsd_test) |>   # as Lab 4
  f_meas(truth = ptsd, estimate = .pred_class,
         event_level = "second")

Stuck?

  • group = .metric is what joins the dots: without it, no lines
  • Test F1 near 0.85? event_level = "second" is missing

Solutions: Part I

knn_model_cv |> collect_metrics()

knn_model_cv |> collect_metrics() |>
  mutate(neighbors = as.factor(neighbors)) |>
  ggplot(aes(neighbors, mean, colour = .metric,
             group = .metric)) +
  geom_point(size = 2.5) + geom_line(linetype = "dashed")

highest_f1 <- select_best(knn_model_cv, metric = "f_meas")

knn_model_final <- wf_knn |>
  finalize_workflow(highest_f1) |>
  fit(ptsd_train)

knn_model_final |>
  augment(new_data = ptsd_train) |>
  f_meas(truth = ptsd, estimate = .pred_class, event_level = "second")

knn_model_final |>
  augment(new_data = ptsd_test) |>
  f_meas(truth = ptsd, estimate = .pred_class, event_level = "second")

Read the cross-validated scores, pick the best k, refit once, then the test set.

12 rows: 4 values of k × 3 metrics. mean is the average of the 5 rounds, n says 5, std_err how much the rounds disagreed. Shown wide here to fit.

# A tibble: 4 × 4
  neighbors f_meas precision recall
      <dbl>  <dbl>     <dbl>  <dbl>
1       100  0.085     0.67   0.046
2       500  0.625     0.7    0.565
3      1000  0.733     0.705  0.763
4      3000  0.736     0.704  0.771

Recall climbs with k; precision is flat; F1 is best at 3,000 and 1,000 is almost as good. k = 100 is useless here (recall 0.05).

select_best() returns the winning row of the grid: neighbors = 3,000.

# A tibble: 1 × 2
  neighbors .config        
      <dbl> <chr>          
1      3000 pre0_mod4_post0

finalize_workflow() swaps tune() for 3,000; fit() then fits that model once, on all 24,924 training rows. The 20 tuning fits are gone.

== Workflow [trained] ==================
Preprocessor: Recipe
Model: nearest_neighbor()

  neighbors = 3000 
  kernel = optimal
  ...

Training F1 0.736, test F1 0.737, cross-validated 0.736: all the same. No overfitting, and the cross-validated score predicted the test score well.

# A tibble: 2 × 4
  set   .metric .estimator .estimate
  <chr> <chr>   <chr>          <dbl>
1 train f_meas  binary         0.736
2 test  f_meas  binary         0.737

Part I: what did we learn?

I · kNN + CVII · TreesForestsBoostingWrap-up

F1 for the chosen k = 3,000 three sets of rows, three jobs

cross-validated0.736
training set0.736
test set0.737
Simplest rule"predict the majority of your own cell" training set
Its test F10.739

Cross-validation did its job It ranked the candidates on training rows only, and its score for the winner matched the test set. The test set was opened once.

Why large k won Every dot is a cell of 275 people. k = 100 can’t see past its own cell; k = 3,000 averages a neighbourhood of cells. With two 0–10 features, the cell majority is already as good as it gets: F1 0.739.

kNN has no coefficients Nothing to read off, no “per unit of anxiety”. Flexible, and slow to predict. Trees, next, are flexible and readable.

Trees, forests, boosting

Part II

2,904 US counties · 2008–2018 · the weekly cost of childcare

Meet the data

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 296–318 read_csv(), the split by year and the recipe: all given, run them nowrun together

23,593county-years: 2,904 counties over 11 years
52features: income, jobs, families, population
$97median weekly price of centre-based care, mcsa

Back to regression A dollar outcome: Lab 3’s RMSE and R², and set_mode("regression") on every model.

Time is in the data Each county appears year after year. Train on 2008–2016, test on 2017–2018: predict the future.

Split by time

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 303–309 already run with read_csv(): what the two filter() lines didread together

childcare_train <- childcare |> filter(study_year <= 2016) childcare_test <- childcare |> filter(study_year > 2016)

  1. Train on the past: 2008–2016, 18,797 county-years.
  2. Test on the future: 2017–2018, 4,796. No initial_split(), no seed: the calendar decides.
0809101112131415161718
Splitfitfitfitfitfitfitfitfitfittesttestfinal exam

Folds would need the same care vfold_cv() shuffles rows. With years: fit on the past, score the next (sliding_period()). Not needed today.

Why not a random split? Each county appears every year. A random split puts a county’s 2017 row in training and its 2018 row in the test set: the model has nearly seen the answer, and the score flatters it.

The honest question Next year’s prices from this year’s data: predict the future. The test years are a little harder, because prices rose.

Who’s a Swiftie? You may ask one question

I · kNN + CVII · TreesForestsBoostingWrap-up
Person Age Likes pop Plays guitar Enjoys lyrics Follows trends Swiftie?
A 19 yes no yes yes Swiftie
B 27 yes yes yes no Swiftie
C 20 no yes yes no not
D 24 yes no yes yes Swiftie
E 22 no yes yes no not
F 29 no yes no yes not
G 18 yes no yes yes Swiftie
H 21 no no yes no not
I 26 yes yes yes yes Swiftie
J 25 no no no yes not
K 23 yes yes no no not
L 31 no no yes yes Swiftie

Ask one yes/no question, then guess “Swiftie” or “not” for each group. Which question gets the most right?

Plays guitar?8 of 12 right
Age under 20?8 of 12
Enjoys lyrics?9 of 12
Follows trends?9 of 12
Likes pop?10 of 12 pop fans 5 of 6 · rest 1 of 6

The first split of a decision tree Try every question (every cut-off, for age). Keep the one that sorts cleanest.

Then ask again, inside each group

I · kNN + CVII · TreesForestsBoostingWrap-up
Likes pop?
yes · A B D G I KEnjoys lyrics?
no · C E F H J Lnot a Swiftie5 of 6 right (L is a Swiftie)
yesSwiftieA B D G I: 5 of 5
nonotK: 1 of 1
The tree11 of 12 right, with two questions

Pop fans: one more question Among the six pop fans, “enjoys lyrics?” sorts all six. K is the pop fan who doesn’t care about lyrics.

Each group gets its own question The second question is asked inside a group, not of everyone. Non-pop fans: no question tidies them up, so the guess stays “not”.

A decision tree A flow chart of yes/no questions, a guess in every leaf. Readable by anyone: that’s its big selling point.

What about L, the Swiftie who isn’t a pop fan?

I · kNN + CVII · TreesForestsBoostingWrap-up

The six non-pop fans

Person Age Plays guitar Enjoys lyrics Follows trends Swiftie?
C 20 yes yes no not
E 22 yes yes no not
F 29 yes no yes not
H 21 no yes no not
J 25 no no yes not
L 31 no yes yes Swiftie
Age 30 or over?finds L, and only L 12 of 12

Can a third question find L too?

Yes: one does L is 31, the oldest. “Age 30 or over?” makes the tree perfect on the table.

Would you bet that non-pop fans over 30 are Swifties?

No: a question invented for one person Ask until every leaf is pure and the tree memorises the table: overfitting. Real trees stop early (min_n, cost_complexity).

Same game, with prices

I · kNN + CVII · TreesForestsBoostingWrap-up

The question is a cut-off Try every income as a cut-off, like age in the game. The cleanest: income below $59.2k?

The guess is the mean Each side predicts its average price: $93 and $125. “Cleanest” = the smallest squared error, Lab 3’s RMSE.

Ask again inside each side A staircase. decision_tree() plays this on all 52 columns.

Your first tree, together

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 314–327 the recipe (already run), then the Task 1 chunk: type the spec and the fitcode together

childcare_rec <- recipe(mcsa ~ ., data = childcare_train) |> update_role(starts_with(c("county", "state")), new_role = "id") dt_model <- decision_tree() |> set_mode("regression") |> set_engine("rpart") dt_fit <- workflow() |> add_recipe(childcare_rec) |> add_model(dt_model) |> fit(data = childcare_train)

  1. Keep but don’t use: county and state names and codes become id labels, out of the model. The year stays in.
  2. A tree, with rpart’s built-in brakes.
  3. A price, so every leaf predicts a mean.
  4. No tune(), so fit straight away: Part I’s pattern without the folds.

No normalising this time A tree asks “income below $47.7k?”. A cut-off doesn’t care about units, so no step_normalize(). kNN measured distances; trees don’t.

One recipe, three models The forest and the boosted trees reuse childcare_rec. Only the model line changes.

Exercises: Task 1, draw and score the tree

Part II · Task 1 · lines 324–338

  1. Line 330, given: draw dt_fit with rpart.plot(). Find the first question.
  2. Line 333, given: metric_set(rsq, rmse).
  3. Line 335, yours: score the training set and the test set with it.
  4. Line 338: what does the tree show? Which variables matter?

Syntax you need

rpart.plot(extract_fit_engine(dt_fit),  # the rpart tree inside
           roundint = FALSE)

regress_metrics <- metric_set(rsq, rmse)  # Lab 3's scores

dt_fit |>
  augment(new_data = childcare_train) |>  # adds .pred
  regress_metrics(mcsa, .pred)            # truth, estimate

dt_fit |>
  augment(new_data = childcare_test) |>
  regress_metrics(mcsa, .pred)

Stuck?

  • could not find function “rpart.plot”: run the library chunk at line 27
  • object ‘dt_fit’ not found: run the spec and the fit first

Solutions: Task 1

rpart.plot(extract_fit_engine(dt_fit), roundint = FALSE)

regress_metrics <- metric_set(rsq, rmse)

dt_fit |>
  augment(new_data = childcare_train) |>
  regress_metrics(mcsa, .pred)

dt_fit |>
  augment(new_data = childcare_test) |>
  regress_metrics(mcsa, .pred)

Draw it, then score: training set, then test set.

The Swiftie game on 52 columns. First question one_race_a < 2.2, then income. 6 columns used, 9 leaves from $80 to $224.

Training set: RMSE 27.5 dollars a week, R² 0.32. 9 numbers can only explain so much.

# A tibble: 2 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rsq     standard       0.320
2 rmse    standard      27.5  

Test set: RMSE 31.5, R² 0.32. A little worse than training; predicting the mean would give 39.1. The baseline the next two models must beat.

# A tibble: 2 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rsq     standard       0.323
2 rmse    standard      31.5  

A forest: ask three friends who each know less

I · kNN + CVII · TreesForestsBoostingWrap-up

Friend 1saw A B C D E G H I
may ask: plays guitar or follows trends

Follows trends?
yes · A D G ISwiftie4 of 4 right
no · B C E Hnot3 of 4 right

Mia: Swiftie

Friend 2saw A B E F I J K L
may ask: follows trends or age

Age 26 or over?
yes · B F I LSwiftie3 of 4 right
no · A E J Knot3 of 4 right

Mia: not

Friend 3saw A B D E F I K L
may ask: age or enjoys lyrics

Enjoys lyrics?
yes · A B D E I LSwiftie5 of 6 right
no · F Knot2 of 2 right

Mia: Swiftie

Each friend sees less 8 of the 12 people (a random resample) and 2 of the 5 questions (random columns). Different evidence, a different first question.

Mia: 22, not a pop fan, plays guitar, enjoys lyrics, follows trends Votes: Swiftie, not, Swiftie. Swiftie, 2 to 1. Our one tree said “not”: not a pop fan.

That is a random forest 1,000 such friends. Each tree on a resample of the rows, each split from a few random columns (mtry). They vote; for a price, they average.

A forest, with prices

I · kNN + CVII · TreesForestsBoostingWrap-up

One friend, one resample Draw the counties at random, with replacement; grow a tree. One staircase.

Three friends, three staircases Each resample has different counties, so the steps land in different places.

Average them: smooth Each tree’s odd steps cancel out in the average of 50. The forest overfits far less than any one deep tree.

The forest, together

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 348–351 the Task 2 sketch: fill in the brackets, remove the #code together

rf_model <- rand_forest(mtry = round(sqrt(52), 0), trees = 1e3) |> set_mode("regression") |> set_engine("ranger", num.threads = parallel::detectCores() - 1)

  1. Many trees, averaged. Same spec–workflow–fit pattern as the tree.
  2. mtry: columns per split. Each split may look at 7 random columns of the 52, the friends’ “2 of 5”. √52 ≈ 7 is a rule of thumb.
  3. trees: how many friends. 1,000; more only costs time.
  4. The engine. ranger grows the trees; num.threads uses all cores but one, about 20 seconds instead of a minute.

Exercises: Task 2, fit and score the forest

Part II · Task 2 · lines 347–358

  1. Line 353: fit rf_model in a workflow with childcare_rec; call it rf_fit. Takes about 20 seconds.
  2. Line 355: score the training set and the test set.
  3. Line 358: better predictions than the tree? Does it help with overfitting?

Task 1’s pattern, new model

rf_fit <- workflow() |>
  add_recipe(childcare_rec) |>
  add_model(rf_model) |>        # was dt_model
  fit(data = childcare_train)

rf_fit |>
  augment(new_data = childcare_test) |>
  regress_metrics(mcsa, .pred)
# and the same with childcare_train

Stuck? Training RMSE far below the test one? That’s normal for a forest: see the solution.

Solutions: Task 2

rf_fit <- workflow() |>
  add_recipe(childcare_rec) |>
  add_model(rf_model) |>
  fit(data = childcare_train)

rf_fit |> augment(new_data = childcare_train) |>
  regress_metrics(mcsa, .pred)
rf_fit |> augment(new_data = childcare_test) |>
  regress_metrics(mcsa, .pred)

Fit, then score both sets.

1,000 trees, each on a resample, each split choosing among 7 random columns. About 20 seconds.

== Workflow [trained] ==================
Preprocessor: Recipe
Model: rand_forest()

Ranger result
  Type:            Regression
  Number of trees: 1000
  Mtry:            7
  ...

Test RMSE 23.9 against the tree’s 31.5: much better. Training 6.6: tiny, and that’s normal. Every row was in most trees’ resamples, so the forest has seen it. Judge it on the test set.

# A tibble: 4 × 4
  set   .metric .estimator .estimate
  <chr> <chr>   <chr>          <dbl>
1 train rsq     standard       0.976
2 train rmse    standard       6.61 
3 test  rsq     standard       0.64 
4 test  rmse    standard      23.9  

Boosting: a relay, each friend fixes the last one’s mistakes

I · kNN + CVII · TreesForestsBoostingWrap-up
Person Swiftie? Friend 1
likes pop?
Friend 2
enjoys lyrics?
Friend 3
follows trends?
Together
2 of 3
A Swiftie Swiftie ✓ Swiftie ✓ Swiftie ✓ Swiftie ✓
B Swiftie Swiftie ✓ Swiftie ✓ not ✗ Swiftie ✓
C not not ✓ Swiftie ✗ not ✓ not ✓
D Swiftie Swiftie ✓ Swiftie ✓ Swiftie ✓ Swiftie ✓
E not not ✓ Swiftie ✗ not ✓ not ✓
F not not ✓ not ✓ Swiftie ✗ not ✓
G Swiftie Swiftie ✓ Swiftie ✓ Swiftie ✓ Swiftie ✓
H not not ✓ Swiftie ✗ not ✓ not ✓
I Swiftie Swiftie ✓ Swiftie ✓ Swiftie ✓ Swiftie ✓
J not not ✓ not ✓ Swiftie ✗ not ✓
K not Swiftie ✗ not ✓ not ✓ not ✓
L Swiftie not ✗ Swiftie ✓ Swiftie ✓ Swiftie ✓

Friend 1 plays the game “Likes pop?”: 10 of 12, wrong about K and L.

Friend 2: “K and L count 4×” The best question now gets K and L right: lyrics? Wrong about C, E, H.

Friend 3: “C, E, H, K, L count 4×” Trends? gets those five right. Wrong about B, F, J.

Together: 12 of 12 Nobody is wrong twice. Boosting: small trees in turn, each aimed at the last ones’ mistakes, then combined.

Boosting, with prices

I · kNN + CVII · TreesForestsBoostingWrap-up

With a number, “the mistakes” are the leftovers Each small tree predicts how far off the trees so far still are, county by county.

Add only a tenth Each tree’s answer is added with a small say: learn_rate = 0.1. One step at a time, so no single tree can wreck it.

After 100 trees The relay has found the shape. Keep going for ever and it starts to chase single counties, like the question for L.

Boosting, together

I · kNN + CVII · TreesForestsBoostingWrap-up

Notebook lines 365–372 the Task 3 sketch: fill in the brackets, remove the #code together

xgb_model <- boost_tree( mtry = round(sqrt(52), 0), trees = 1000, learn_rate = 0.1 ) |> set_mode("regression") |> set_engine("xgboost")

  1. Small trees in a relay, summed. Same pattern again; mtry as in the forest.
  2. trees: the length of the relay. 1,000 legs.
  3. learn_rate: each tree’s say. A tenth of its answer is added.
  4. The engine: XGBoost, the boosting library that wins most competitions on tables of data.

Exercises: Tasks 3 and 4, boost and compare

Part II · Tasks 3–4 · lines 364–390

  1. Line 374: fit xgb_model as xgb_fit; score both sets (376). A few seconds.
  2. Line 380: better than the forest? Overfitting?
  3. Lines 386–390, Task 4: test-set RMSE for all three models in one table, then a bar chart.

Task 4: three fits, one table

model_results <-
  list("Decision Tree" = dt_fit,
       "Random Forest" = rf_fit,
       "XGBoost"       = xgb_fit) |>
  map_dfr(~ augment(.x, new_data = childcare_test),
          .id = "model") |>          # list names → a column
  group_by(model) |>
  regress_metrics(mcsa, .pred) |>    # one score per model
  filter(.metric == "rmse")

ggplot(model_results,
       aes(reorder(model, .estimate), .estimate)) +
  geom_col() + coord_flip()

Stuck? map_dfr() is Lab 4’s threshold sweep: a list in, one stacked table out.

Solutions: Task 3

xgb_fit <- workflow() |>
  add_recipe(childcare_rec) |>
  add_model(xgb_model) |>
  fit(data = childcare_train)

xgb_fit |> augment(new_data = childcare_train) |>
  regress_metrics(mcsa, .pred)
xgb_fit |> augment(new_data = childcare_test) |>
  regress_metrics(mcsa, .pred)

Fit, then score both sets.

1,000 small trees in sequence, each adding a tenth of its correction. A few seconds.

== Workflow [trained] ==================
Preprocessor: Recipe
Model: boost_tree()

Main Arguments:
  mtry = 7
  trees = 1000
  learn_rate = 0.1

Computational engine: xgboost
  ...

Test RMSE 22.3, the best of the three (forest 23.9). Training 3.6: the same big gap as the forest, for the same reason.

# A tibble: 4 × 4
  set   .metric .estimator .estimate
  <chr> <chr>   <chr>          <dbl>
1 train rsq     standard       0.989
2 train rmse    standard       3.59 
3 test  rsq     standard       0.655
4 test  rmse    standard      22.3  

Solutions: Task 4

model_results <-
  list(
    "Decision Tree" = dt_fit,
    "Random Forest" = rf_fit,
    "XGBoost"       = xgb_fit
  ) |>
  map_dfr(~ .x |> augment(new_data = childcare_test),
          .id = "model") |>
  group_by(model) |>
  regress_metrics(mcsa, .pred) |>
  filter(.metric == "rmse")

ggplot(model_results,
       aes(x = reorder(model, .estimate),
           y = .estimate, fill = model)) +
  geom_col(show.legend = FALSE) +
  geom_text(aes(label = round(.estimate, 1)),
            hjust = -0.1) +
  coord_flip() +
  labs(x = NULL, y = "RMSE (lower is better)")

Score the three fits on the test set in one go, then draw the result.

map_dfr() runs augment() on each fit and stacks the three tables; .id keeps the list names. group_by(model) then gives one RMSE per model.

# A tibble: 3 × 4
  model         .metric .estimator .estimate
  <chr>         <chr>   <chr>          <dbl>
1 Decision Tree rmse    standard        31.5
2 Random Forest rmse    standard        23.9
3 XGBoost       rmse    standard        22.4

reorder() sorts the bars by RMSE; coord_flip() turns them sideways so the names fit. The notebook adds a title, theme and colours.

Tree, forest, boosting: what’s the difference?

I · kNN + CVII · TreesForestsBoostingWrap-up
One tree Random forest Boosting
In the game "Pop?", then "lyrics?" Three friends who each saw part of the table, voting A relay: each friend aimed at the last one's mistakes
How the trees are grown One, until the brakes stop it 1,000 at once, independently: each on a resample, random columns 1,000 small ones in turn, each on the leftovers
How they're combined No combining Average (or vote) Sum, each with a small say
The knobs tree_depth, min_n, cost_complexity trees, mtry trees, learn_rate, mtry
Can you read it? Yes: a flow chart No No
Test RMSE (train) 31.5 (27.5) 23.9 (6.6) 22.3 (3.6)

Mind the gap An ensemble’s training error is always tiny: it has seen every row many times. The test set is the honest number. Part of the gap is time: prices rose in 2017–2018.

Nothing was tuned 1,000 trees, mtry 7, learning rate 0.1: rules of thumb. Part I’s tune() and cross-validation are how you’d do better.

Summary

I · kNN + CVII · TreesForestsBoostingWrap-up
1A hyperparameter is a setting you choose, not one fit() estimates: the number of neighbours, the depth of a tree, the number of trees, the learning rate. Choose it by cross-validation: cut the training set into folds, score every candidate on held-out folds, average, pick. The test set stays sealed until the end. In code: vfold_cv(), tune(), tune_grid(), select_best(), finalize_workflow()
2k nearest neighbours is the estate agent's comparables: the k most similar training rows vote. Small k is jagged (k = 1 memorises), large k is smooth. A decision tree is the Swiftie game: the question that sorts cleanest, then again inside each group, and stop before you invent a question for one person (L)
3Ensembles fix the tree's instability. A random forest is a crowd that doesn't talk: many trees, each on resampled rows with a random handful of features, averaged. Boosting is a relay: small trees in turn, each aimed at what the last ones got wrong, summed. Both beat one tree; neither can be read like one

Question: you have the 2019 prices. Which model do you deploy, and how would you check whether the forest’s training RMSE of $7 means anything?

Further reading

30 seconds of feedback

QR code linking to the feedback survey

How was today’s lab? Scan the code, or open forms.gle/i3uobCAaGmkzf1ns7. It takes 30 seconds and changes what next week looks like.