Classification: predicting yes or no

MY202 · Lab 4 · Self-paced guide

Author

Jonas Weinert

Published

22 October 2026

Before you start

This page walks you through Lab 4 one small step at a time. It assumes you have done Labs 1 to 3, so you know what a notebook, a chunk, a pipe |>, a ggplot and last week’s initial_split(), fit() and augment() are. Everything new is explained the first time it appears, and there is a glossary at the end.

What’s new this week. Last week you predicted a number, a grade. This week you predict a category: does this person have diabetes, yes or no? That’s called classification. You’ll meet one new model, logistic regression, and new ways to say whether a classifier is any good. The second part matters as much as the first: a model can be “86.7% accurate” and still be close to useless.

Your turn Something to do in the notebook. The box says which notebook lines to go to.

What you should see The real result R gives back. Compare it with yours.

Big idea A concept worth remembering beyond today.

Check yourself A quick question. Click an answer to see if you were right.

At the end of each step there is a Mark as done button. Your ticks are saved in this browser, so you can close the page and pick up where you left off.

The student notebook has an empty # Code here chunk for every task. Where a task asks you to write code, try it first; the code is in a “Show the code” box underneath if you get stuck.

Plan for about 2–3 hours

Part I takes about 40 minutes, the big idea 20, Part II 30 and Part III an hour. Part III is the most important part of the lab: take your time with the confusion matrix and the thresholds.

Open the notebook and run the setup chunks

  1. Open the Lab 4 student notebook on Nuvolos. The packages and the data are already installed there.
  2. Run the chunk at lines 18–25. It loads five packages.
  3. Run the chunk at lines 53–56. It reads the data into an object called diabetes_data.
Package What it gives you
tidymodels Last week’s modelling family: initial_split(), fit(), augment(), metric_set(). Today also logistic_reg() and conf_mat().
tidyverse, ggsci Wrangling, plots and colour schemes, as before.
reshape2 Used once in the solutions, to reshape a correlation table for a heatmap.
kknn Loaded, but not used in this lab.

Part I: Explore the data

About 40 minutes. Before modelling, find out what’s in the data, and tell R what kind of thing each column is.

Meet the data

diabetes_data has one row per person: 25,368 adults who answered a US health survey. The outcome is diabetes: 1 if the person has diabetes, 0 if not. The other 19 columns are features, things that might help predict it:

Group Features
Health conditions high_bp (high blood pressure), high_chol, chol_check (had cholesterol checked), bmi, stroke, heart_diseaseor_attack, diff_walk (difficulty walking), gen_hlth (self-rated general health)
Lifestyle smoker, phys_activity, fruits, veggies, hvy_alcohol_consump (heavy drinking)
Who they are sex, age, education, income, any_healthcare, no_docbc_cost (couldn’t see a doctor because of cost)
Three things the notebook’s variable list doesn’t say
  • Most columns are 0/1: 1 means yes, 0 means no.
  • age is not an age in years. It’s a code for 5-year bands: 1 = 18–24, 2 = 25–29, … 13 = 80 or older.
  • gen_hlth runs the “wrong” way: 1 = excellent, 5 = poor. A higher number means worse health.

Also, line 43 calls a column hvy_alcohol_consum; its real name is hvy_alcohol_consump.

Question 1: how big is it, and is anything missing?

In the chunk at line 61, find the number of rows and columns. In the chunk at line 65, count the missing values in every column.

The first is one function you know: dim(). The second is new. It uses across(), which runs the same function on many columns at once:

diabetes_data |> summarise(across(everything(), ~ sum(is.na(.x))))

  1. Collapse the table to one row of summaries, as in Lab 1.
  2. Do the same thing to every column. everything() means “all columns”.
  3. What to do to each column. The ~ starts a mini-function; .x stands for “the current column”. is.na() gives TRUE for each missing value, and sum() counts the TRUEs.
dim(diabetes_data)

diabetes_data |>
  summarise(across(everything(), ~ sum(is.na(.x)))) |>
  glimpse()

The glimpse() at the end prints the one-row result sideways, which is easier to read with 20 columns.

[1] 25368    20
Rows: 1
Columns: 20
$ diabetes               <int> 0
$ high_bp                <int> 0
$ high_chol              <int> 0
$ chol_check             <int> 0
$ bmi                    <int> 0
$ smoker                 <int> 0
$ stroke                 <int> 0
$ heart_diseaseor_attack <int> 0
$ phys_activity          <int> 0
$ fruits                 <int> 0
$ veggies                <int> 0
$ hvy_alcohol_consump    <int> 0
$ any_healthcare         <int> 0
$ no_docbc_cost          <int> 0
$ gen_hlth               <int> 0
$ diff_walk              <int> 0
$ sex                    <int> 0
$ age                    <int> 0
$ education              <int> 0
$ income                 <int> 0

25,368 people, 20 columns, and no missing values anywhere.

Step 1: what type is each column now?

Look at the type of every column with glimpse(diabetes_data).

Rows: 25,368
Columns: 20
$ diabetes               <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
$ high_bp                <dbl> 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, …
$ high_chol              <dbl> 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, …
$ chol_check             <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
$ bmi                    <dbl> 21, 28, 24, 27, 31, 33, 29, 27, 25, 33,…
$ smoker                 <dbl> 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, …
$ stroke                 <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
$ heart_diseaseor_attack <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
$ phys_activity          <dbl> 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, …
$ fruits                 <dbl> 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, …
$ veggies                <dbl> 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, …
$ hvy_alcohol_consump    <dbl> 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, …
$ any_healthcare         <dbl> 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, …
$ no_docbc_cost          <dbl> 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, …
$ gen_hlth               <dbl> 3, 3, 1, 2, 4, 4, 1, 3, 1, 2, 2, 3, 4, …
$ diff_walk              <dbl> 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, …
$ sex                    <dbl> 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, …
$ age                    <dbl> 7, 13, 1, 2, 8, 7, 10, 10, 8, 8, 10, 7,…
$ education              <dbl> 4, 6, 4, 4, 3, 3, 6, 4, 6, 5, 6, 4, 6, …
$ income                 <dbl> 2, 6, 7, 7, 2, 6, 8, 8, 8, 8, 8, 6, 7, …

Every column is <dbl>, a decimal number. Even high_bp, which is really a yes/no answer, and diabetes, our outcome. R has no way of knowing that sex = 1 is a category and bmi = 30 is a measurement. The next two steps fix that.

The solutions also show map_chr(diabetes_data, class). It runs class() on every column and returns one word each: "numeric", "factor" and so on. Same information, more compact. You’ll use it in Step 4.

Step 2: what type should each column be?

Sort the columns into three groups, and write your answer at line 86.

Kind of column Example R type Why
Continuous: any value in a range bmi (and age, as the notebook treats it) numeric Arithmetic makes sense: a BMI of 30 is 5 more than 25
Ordered levels: whole-number scales gen_hlth, education, income integer The order means something, and only whole numbers occur
Categories: yes/no and other labels high_bp, smoker, sex, … and the outcome diabetes factor The numbers are just labels: sex = 1 is not “one more” than sex = 0

For classification, the outcome must be a factor. That’s how tidymodels knows you want to predict a category, not a number. If diabetes stays numeric, the model in Part II will complain.

Which of these columns should become a factor?

bmi

BMI is a real measurement: arithmetic on it makes sense. Keep it numeric.

smoker

Yes. 1 means “smokes”, 0 means “doesn’t”. The numbers are labels, not amounts.

income

Income levels 1–8 have an order, so the notebook keeps them as whole numbers (integers).

Steps 3 and 4: convert many columns at once

Convert all three groups in one mutate() at line 93, then check the types at line 103.

You could write mutate(high_bp = as.factor(high_bp)) fifteen times. across() does it in one line per group. First, write the three groups as vectors of column names; then apply one function to each group:

categorical_vars <- c("diabetes", "high_bp", …) … mutate(across(all_of(categorical_vars), ~ as.factor(.x)))

  1. A vector of column names: your answer to Step 2, written as code. Make one for each group.
  2. Do the same thing to many columns.
  3. Which columns: every name in the vector. all_of() takes a vector of names.
  4. What to do to each: turn the current column .x into a factor. For the other groups: as.numeric(.x) and as.integer(.x).
continuous_vars <- c("age", "bmi")
discrete_vars <- c("gen_hlth", "education", "income")
categorical_vars <- c(
  "diabetes", "high_bp", "high_chol", "chol_check", "smoker", "stroke",
  "heart_diseaseor_attack", "phys_activity", "fruits", "veggies",
  "hvy_alcohol_consump", "any_healthcare", "no_docbc_cost", "diff_walk", "sex"
)

diabetes_data <-
  diabetes_data |>
  mutate(
    across(all_of(continuous_vars), ~ as.numeric(.x)),
    across(all_of(discrete_vars), ~ as.integer(.x)),
    across(all_of(categorical_vars), ~ as.factor(.x))
  )

map_chr(diabetes_data, class)
              diabetes                high_bp              high_chol 
              "factor"               "factor"               "factor" 
            chol_check                    bmi                 smoker 
              "factor"              "numeric"               "factor" 
                stroke heart_diseaseor_attack          phys_activity 
              "factor"               "factor"               "factor" 
                fruits                veggies    hvy_alcohol_consump 
              "factor"               "factor"               "factor" 
        any_healthcare          no_docbc_cost               gen_hlth 
              "factor"               "factor"              "integer" 
             diff_walk                    sex                    age 
              "factor"               "factor"              "numeric" 
             education                 income 
             "integer"              "integer" 

Why integers rather than numeric? An integer says “only whole numbers here”, which documents what the column is. For a model, both work the same.

What if you had 50+ columns? Don’t type the names; select them by a rule. across(where(is.double), …) picks every decimal column; across(starts_with("high_"), …) every column whose name starts with high_.

Explore: the EDA challenge

Make two or three plots or summaries that show something interesting about diabetes. Start with: (1) how many people have diabetes, and (2) how the share with diabetes changes with age. Then pick one of your own from the list at lines 136–142. Write one sentence under each. Post your best one on Moodle (line 150).

The share of people who have a condition is called its prevalence. To compute it, use a trick: diabetes == "1" is TRUE or FALSE for each person, and the mean() of TRUE/FALSE values is the share of TRUEs.

# How many have diabetes?
diabetes_data |>
  count(diabetes) |>                # one row per class
  mutate(pct = n / sum(n) * 100)    # turn counts into %

# Prevalence by age band
diabetes_data |>
  group_by(age) |>
  summarise(prev = mean(diabetes == "1")) |>   # share with diabetes
  ggplot(aes(x = age, y = prev)) +
  geom_line() +
  geom_point()

The solutions write the prevalence as mean(as.numeric(diabetes) == 2). That works because as.numeric() on a factor gives the position of each level (1 for "0", 2 for "1"), not its label. Same result, more confusing.

What these show:

  • Only 13.8% have diabetes. The two classes are very unequal in size: the data are imbalanced. Keep this number in mind; it matters a lot in Part III.
  • Prevalence rises from about 1% at 18–24 to over 20% in people’s 70s.
  • Physically active people have about half the prevalence of inactive people. Careful: active people are also younger and lighter on average, so this plot alone can’t say that activity itself is what matters. A model can hold the other features fixed (Part II).
  • Diabetes is four times as common among people with high blood pressure.

The solutions label sex = 1 as “Female”. In the survey this extract comes from, 1 usually means male, which would reverse the conclusion. Until that’s checked, don’t read anything into a difference by sex.

The big idea: from a line to an S-curve

About 20 minutes, almost no code. The notebook leaves this to your class teacher (“teaching moment”, line 168). Here it is with pictures.

Predicting yes or no with a line?

Take a made-up chess club. For each player we know how many games they had played before, and whether they won (1) or lost (0) their next game. What’s the chance that a player wins?

Last week’s tool would be to fit a straight line through the points:

For a player with 300 games the line says 1.46: a 146% chance of winning. That makes no sense. A chance lives between 0 and 1, but a straight line keeps going forever in both directions.

The fix: an S-curve

Logistic regression fits an S-shaped curve instead: flat near 0 for beginners, flat near 1 for experienced players, and steep in between. It can never go above 1 or below 0.

What comes out of a logistic regression is a probability: the chance that the outcome is 1. In R, this column will be called .pred_1.

Inside, logistic regression still has last week’s line. It just bends the line’s output into an S, so that every prediction is a probability between 0 and 1.

From a probability to a yes or no: the threshold

A probability isn’t yet a prediction of “win” or “lose”. To get one, we pick a cut-off, called the threshold, and say “win” above it, “lose” below it. The usual default is 0.5.

New player150 games: probability 0.75
Threshold0.5
Rule0.75 > 0.5 → predict "win"

0.5 is a choice, not a law. When the 1s are rare, as with diabetes, a different threshold is often much better. Part III is about choosing it.

What the model learns: two dials

Fitting a logistic regression means setting the S-curve so that it matches the data. The curve has two kinds of dials. Here they are for one feature, BMI, with everything else about a person held fixed:

  • Dial 1, one per feature: its coefficient. A steeper S means the chance changes faster as the feature rises; a flat line means the feature makes no difference. A rising curve means more of the feature, more risk; a falling curve, less risk.
  • Dial 2: the intercept. It moves the whole curve up or down: the baseline risk before the features do anything.

These dials are the numbers summary() will print in Part II. R prints them on a scale called the log-odds, the scale on which the S-curve becomes a straight line again. You never have to calculate on that scale: one function, exp(), turns each coefficient into something readable, the odds ratio.

Reading a coefficient: the odds ratio

First, odds: the chance that something happens compared with the chance that it doesn’t. A 1-in-10 chance is odds of 1 to 9; a 50% chance is odds of 1 to 1, “evens”.

The odds ratio (OR) of a feature says how much the odds are multiplied by when the feature is present (or goes up by one step). You get it from a coefficient with exp(estimate). Here are the odds ratios of today’s model:

How to read it:

  • OR above 1 (right of the dashed line): higher odds of diabetes. Below 1 (left): lower odds. Exactly 1: no difference.
  • BMI, OR 1.06: each extra BMI point multiplies the odds by 1.06, about 6% higher odds. Ten points: about × 1.83.
  • Heavy drinker, OR 0.37: about 63% lower odds than non-heavy drinkers, everything else the same.
One persona 10% chance: odds of 1 to 9
High blood pressureOR = 2.29
With itabout 20%roughly double, when the risk is small

A coefficient’s sign tells you the direction; exp() turns it into an odds ratio, a multiplier on the odds. That’s the one number to report for each feature.

A feature has an odds ratio of 0.8. What does that mean?

An 80% chance of diabetes

An odds ratio isn’t a probability. It’s a multiplier on the odds.

The odds of diabetes are multiplied by 0.8: about 20% lower

Yes. Below 1 means lower odds; 0.8 means 20% lower, everything else the same.

The feature makes no difference

That would be an odds ratio of exactly 1.

Part II: Fit a logistic regression

About 30 minutes. Almost everything here is last week’s code with two small changes.

Question 1: why not score on the training set?

Write one or two sentences.

On the rows it was fitted on, a model looks better than it is: it has partly learned the quirks of those particular rows (overfitting). To see how well it predicts new people, score it on rows it has never seen: the test set. Like an exam: revising with last year’s paper and then sitting last year’s paper again tells you little.

Question 2: split the data

Split diabetes_data into 75% training and 25% test, with strata = diabetes. Use set.seed(123) so your numbers match this page.

The split is last week’s, with one new argument:

split_grid <- initial_split(diabetes_data, prop = 0.75, strata = diabetes)

  1. Stratify on the outcome: make sure both sets get the same share of people with diabetes. With a rare outcome, a purely random split could leave noticeably fewer 1s in one set by chance.
set.seed(123)

split_grid <- initial_split(diabetes_data, prop = 0.75, strata = diabetes)
train <- training(split_grid)
test <- testing(split_grid)

19,025 people for training, 6,343 for testing, both with 13.8% diabetes.

The challenge: fit a logistic regression

Fit a logistic regression of diabetes on all other columns, on the training set, and save it as model.

Compare with last week’s linear model. Only the first function changes:

model <- logistic_reg() |> fit(diabetes ~ ., data = train)

  1. Which kind of model: logistic instead of last week’s linear_reg(). Its default engine, glm, is R’s built-in logistic regression, so no set_engine() is needed.
  2. The formula: the outcome diabetes, explained by ., every other column. Because diabetes is a factor, tidymodels knows this is classification.
  3. Fit on the training rows only.
model <-
  logistic_reg() |>
  fit(diabetes ~ ., data = train)

diabetes is still a number. Run the Step 3 chunk (line 93) first, then the split again.

Question 3: should all variables be in the model?

Write one or two sentences.

Not necessarily. Features that add nothing make the model noisier and can lead to overfitting. You can decide with theory (what should matter for diabetes?), with data (does the model predict new people better without it? Last week’s lasso does this automatically), and with practicality (will this information be available when the model is used?).

Question 4: the coefficient table

Print the coefficients with summary(), as in Lab 3.

model |> extract_fit_engine() |> summary()

  1. Take out the model R fitted underneath (a glm object). tidymodels wraps it; summary() needs the original.
  2. Print the coefficient table, as for last week’s lm model.

Call:
stats::glm(formula = diabetes ~ ., family = stats::binomial, 
    data = data)

Coefficients:
                         Estimate Std. Error z value Pr(>|z|)    
(Intercept)             -7.534466   0.334759 -22.507  < 2e-16 ***
high_bp1                 0.828299   0.055054  15.045  < 2e-16 ***
high_chol1               0.657774   0.050809  12.946  < 2e-16 ***
chol_check1              1.028935   0.244147   4.214 2.50e-05 ***
bmi                      0.060443   0.003311  18.253  < 2e-16 ***
smoker1                 -0.077750   0.048463  -1.604   0.1086    
stroke1                  0.060800   0.093133   0.653   0.5139    
heart_diseaseor_attack1  0.274445   0.065499   4.190 2.79e-05 ***
phys_activity1          -0.045627   0.052866  -0.863   0.3881    
fruits1                 -0.112121   0.049981  -2.243   0.0249 *  
veggies1                -0.072613   0.058018  -1.252   0.2107    
hvy_alcohol_consump1    -0.983620   0.152806  -6.437 1.22e-10 ***
any_healthcare1          0.031944   0.120491   0.265   0.7909    
no_docbc_cost1           0.061150   0.082576   0.741   0.4590    
gen_hlth                 0.507024   0.027349  18.539  < 2e-16 ***
diff_walk1              -0.066814   0.061151  -1.093   0.2746    
sex1                     0.230670   0.049621   4.649 3.34e-06 ***
age                      0.125081   0.010148  12.326  < 2e-16 ***
education               -0.027124   0.025683  -1.056   0.2909    
income                  -0.053043   0.013027  -4.072 4.67e-05 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 15250  on 19024  degrees of freedom
Residual deviance: 11965  on 19005  degrees of freedom
AIC: 12005

Number of Fisher Scoring iterations: 6

The table has one row per coefficient, like last week. The columns:

Column What it means
Estimate The coefficient (one dial): the change in the log-odds of diabetes when the feature goes up by one, everything else held fixed. Read its sign: positive means higher risk. exp(Estimate) is the odds ratio.
Std. Error How uncertain the estimate is.
z value Estimate ÷ standard error, like last week’s t value.
Pr(>|z|) The p-value: if the true effect were zero, how surprising would an estimate this far from zero be? Small means “very”.

To see all the odds ratios at once, this one extra line (not in the notebook) does the conversion:

tidy(model) |>
  mutate(odds_ratio = exp(estimate)) |>
  select(term, estimate, odds_ratio, p.value) |>
  arrange(p.value)
Odds ratios below 1 are not proof of protection

Heavy drinkers have odds × 0.37: about 63% lower. Does alcohol protect against diabetes? More likely, people with diabetes are told to stop drinking heavily. The model finds associations, not causes.

  1. How do we read the output? One row per feature. The estimate is the change in the log-odds; the standard error its uncertainty; the p-value whether it’s clearly different from zero.
  2. What do the coefficients represent? The change in the log-odds of diabetes for a one-unit increase in the feature, others held fixed. exp() turns them into odds ratios: above 1 increases the odds, below 1 decreases them. BMI: OR 1.06, about 6% higher odds per point.
  3. How do p-values help? high_bp, bmi, gen_hlth and age have tiny p-values: strong evidence of an association. But a p-value says nothing about how well the model predicts.
  4. Which metrics for classification? Not accuracy alone: only 14% have diabetes, so predicting “no” for everyone is already 86% right. Part III gives better ones.

Part III: Is the model any good?

About an hour. The most important part of the lab.

Question 1: predictions

Get predictions for the training set and the test set, saved as train_preds and test_preds.

train_preds <- model |> augment(new_data = train)

  1. Add the model’s predictions to a table, as last week. For a classifier you get three new columns: .pred_class (0 or 1), and .pred_0 and .pred_1, the two probabilities. .pred_1 is the chance of diabetes.
train_preds <- model |> augment(new_data = train)
test_preds <- model |> augment(new_data = test)

The notebook’s tip says to add type.predict = "response", and the solutions also add data = diabetes_data. tidymodels’ augment() ignores both and gives you the probabilities anyway. You can leave them out.

Question 2: what do the predictions look like?

Print the first ten predicted probabilities for the test set with pull(test_preds, .pred_1)[1:10]. How could you turn them into a yes or no?

pull() takes one column out of a table as a plain vector; [1:10] keeps its first ten elements.

 [1] 0.007612360 0.142840941 0.102032878 0.259230352 0.389565277
 [6] 0.018752068 0.056227020 0.223643890 0.329659820 0.006778071

These are probabilities, not classes, and most are small: the model is rarely confident that someone has diabetes, because few people do. To get a yes or no, we need a threshold: above it, predict 1.

Question 3: apply a threshold of 0.5

Add a column .pred_class that is 1 when .pred_1 is above 0.5 and 0 otherwise, for both train_preds and test_preds.

mutate(.pred_class = as.factor(if_else(.pred_1 > 0.5, 1, 0)))

  1. Make it a factor, like the truth column diabetes, so the two can be compared.
  2. The threshold: if the probability is above 0.5, a 1; otherwise a 0.
train_preds <-
  train_preds |>
  mutate(.pred_class = as.factor(if_else(.pred_1 > 0.5, 1, 0)))

test_preds <-
  test_preds |>
  mutate(.pred_class = as.factor(if_else(.pred_1 > 0.5, 1, 0)))

Where do the test-set probabilities fall? Grey bars are people without diabetes, red bars people with it. The dashed line is the threshold:

Only 3.8% of people get a probability above 0.5. Most of the red bars, people who do have diabetes, are to the left of the line: the model will call them “no”.

Question 4: the confusion matrix

Make a confusion matrix for the training set (line 264) and for the test set (line 268).

Every prediction lands in one of four boxes:

Truth: diabetes Truth: no diabetes
Predicted: diabetes True positive (TP): has it, flagged False positive (FP): healthy, flagged anyway, a false alarm
Predicted: no diabetes False negative (FN): has it, missed True negative (TN): healthy, correctly left alone

A confusion matrix counts how many people are in each box.

test_preds |> conf_mat(truth = diabetes, estimate = .pred_class) |> autoplot(type = "heatmap")

  1. Count the four boxes.
  2. The column with the real answers.
  3. The column with the model’s answers: the classes from Q3.
  4. Draw it as a heatmap: darker = more people.
train_preds |>
  conf_mat(truth = diabetes, estimate = .pred_class) |>
  autoplot(type = "heatmap")

test_preds |>
  conf_mat(truth = diabetes, estimate = .pred_class) |>
  autoplot(type = "heatmap")

R puts the 0 class first: the top-left cell is the true negatives (5,363), and the bottom-right the true positives (136). Rows are the predictions, columns the truth. So: of the 874 people with diabetes in the test set, the model flags 136 and misses 738.

Question 5: is this model good or bad?

Write your answer, then compare with the numbers below.

Each common metric answers one question about the confusion matrix, using two of its cells:

Accuracy(TP + TN) / all = (136 + 5,363) / 6,343 = 0.867share of all predictions that are right
RecallTP / (TP + FN) = 136 / 874 = 0.156share of the sick we find · also called sensitivity
PrecisionTP / (TP + FP) = 136 / 242 = 0.562share of our flags that are right
SpecificityTN / (TN + FP) = 5,363 / 5,469 = 0.981share of the healthy we leave alone
F12 × P × R / (P + R) = 0.244one score, high only if precision and recall both are

With imbalanced classes, accuracy flatters. A “model” that predicts “no diabetes” for everyone is right about everyone without diabetes: accuracy 0.862. Ours scores 0.867, barely better. Recall tells the real story: our model finds only 16% of the people who have diabetes.

The solutions round a little differently

The solutions’ table gives accuracy as 0.86 and F1 as about 0.25. With the same counts the exact values are 0.867 and 0.244.

Which mistake is worse? (The table-partner questions at lines 275–280.) Imagine the model decides who gets invited for a free diabetes blood test. A false negative is a person with diabetes who never gets tested; a false positive costs one blood test. For screening, missing people is worse, so we want high recall and accept lower precision. If instead a flag meant starting medication straight away, false positives would matter more, and precision would be the priority. What counts as “good” depends on what the predictions are used for.

Of 100 people the model flags as “diabetes”, 56 really have it. Which metric is that?

Recall

Recall starts from the people who really have diabetes: how many of them did we find?

Precision

Yes: of the people we flagged, the share who really have it.

Accuracy

Accuracy counts all correct predictions, flagged or not.

Question 6: what happens if we move the threshold?

What’s the problem with a threshold of 0.8? What happens to precision and recall if the threshold goes up, or down?

Three possible thresholds: 0.5, 0.3 and 0.1. Moving the line left, every red bar to the right of the line is a person with diabetes we now catch; every grey bar to the right is a false alarm.

Threshold 0.5find 136 of 874, with 106 false alarms
Threshold 0.3find 371, with 500 false alarms
Threshold 0.1find 742, with 1,944 false alarms
  • Lower the threshold: more people are flagged. Recall goes up, precision goes down.
  • Raise it: fewer flags, more of them right. Precision up, recall down.
  • 0.8 is a poor choice here: hardly anyone gets a probability that high, so recall would be close to zero.

Question 7: precision, recall and F1 for seven thresholds

On the training set, compute precision, recall and F1 for thresholds 0.1, 0.2, …, 0.7.

Why the training set? Choosing a threshold is part of building the model, like choosing the lasso’s penalty last week. We choose it on the training set and keep the test set for the final check (Q10).

This is the hardest code of the lab. Take it one piece at a time:

class_metrics <- metric_set(precision, recall, f_meas)

  1. Bundle three metrics into one function, as metric_set(rsq, rmse) last week. f_meas is the F1-score.

class_metrics <- partial(class_metrics, event_level = "second")

  1. Tell it which class to score. The class we care about, "1", is the second level of the factor. Without this, R treats "0" as the event and your precision and recall will look suspiciously high, around 0.9. partial() fixes the argument once, so you don’t have to repeat it.

ev_metrics <- map_dfr(alpha_list, ~ augment(model, new_data = train) |> mutate(.pred = … .pred_1 > .x …) |> class_metrics(…) |> mutate(alpha = .x))

  1. Run some code once for each threshold in alpha_list, and stack the results into one table.
  2. The code to run: last week’s evaluation pipeline. .x is the current threshold.
  3. Label each row with its threshold, so you can plot by it later.

Step through what happens for one threshold, then all seven:

augment(model, new_data = train) |>
  mutate(.pred = as.factor(if_else(.pred_1 > 0.3, "1", "0"))) |>
  class_metrics(truth = diabetes, estimate = .pred)

For one threshold, here 0.3: predictions on the training set, cut at 0.3. Only the truth, the probability and the new class are shown.

class_metrics() scores that one threshold: three rows, one per metric.

map_dfr() does exactly this for every threshold in alpha_list, replacing 0.3 with .x, and stacks the seven results: 21 rows.

# Create a list of probability thresholds
alpha_list <- c(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7)

# Define our metric set with precision, recall, and F1-score
class_metrics <- metric_set(precision, recall, f_meas)
class_metrics <- partial(
  class_metrics,
  event_level = "second"
)

ev_metrics <- map_dfr(
  alpha_list,
  ~ augment(model, new_data = train) |>
    mutate(.pred = as.factor(if_else(.pred_1 > .x, "1", "0"))) |>
    class_metrics(truth = diabetes, estimate = .pred) |>
    mutate(alpha = .x), # Add the actual alpha value
  .id = NULL # Remove the automatic ID column
)

Question 8: plot the metrics against the threshold

Plot the three metrics against the threshold, one colour per metric.

ev_metrics is already in the right shape for this: one row per threshold and metric. Put alpha on x, .estimate on y and .metric on colour.

ev_metrics |>
  mutate(.metric = if_else(.metric == "f_meas", "F1-Score", str_to_title(.metric))) |>
  ggplot(aes(alpha, .estimate, colour = .metric)) +
  geom_line(linetype = "dashed") +
  geom_point() +
  scale_colour_uchicago() +
  labs(
    x = "Probability threshold",
    y = "Evaluation metric estimate",
    colour = "Evaluation metric"
  ) +
  theme_minimal()

The mutate() only makes the legend labels nicer: f_meas becomes “F1-Score”, precision becomes “Precision”.

As the threshold rises, recall falls steeply and precision creeps up. F1 is highest at 0.2 (0.464), against 0.240 at the default 0.5.

Question 9: the precision–recall curve

Plot precision (y) against recall (x), one point per threshold. What trade-off do you see?

For this plot, precision and recall must be two separate columns, so that one can go on each axis. pivot_wider() (Lab 1) does that:

ev_metrics |> filter(.metric %in% c("precision", "recall")) |> pivot_wider(id_cols = alpha, names_from = .metric, values_from = .estimate)

  1. Keep the two metrics we need.
  2. One row per threshold, one column per metric.
ev_metrics |>
  filter(.metric %in% c("precision", "recall")) |>
  pivot_wider(
    id_cols = alpha,
    names_from = .metric,
    values_from = .estimate
  ) |>
  ggplot(aes(x = recall, y = precision)) +
  geom_point(color = "turquoise4", size = 2) +
  geom_text(
    aes(label = paste0("α = ", alpha)),
    position = position_nudge(x = -0.045)
  ) +
  geom_line(linetype = "dashed", color = "turquoise4") +
  theme_minimal()

Each point is one threshold. Moving right (lower threshold), we find more of the people with diabetes (recall up) but each flag is less often right (precision down). You can’t have both for free: the threshold is how you choose the balance.

It says “when the probability threshold is lowered, recall decreases and precision increases”. It’s the other way: lowering the threshold raises recall and lowers precision, as the text under the plot in the solutions says.

Question 10: did the model overfit?

Take the threshold with the best F1 on the training set. Apply the model to the test set with that threshold and compute its F1. Compare.

Two new small functions find the best threshold:

best_alpha <- ev_metrics |> filter(.metric == "f_meas") |> slice_max(.estimate) |> pull(alpha)

  1. Keep the row with the largest value of .estimate: the best F1.
  2. Take out one column as a plain value: the threshold itself.

Then it’s Q3’s threshold rule and Q7’s scoring, on the test set, with best_alpha instead of 0.5:

best_alpha <-
  ev_metrics |>
  filter(.metric == "f_meas") |>
  slice_max(.estimate) |>
  pull(alpha)

model |>
  augment(new_data = test) |>
  mutate(prediction = if_else(.pred_1 > best_alpha, 1, 0),
         prediction = as.factor(prediction)) |>
  f_meas(truth = diabetes, estimate = prediction, event_level = "second")
[1] 0.2
Threshold 0.2 F1
Training set 0.464
Test set 0.462

No overfitting: the model does just as well on people it has never seen. (Compare last week, when a model with 67 coefficients on 486 students did far worse on the test set.) But an F1 below 0.5 is still not great: people with and without diabetes look very similar on these features, and no threshold can fix that.

At threshold 0.2, on the test set, the model finds 558 of the 874 people with diabetes (64%), with 981 false alarms: about 1.8 false alarms for every person found. Compare 0.5: 136 found, 16%.

You choose the threshold that gives the best F1 on the test set, and report that F1 as your model’s performance. What’s the problem?

The test set was used to make a choice, so its score is no longer a fair, fresh exam

Exactly. Choose on the training set, check on the test set. That’s why Q7 uses train.

There’s no problem: the test set is the right place to measure performance

It’s the right place to measure, but not to choose. Once you’ve tuned on it, it’s part of the training.

F1 can’t be computed on a test set

It can; Q10 does exactly that.

Wrap-up

What you learned

Logistic regression Bends a line into an S-curve, so predictions are probabilities between 0 and 1. A coefficient is a change in the log-odds: its sign says which way, and exp() turns it into an odds ratio.

Thresholds A threshold turns probabilities into yes or no. 0.5 is only a default: choose it on the training set, check it on the test set.

The confusion matrix Four boxes: TP, FP, FN, TN. Recall (how many of the 1s we find) and precision (how many of our flags are right) trade off as the threshold moves.

Imbalance When one class is rare, accuracy flatters. Predicting “no” for everyone already scores 86% here.

One question to take away: you’re designing a screening programme. Which threshold would you pick, and how many false alarms per person found would you accept?

Further reading

Glossary

Underlined words on this page are explained in this list. Click or tap one anywhere on the page to see what it means and where it was first explained.

Word Meaning
accuracy The share of all predictions that are right. Misleading when one class is rare
classification Predicting a category (here: diabetes yes or no) instead of a number
coefficient A number the model learns for each feature. In logistic regression, a change in the log-odds
confusion matrix A table counting true positives, false positives, false negatives and true negatives
F1 One score combining precision and recall; high only if both are high
factor R’s type for categories. The outcome of a classification model must be a factor
false negative Has the condition, but the model says no
false positive Doesn’t have the condition, but the model says yes
imbalanced When one class is much rarer than the other, like diabetes here (13.8%)
intercept The model’s baseline, before any feature does anything
log-odds The scale summary() prints coefficients on. exp() turns a coefficient into an odds ratio
logistic regression A model that predicts a probability between 0 and 1 by bending a line into an S-curve
odds The chance something happens over the chance it doesn’t. A 1-in-10 chance is odds of 1 to 9
odds ratio exp(coefficient): how much a feature multiplies the odds. Above 1 raises them, below 1 lowers them
overfitting Learning quirks of the training data, so the model does worse on new data
p-value If the true effect were zero, how surprising a result like this would be. Small means surprising
precision Of the people flagged, the share who really have the condition. TP / (TP + FP)
prevalence The share of people who have a condition
recall Of the people who have the condition, the share the model finds. TP / (TP + FN). Also called sensitivity
specificity Of the people without the condition, the share the model leaves alone. TN / (TN + FP)
strata Splitting so that both sets keep the same share of each outcome class
test set The data held back to score the finished model
threshold The probability above which we predict “yes”. 0.5 by default
training set The data the model is fitted on
true negative Doesn’t have the condition, and the model says no
true positive Has the condition, and the model says yes