Working with Data in the Tidyverse

MY202 · Lab 1 · Self-paced guide

Author

Jonas Weinert

Published

1 October 2026

Before you start

This page walks you through Lab 1 one small step at a time. You don’t need any experience with R, coding, or statistics. If a word is new to you, it is explained the first time it appears, and there is a glossary at the end.

How it works. You keep this page open next to RStudio. The page explains each step and shows you exactly what you should see. You type the code into the lab notebook, a file that mixes instructions with boxes for code, and run it, which means asking R to carry it out. Both are explained step by step below. Take your time: there is no timer.

Your turn Something to do in RStudio. Follow the numbered instructions.

What you should see The real result R gives back. Compare it with yours.

Big idea A concept worth remembering beyond today.

Check yourself A quick question. Click an answer to see if you were right.

At the end of each step there is a Mark as done button. Your ticks are saved in this browser, so you can close the page and pick up where you left off. The bar at the top shows how far you’ve got.

Plan for about 2–3 hours

You don’t have to do it all in one go. Parts 1 and 2 are the most important. Part 3 is harder, and it’s fine to leave it for another day.

Open RStudio and the notebook

R is a programming language for working with data. RStudio is the program you use to write and run R code. Think of R as the engine and RStudio as the car’s dashboard: you only ever touch the dashboard.

  1. Find the lab folder you downloaded from Moodle, the course website. It contains a file ending in .qmd (the notebook) and a folder called data. Keep them together in the same folder.
  2. Open RStudio (not “R”). On Windows, including the lab computers, search for “RStudio” in the Start menu. On a Mac, press Cmd + Space and type “RStudio”.
  3. In RStudio, click File → Open File… in the menu at the top, go to the lab folder and open the .qmd file whose name contains student.

A zip file is a folder squashed into one file to make downloading easier. Unpack it first: on Windows, right-click it and choose Extract All…; on a Mac, double-click it. Then use the normal folder that appears, not the zip file.

Windows often hides the last part of file names (the file extension, which tells the computer what kind of file it is). Look for the file whose name contains “student” and whose type is shown as “Quarto” or has a blue RStudio-style icon. Opening it through File → Open File… in RStudio also works.

When the notebook is open, RStudio looks roughly like this. Each area is called a pane.

That’s fine. The layout can differ between computers. When the notebook is open, the Source pane appears; before that, the Console takes up the whole left side.

Notebooks, chunks and running code

The notebook mixes two things:

  • Text, which you read, like the instructions for each question.
  • Code chunks: grey boxes that contain R code. R only ever looks at what’s inside the grey boxes.

Here is what a chunk looks like, and what happens when you run it:

```{r}⚙ ▼ ▶ ← click the green triangle to run
1 + 1
```
[1] 2   ← the output appears here, under the chunk

There are two ways to run code:

To run… Do this
the whole chunk Click the green ▶ at the top right of the chunk, or press Ctrl + Shift + Enter
one line Click anywhere on that line and press Ctrl + Enter

Lines starting with # are comments. R ignores them: they are notes for humans. That’s why the empty chunks in the notebook say # Code here. You can type your code underneath that line.

Don’t type the ```{r} lines

The lines with ```{r} and ``` mark where a chunk starts and ends. Leave them alone and type your code between them.

Run the setup chunk

R comes with a set of basic tools. Other people have written extra tools and shared them as packages. Before you can use a package, you load it with library(). You do this once each time you open RStudio.

The notebook loads four packages:

Package What it gives you
tidyverse The main toolkit for today. The tidyverse is really a family of packages that are designed to work together and loads them all at once, including dplyr (summarising tables, Part 1), tidyr (reshaping tables, Part 2), readr (reading files, Part 2), purrr (repeating steps, Part 3) and tibble (a neater kind of table).
psymetadata The research data we use in Parts 1 and 3.
gt Turns a table into a nicely formatted one for reports (Part 2).
ggsci Colour schemes for graphs. Not needed today.

You don’t need to remember which function comes from which package. When this page says “from dplyr”, it just tells you where a tool lives.

  1. Scroll to the top of the notebook and find the chunk under Setup. It contains four lines starting with library(.
  2. Click the green ▶ on that chunk.
  3. Wait a few seconds. Messages may appear underneath.

You may see some text in red, like this (your version numbers may differ). That’s normal. Red text is not always an error (R refusing to run something; there’s more on errors below).

── Attaching core tidyverse packages ──────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.2
── Conflicts ───────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package to force all conflicts to become errors

The “Conflicts” part just says that two packages have a tool with the same name (for example filter()), and the one loaded last wins (“masks” the other). You can ignore it.

The package isn’t installed on this computer yet. Installing is a one-off step, like installing an app. Click in the Console (bottom left), type the line below with the missing package’s name, press Enter, wait until the > prompt comes back, then run the setup chunk again.

install.packages("psymetadata")

Do not put install.packages() inside the notebook: you only need it once, not every time.

R basics

Before we touch any data, let’s learn the handful of words R uses. Try each example yourself: click into the Console, type the code, and press Enter. Mistakes here cost nothing.

R as a calculator

Type each line into the Console and press Enter after each one.

2 + 3
10 / 4
2 * (3 + 1)
2 + 3
[1] 5
10 / 4
[1] 2.5
2 * (3 + 1)
[1] 8

The [1] at the start of each answer just means “this is the first value”. You’ll see it before almost every result, and you can ignore it for now.

Objects: giving a value a name

Usually you want R to remember a result so you can use it later. You do this by giving it a name. A named thing in R is called an object.

x <- 5

  1. The name of the new object. You choose it. Names can contain letters, numbers, _ and ., but no spaces.
  2. The assignment arrow: a < followed by a -. Read it as “gets”: “x gets 5”. Shortcut: Alt + -.
  3. The value to store.
x <- 5
x
x * 2
x <- 5
x
[1] 5
x * 2
[1] 10

Notice: the first line printed (showed) nothing. When you assign with <-, R stores the value silently. Look at the Environment pane (top right): x now appears there. Typing the name on its own (x) asks R to show you what’s inside.

If you don’t assign it, R forgets it. x * 2 showed 10, but x is still 5. To keep a result, you must save it with <-, either under a new name or by overwriting the old one.

R doesn’t know a thing called x yet. Either the line x <- 5 hasn’t been run, or the name is spelled differently. Capitals matter: X and x are two different names.

Functions: tools that do something

A function is a tool that takes something in, does something with it, and gives something back. You always recognise a function by its round brackets ( ).

round(3.14159, digits = 2)

  1. The function’s name: round rounds a number.
  2. The first argument: the input. Arguments are what you put inside the brackets.
  3. A named argument: an extra setting, written as name = value. Arguments are separated by commas.
round(3.14159, digits = 2)
[1] 3.14

Every function has a help page that lists its arguments. To open it, put a ? in front of the name and run it, for example ?round. The help appears in the Help tab, bottom right. Help pages are written for experienced users, so don’t worry if they look dense: the Description at the top and the Examples at the bottom are the useful parts.

Text, numbers and TRUE/FALSE

R treats different kinds of values differently. The kind is called the type.

Type Example What it is
numeric 57.1 a number
character "Niger" text. Always in quotes.
logical TRUE, FALSE yes or no; the answer to a question
missing NA “not available”: the value is missing

The quotes matter. Without them, R thinks a word is the name of an object and goes looking for it:

"Niger"
[1] "Niger"
Niger
Error:
! object 'Niger' not found

And "1970" in quotes is text, not a number: you can’t do maths with it. We’ll fix exactly this problem in Part 2.

Vectors: several values in a row

A vector is a sequence of values of the same type. You make one with the function c(), short for “combine”.

years <- c(1970, 1990, 2010)
years
[1] 1970 1990 2010
years + 1
[1] 1971 1991 2011

Most of R works on whole vectors at once: years + 1 added 1 to every value. A column in a table is a vector too.

Asking questions: == and %in%

You can ask R yes/no questions. The answer is TRUE or FALSE, once for every value.

Question Write Example
is equal to? == year == 1990
is not equal to? != design != "Experiment"
greater / less than? > < n > 100
is one of? %in% year %in% c(1970, 1990)
years == 1990
[1] FALSE  TRUE FALSE
years %in% c(1970, 2010)
[1]  TRUE FALSE  TRUE

filter() is a function that keeps only some rows of a table; you’ll use it in Part 2. One = sets an argument, like digits = 2. Two == ask “is this equal?”. If you mix them up, R tells you: “We detected a named input. This usually means that you’ve used = instead of ==.”

You have n <- 50. What does n > 100 give back?

50

> asks a question, so the answer is TRUE or FALSE, not a number.

FALSE

50 is not greater than 100, so the answer is FALSE.

An error

This is valid code: n exists and > compares two numbers.

Reading error messages

You will see a lot of errors today. Everyone does, including people who have used R for twenty years. An error doesn’t break anything: R just refuses to run that line and tells you why. Read the message, fix the line, run it again.

The message says… It usually means…
object 'xyz' not found A typo in a name, or you haven’t run the line that creates it yet
could not find function "xyz" A typo in the function name, or the setup chunk hasn’t been run
unexpected symbol / unexpected ')' A missing comma, bracket or quote. Count your brackets: every ( needs a )
there is no package called 'xyz' The package needs installing (see the setup step)
The + trap

If the Console shows + instead of > at the start of the line, R is waiting for you to finish something, usually a missing ) or ". Press Esc to cancel and start the line again.

Part 1: Summarising data

About 45–60 minutes. In this part you’ll explore a real research dataset and answer a question with it: do different kinds of studies find different results? You’ll learn the tools count(), mutate(), group_by() and summarise(), from a package called dplyr.

Where are we in the notebook?

Scroll down in the notebook to Part 1. Each step below matches a numbered question there. The notebook skips number 5 (it goes 4, 6, 7…). That’s a typo in the notebook, not something you missed.

About the data

Researchers often want to know what all the studies on a topic found, not just one. A meta-analysis collects the results of many studies and combines them.

Our dataset, agadullina2018, is from a meta-analysis on prejudice. It asks: are people more prejudiced towards groups they see as tight-knit and uniform (psychologists call this entitativity)? Each row is one result from one published study.

Each result is summarised as an effect size: a single number for how strong the relationship was. Bigger numbers mean a stronger link between seeing a group as uniform and being prejudiced against it. In this data the effect size is in the column yi.

The data comes with the psymetadata package, so it is already loaded: you don’t need to open a file.

Q1 Read the documentation

Before using any dataset, find out what it contains. Datasets that come in packages have a help page (also called documentation), just like functions.

  1. Find question 1 in Part 1 of the notebook.
  2. Click into the chunk under it and type the line below. Then run it.
?agadullina2018
  1. Look at the Help tab, bottom right.
Studies on Out-Group Entitativity and Prejudice

Description:

     Results from 21 studies, including 85 effect sizes
     (fisher-z), on the effect of out-Group entitativity and
     prejudice (Agadullina and Lovakov 2018).

Usage:

     data("agadullina2018")
     
Format:

     A data frame with 85 rows and 9 variables:

        • 'es_id': effect size id

        • 'study_id': study id

        • 'author': study author

        • 'pub_year': year of publication

        • 'n': sample size

        • 'design': within or between subjects design

        • 'ent_alpha': reliability of the entitativity measure

        • 'yi': effect size (fisher-z)

        • 'vi': sampling variance (SE^2)

Details:

     Further details can be found at https://osf.io/8dw5y/.

References:

     Agadullina ER, Lovakov AV (2018). "Are people more
     prejudiced towards groups that are perceived as coherent?
     A meta-analysis of the relationship between out-group
     entitativity and prejudice." _British Journal of Social
     Psychology_, *57*(4), 703-731.

The most useful part is Format: it lists each column (also called a variable) and what it means. Note three of them:

  • design: how the study was done. Cross-sectional studies measure people’s views at one point in time; experiments change something on purpose and see what happens.
  • yi: the effect size.
  • vi: how uncertain that effect size is (smaller is more precise). We won’t use it today.

The psymetadata package isn’t loaded yet. Run the setup chunk at the top of the notebook first, then try again.

Q2 What kind of object is it?

Everything in R is an object, and objects come in different kinds, called classes. The function class() tells you which kind you have.

In the chunk under question 2, type and run:

class(agadullina2018)
[1] "data.frame"

A data frame is R’s word for a table: rows and columns, like a spreadsheet. Each column is a vector, and all columns have the same length.

Q3 Print the whole thing

Typing the name of an object on its own and running it prints it: R shows you what’s inside.

In the chunk under question 3, type and run:

agadullina2018

In the notebook, a table appears under the chunk with arrows to page through it. Scroll and click around: there are 85 rows (one per result) and 9 columns.

Why we don’t usually do this

This table is small. Real datasets often have millions of rows, and printing them all would flood your screen. The notebook warns you not to print whole data frames in the exam. The next step shows a better way.

Q4 Convert it to a tibble

A tibble is a modern kind of data frame. It’s the same table, but it prints more helpfully: only the first few rows, plus the size of the table and the type of each column. All the tidyverse tools work with tibbles.

We’ll make a tibble version of the data and give it a new name, so we can use it in the rest of Part 1.

agadullina2018_tbl <- as_tibble(agadullina2018)

  1. The new name we’re choosing. _tbl at the end is a reminder that it’s a tibble. Any name works, but you must use the same one later.
  2. Assign: store the result under that name.
  3. The function as_tibble() converts a data frame to a tibble.
  4. The input: the original data frame. It is not changed: we get a new copy.

In the chunk under question 4, type both lines and run the chunk:

agadullina2018_tbl <- as_tibble(agadullina2018)
agadullina2018_tbl

The first line creates the tibble (silently). The second line prints it. Check the Environment pane: agadullina2018_tbl should now be listed.

How to read this table:

  • The words under each column name, like <chr> and <dbl>, are the column’s type: <chr> is character (text), <dbl> is a number with decimals (“double”), <int> a whole number.
  • At the bottom, the table says how many rows there are in total.

If you type agadullina2018_tbl in the Console instead of the notebook, the tibble prints as text. The first line tells you its size: # A tibble: 85 × 9 means 85 rows and 9 columns.

# A tibble: 85 × 9
  es_id study_id authors  pub_year     n design ent_alpha     yi      vi
  <int>    <int> <chr>    <chr>    <int> <chr>      <dbl>  <dbl>   <dbl>
1     1        1 Agadull… 2017       193 Cross…     0.897  0.814 0.00526
2     2        1 Agadull… 2017       193 Cross…     0.897  0.228 0.00526
3     3        1 Agadull… 2017       193 Cross…     0.897  0.521 0.00526
4     4        1 Agadull… 2017       193 Cross…     0.897  0.657 0.00526
5     5        2 Agadull… 2015        88 Cross…     0.839 -0.383 0.0118 
# ℹ 80 more rows

glimpse(agadullina2018_tbl) turns the table on its side: one line per column, with the first few values. It’s handy when a table has too many columns to fit on the screen.

Rows: 85
Columns: 9
$ es_id     <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1…
$ study_id  <int> 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3…
$ authors   <chr> "Agadullina & Lovakov", "Agadullina & Lovakov", "Aga…
$ pub_year  <chr> "2017", "2017", "2017", "2017", "2015", "2015", "201…
$ n         <int> 193, 193, 193, 193, 88, 88, 298, 587, 587, 559, 559,…
$ design    <chr> "Cross-sectional", "Cross-sectional", "Cross-section…
$ ent_alpha <dbl> 0.897, 0.897, 0.897, 0.897, 0.839, 0.839, 0.765, 0.8…
$ yi        <dbl> 0.81439746, 0.22761291, 0.52077996, 0.65650427, -0.3…
$ vi        <dbl> 0.005263158, 0.005263158, 0.005263158, 0.005263158, …

You printed the tibble before creating it. Run the line with <- first. If you run the whole chunk with ▶, both lines run in order.

Q6 Count the study designs

Our question is whether different kinds of studies find different results. First: how many results come from each kind of study? The function count() counts how often each value appears in a column.

count(agadullina2018_tbl, design)

  1. The function: count how many rows have each value.
  2. Which table to use. In the tidyverse, the table always comes first.
  3. Which column to count. Column names don’t need quotes inside tidyverse functions.
count(agadullina2018_tbl, design)

The result is a new, small table: each design, and in the column n, how many rows have it. So 71 results come from cross-sectional studies and 14 from experiments.

Q7 The pipe |>

The pipe |> is the most useful symbol you’ll learn today. It takes whatever is on its left and passes it into the function on its right, as the first input. Read it as “and then”.

Without a pipe

count(agadullina2018_tbl, design)

With a pipe

agadullina2018_tbl |>
  count(design)

Both lines do exactly the same thing. The second reads: “take agadullina2018_tbl, and then count design”.

  1. In the chunk under question 7, type the piped version.
  2. To type the pipe, press Ctrl + Shift + M, or type | then >. The | key is usually above Enter or next to Shift.
  3. Press Enter after the pipe to start a new line. RStudio indents the next line for you (moves it in a little), which shows that it belongs to the same instruction.
  4. Run the chunk.

The same result as Q6. With one step the pipe makes no difference. Its benefit shows when you chain several steps, as you’ll do next.

%>% is an older version of the pipe and works the same way for today. To switch: Tools → Global Options → Code, tick Use native pipe operator, |>, and click OK.

There’s a space between | and >. They must be typed right next to each other: |>.

Q8 Add a percentage column with mutate()

Counts are useful, but percentages are easier to compare. mutate() adds a new column (or changes an existing one), calculated from the columns you already have.

mutate(pct = (n / sum(n)) * 100)

  1. The function: add or change a column.
  2. The name of the new column. You choose it; pct is short for percentage.
  3. A single =: here it means “set the new column to…”.
  4. The calculation: each count n, divided by the total of all counts sum(n), times 100.

Now chain it onto the counting step with another pipe. Click Start below to go through the code one line at a time.

agadullina2018_tbl |>
  count(design) |>
  mutate(pct = (n / sum(n)) * 100)

Start with the data: one row per result, 85 rows.

and then count the designs. The pipe passes the table into count(). Now we have a small table with 2 rows.

and then add a column. mutate() receives the small count table, and calculates pct for each row: 71 out of 85 is 83.5%, 14 out of 85 is 16.5%.

Type the three lines into the chunk under question 8 and run it. Notice that every line except the last ends with a pipe: that’s what tells R the next line continues the same instruction.

n only exists after count() has made it. Check that count(design) comes before mutate(), and that the line with count ends with |>.

Q9 Average effect size with summarise()

Now the actual research question. What is the average effect size? summarise() collapses a whole table into a single row of summary numbers. Inside it, mean() calculates an average.

summarise(agadullina2018_tbl, yi = mean(yi))

  1. The function: collapse many rows into one.
  2. Which table.
  3. The name of the column in the result. We reuse yi, but you could call it mean_effect or anything else.
  4. What to calculate: the mean (average) of the column yi.
summarise(agadullina2018_tbl, yi = mean(yi))

The notebook says “per study”, but this calculates one average over all 85 results. Splitting it up comes in the next step.

One row, one number: on average, the effect size is about 0.41.

Q10 Averages per group with group_by()

But do cross-sectional studies and experiments find the same thing? To get one average per design, put group_by(design) before summarise().

group_by() on its own changes nothing you can see. It puts an invisible label on each row, saying which group it belongs to. It changes how the next step behaves: summarise() then makes one row per group instead of one row in total.

agadullina2018_tbl |>
  group_by(design) |>
  summarise(yi = mean(yi))

Start with the data: 85 rows.

group_by(design) sorts the rows into two groups. The table looks exactly the same. Only when it’s printed as text do you see a hint: the line # Groups: design [2] says there are 2 groups.

# A tibble: 85 × 9
# Groups:   design [2]
  es_id study_id authors  pub_year     n design ent_alpha     yi      vi
  <int>    <int> <chr>    <chr>    <int> <chr>      <dbl>  <dbl>   <dbl>
1     1        1 Agadull… 2017       193 Cross…     0.897  0.814 0.00526
2     2        1 Agadull… 2017       193 Cross…     0.897  0.228 0.00526
3     3        1 Agadull… 2017       193 Cross…     0.897  0.521 0.00526
4     4        1 Agadull… 2017       193 Cross…     0.897  0.657 0.00526
5     5        2 Agadull… 2015        88 Cross…     0.839 -0.383 0.0118 
# ℹ 80 more rows

summarise() now works within each group: one row per design, each with its own average. Cross-sectional studies: 0.468. Experiments: 0.111.

Type the three lines into the chunk under question 10 and run it. Then try removing the group_by line and running it again: you’re back to one row.

group_by(design) is missing, or it comes after summarise(). The order matters: group first, then summarise.

Q11 What have we found out?

Time to put the results into words. This is the point of all the code: answering a question.

Which statement matches our results?

Experiments find larger effects than cross-sectional studies.

Look again at the Q10 result: experiments average 0.111 and cross-sectional studies 0.468.

Most results come from cross-sectional studies, and they find much larger effects than experiments.

84% of the results are cross-sectional (Q8), and their average effect is about four times as large as for experiments (Q10).

Both kinds of study find about the same effect.

0.468 and 0.111 are quite different. The kind of study seems to matter.

Something to think about

A simple average treats every result as equally trustworthy. But some come from big, precise studies and some from small ones (that’s what vi measures). Real meta-analyses give more weight to more precise results. You’ll learn how later in the course.

Part 1 done. You’ve used four of the tools that cover most everyday data work. A good moment for a break.

Part 2: Tidy data and reshaping

About 45–60 minutes. Here you’ll load a real file from the World Bank, discover that its layout makes it hard to analyse, and reshape it. You’ll learn read_csv(), rename(), pull(), pivot_longer(), filter(), pivot_wider() and gt(), from the readr, dplyr, tidyr and gt packages. Each is explained when you get to it.

What is tidy data?

The same information can be laid out in a table in many ways. Most R tools expect one particular layout, called tidy data:

Illustration: in tidy data, each variable forms a column, each observation forms a row, and each cell is a single measurement

Illustration: Julia Lowndes and Allison Horst, Tidy Data for reproducibility, efficiency, and collaboration, Openscapes, 2020. CC BY 4.0.

  • A variable is something you measure: a country, a year, a life expectancy.
  • An observation is one “case” you measured: here, one country in one year.

Our World Bank file shows life expectancy (how many years a newborn baby can expect to live) for every country, for every year from 1960 to 2024. It’s laid out like this:

Wide: how the file looks 263 rows × 67 columns

Country Name 1970 1990 2010 …
Burkina Faso 39.5 49.3 56.3 …
Honduras 54.4 65.2 69.8 …
Niger 35.9 41.7 57.1 …

Easy for a human to read. But year is a variable, and here it is hidden in the column names. There’s no column called “year” that you could filter or group by.

Long (tidy): what we want 3 columns

country_name year lifeexp
Burkina Faso 1970 39.5
Burkina Faso 1990 49.3
Burkina Faso 2010 56.3
Honduras 1970 54.4
… … …

One row per country and year. Every variable has its own column, so the tools from Part 1 work.

Wide is good for showing a table to people. Long is good for analysing it. Most data cleaning is about getting data into the long, tidy shape.

Q1 Load the file and rename a column

So far the data came inside a package. Now we load a file from your computer. The file is a CSV (“comma-separated values”): a plain-text table where commas separate the columns. Excel can open it too.

read_csv() reads a CSV file and turns it into a tibble. You tell it where the file is with a path:

read_csv("data/wb-life-exp.csv")

  1. The function that reads CSV files.
  2. The path: “inside the folder data, the file wb-life-exp.csv”. It’s text, so it goes in quotes. The / separates a folder from what’s inside it.

The path starts from the folder the notebook is in. So this only works if the data folder sits next to the notebook file. You can check in the Files tab, bottom right.

The first column is called Country Name, with a space and capitals. Spaces in names are awkward in code, so we rename it to country_name, using rename(new_name = old_column). Instead of typing the old name, you can give its position: 1 means “the first column”.

lifeexp <- read_csv("data/wb-life-exp.csv") |>
  rename(country_name = 1)
lifeexp

read_csv() reads the file. Look at the first column’s name, Country Name, and the year columns: 1960, 1961, … R also prints a short message about the column types it guessed.

and then rename() changes the first column’s name to country_name. The whole result is saved under the name lifeexp (short for life expectancy), thanks to the <- at the start.

The arrow saves the result of the whole pipeline, not just the first line.

Print it to check. The first column is now called country_name.

  1. Scroll to Part 2 in the notebook, question 1.
  2. Type the three lines and run the chunk.
  3. Check that lifeexp appears in the Environment pane, with 263 obs. (observations, or rows) of 67 variables.

R can’t find the file at that path. The working directory is the folder R starts looking in; for a notebook, that’s the folder the notebook is in. Check, in this order:

  1. In the Files tab, go to the folder that contains the notebook. Is there a folder called data, with wb-life-exp.csv inside?
  2. If the CSV is next to the notebook instead (not in a data folder), drop the folder from the path: read_csv("wb-life-exp.csv").
  3. Did the download rename the file, for example to wb-life-exp (1).csv? Rename the file, or use its exact name in the path.
  4. Did you open the notebook straight from your Downloads or from inside a zip file? Unzip the folder first, then open the notebook again from the unzipped folder.

Q2 Look at all the countries with pull()

pull() takes one column out of a table and gives it to you as a plain vector: just the values, in a row. It’s a quick way to look through everything in a column.

pull(lifeexp, country_name)

Scroll through the output. Is every name a country?

 [1] "Aruba"                          "Africa Eastern and Southern"   
 [3] "Afghanistan"                    "Africa Western and Central"    
 [5] "Angola"                         "Albania"                       
 [7] "Andorra"                        "Arab World"                    
 [9] "United Arab Emirates"           "Argentina"                     
[11] "Armenia"                        "American Samoa"                
[13] "Antigua and Barbuda"            "Australia"                     
[15] "Austria"                        "Azerbaijan"                    
[17] "Burundi"                        "Belgium"                       
[19] "Benin"                          "Burkina Faso"                  
[21] "Bangladesh"                     "Bulgaria"                      
[23] "Bahrain"                        "Bahamas, The"                  
[25] "Bosnia and Herzegovina"         "Belarus"                       
[27] "Belize"                         "Bermuda"                       
[29] "Bolivia"                        "Brazil"                        
[31] "Barbados"                       "Brunei Darussalam"             
[33] "Bhutan"                         "Botswana"                      
[35] "Central African Republic"       "Canada"                        
[37] "Central Europe and the Baltics" "Switzerland"                   
[39] "Channel Islands"                "Chile"                         
[41] "China"                          "Cote d'Ivoire"                 
[43] "Cameroon"                       "Congo, Dem. Rep."              
[45] "Congo, Rep."                    "Colombia"                      
[47] "Comoros"                        "Cabo Verde"                    
[49] "Costa Rica"                     "Caribbean small states"        
[51] "Cuba"                           "Curacao"                       
[53] "Cayman Islands"                 "Cyprus"                        
[55] "Czechia"                        "Germany"                       
[57] "Djibouti"                       "Dominica"                      
[59] "Denmark"                        "Dominican Republic"            
Real data is messy

Not every row is a country. About 46 rows are groups of countries: “Arab World”, “Euro area”, “High income”, “World”, and so on. If you averaged across all rows, you’d count many countries twice. Always look at your data before you analyse it.

Q3 Reshape from wide to long with pivot_longer()

Now the main event. pivot_longer() turns a wide table into a long one. It needs answers to three questions:

pivot_longer(-country_name, names_to = "year", values_to = "lifeexp")

  1. The function: make the table longer.
  2. Which columns should be stacked? All the year columns. Rather than listing 65 years, we say “everything except country_name”: the minus sign means “not this one”.
  3. Where should the old column names go? Into a new column called year. Its name is new, so it’s in quotes.
  4. Where should the numbers go? Into a new column called lifeexp.

Before one row per country

country_name 1970 1990
Niger 35.9 41.7

After one row per country and year

country_name year lifeexp
Niger "1970" 35.9
Niger "1990" 41.7
lifeexp |>
  pivot_longer(-country_name, names_to = "year", values_to = "lifeexp")

You can split a long line after any comma to make it easier to read. RStudio keeps it as one instruction until the closing bracket.

The table now has 3 columns and 17,358 rows: 263 rows × 66 year columns. Look at the type under year: <chr>, text. The years came from column names, and column names are always text. We fix that next.

Why are there NAs?

NA means a value is missing. Some countries have no data for early years, and the file’s last column, 2025, is completely empty because that year isn’t over yet. You could remove those rows by adding values_drop_na = TRUE inside pivot_longer().

New column names in names_to and values_to need quotes: "year", "lifeexp". And check the commas between the three parts.

Q4 Turn the years into numbers with parse_number()

parse_number() takes text and pulls the number out of it: "1970" becomes 1970. We use it inside mutate(), which you know from Part 1. This time, mutate() replaces the year column with a new version of itself.

mutate(year = parse_number(year))

  1. Change or add a column.
  2. The column to create. It already exists, so it is overwritten.
  3. The new values: the numbers from the old text in year.

Copy your code from Q3 into the chunk under Q4, add a pipe at the end of the last line, and add the mutate line:

lifeexp |>
  pivot_longer(-country_name, names_to = "year", values_to = "lifeexp") |>
  mutate(year = parse_number(year))

The type under year is now <dbl>: a number. Now we can ask number questions like “is the year 1990?”.

Each question adds one line to the end of the previous answer. You never rewrite from scratch: copy the last working version, add |>, add the next step, run. If it breaks, the problem is almost always in the line you just added.

Q5 Keep four countries and three years with filter()

filter() keeps the rows for which a question is answered TRUE, and drops the rest. Remember %in% from the basics: it asks “is this value one of these?”.

filter(country_name %in% c("Niger", "Honduras"), year %in% c(1970, 1990, 2010))

  1. Keep the rows where the answers are TRUE.
  2. First question: is the country one of these? Country names are text, so they’re in quotes, and must match the data exactly, including capitals.
  3. Second question: is the year one of these? Years are now numbers, so no quotes. A comma between questions means both must be true.
  1. Pick four countries you’re curious about. Use the names from Q2 to copy their exact spelling.
  2. Add a filter() line to your pipeline. Here’s an example with four countries; swap in your own:
lifeexp |>
  pivot_longer(-country_name, names_to = "year", values_to = "lifeexp") |>
  mutate(year = parse_number(year)) |>
  filter(country_name %in% c("Burkina Faso", "Honduras", "Niger", "Suriname"),
         year %in% c(1970, 1990, 2010))

4 countries × 3 years = 12 rows.

  • A country name doesn’t match the data exactly. Copy it from the pull() output. For example, the data says “Egypt, Arab Rep.”, not “Egypt”.
  • The years are in quotes. After Q4 they’re numbers, so write 1970, not "1970".
  • You used == instead of %in%. == compares values one by one in order, and quietly gives the wrong rows. To check against several values, always use %in%.

Q6 Back to wide with pivot_wider()

The long format was good for filtering. But to show the result to someone, one row per country with a column per year is easier to read. pivot_wider() is the reverse of pivot_longer().

pivot_wider(names_from = year, values_from = lifeexp)

  1. The function: make the table wider.
  2. Where do the new column names come from? From the values in year. The column exists already, so no quotes.
  3. Where do the values in the cells come from? From lifeexp.

Add one more pipe and the pivot_wider() line to your pipeline from Q5:

lifeexp |>
  pivot_longer(-country_name, names_to = "year", values_to = "lifeexp") |>
  mutate(year = parse_number(year)) |>
  filter(country_name %in% c("Burkina Faso", "Honduras", "Niger", "Suriname"),
         year %in% c(1970, 1990, 2010)) |>
  pivot_wider(names_from = year, values_from = lifeexp)

To keep one decimal place, add lifeexp = round(lifeexp, 1) to the mutate(): mutate(year = parse_number(year), lifeexp = round(lifeexp, 1)). The completed notebook does this, and also shows five years (1970, 1980, 1990, 2000, 2010).

Q7 Make a nice table with gt()

gt() (from the gt package, short for “grammar of tables”) turns a tibble into a formatted table that you could put in a report.

Add |> gt() to the end of your pipeline. Here is the whole thing. Step through it below to see what each line does.

This is the full pipeline from Part 2. Press Start to replay it one step at a time.

lifeexp |>
  pivot_longer(-country_name,
               names_to = "year",
               values_to = "lifeexp") |>
  mutate(year = parse_number(year),
         lifeexp = round(lifeexp, 1)) |>
  filter(country_name %in% c("Burkina Faso",
           "Honduras", "Niger", "Suriname"),
         year %in% c(1970, 1990, 2010)) |>
  pivot_wider(names_from = year,
              values_from = lifeexp) |>
  gt()

The wide table we loaded: 263 rows, one per country or region; one column per year.

pivot_longer() stacks the year columns: one row per country and year, 17,358 rows.

mutate() turns year into a number and rounds life expectancy to one decimal.

filter() keeps four countries in three years: 12 rows.

pivot_wider() spreads the years back out into columns: one row per country.

gt() formats the table for a report.

country_name 1970 1990 2010
Burkina Faso 39.5 49.3 56.3
Honduras 54.4 65.2 69.8
Niger 35.9 41.7 57.1
Suriname 59.5 62.1 69.6

Q8 What have you found out?

Look at your table. Which of your countries gained the most years of life expectancy between 1970 and 2010? Did any of them go down for a while?

In the example table, which country gained the most between 1970 and 2010?

Suriname

Suriname went from 59.5 to 69.6: about +10 years, the smallest gain of the four.

Niger

Niger went from 35.9 to 57.1: more than 21 extra years. Most of that gain came after 1990.

Honduras

Honduras gained about 15 years (54.4 to 69.8): a lot, but less than Niger.

read → rename → reshape → fix types → filter → reshape back → present. You’ve just done a complete small analysis. Most data projects follow this same shape, just with more steps.

Part 3: Doing the same thing many times

About 30–45 minutes. This part is harder, so go slowly, and it’s fine to stop here and come back another day. You’ll learn map() and bind_rows(), from the purrr and dplyr packages.

Data that comes in pieces

Sometimes data doesn’t arrive as one table but as several: one file per year, one spreadsheet tab per region. In R, you can keep several tables together in a list.

A list is a container that can hold anything, including whole tables. Think of it as a box with several labelled compartments.

The data for this part is another meta-analysis, maldonado2020: 1,268 results on how ageing affects executive functions, the mental skills you use to plan, focus and switch between tasks. The notebook splits it into a list of four tables, one per skill:

split_data: a list of 4 tables

289 rowsinhibition
ignoring distractions
270 rowsprocessing speed
how fast you think
208 rowsshifting
switching between tasks
501 rowsupdating
working memory

Find question 1 in Part 3. The chunk already contains code that builds split_data. You don’t need to understand it: just run it with the green ▶. Check that split_data appears in the Environment pane as a “List of 4”.

maldonado2020 comes with the psymetadata package. Run the setup chunk at the top of the notebook, then this chunk again.

Q2 Apply a function to every piece with map()

Say you want to see the first 5 rows of each table. You could write the same line four times, once per table. But that’s tedious, and every copy is a chance for a typo. map() does it for you: it takes a list and a function, and applies the function to every piece.

A list
4 tables
→
map(list, f)
do f to each one
→
A list
4 results

To say what to do to each piece, you write a small recipe with a ~ (tilde) and .x:

map(split_data, ~ head(.x, 5))

  1. The function: do something to every piece of a list.
  2. Which list.
  3. The tilde ~ means “here comes a recipe”. On most keyboards it’s Shift + the key next to 1 or next to Enter.
  4. The recipe: what to do with each piece. head() shows the first rows of a table.
  5. .x is a placeholder for “the current piece”. map() swaps in each table in turn: first inhibition, then processing speed, and so on.
  6. Show 5 rows.
map(split_data, ~ head(.x, 5))
$inhibition
           author       yi         vi              task
12    Bugg, 2014A 1.957228 272.179381            Stroop
13    Bugg, 2014B 3.391032 164.365623            Stroop
14    Bugg, 2014C 3.409401 163.539770            Stroop
17 Circelli, 2013 1.077655   1.261782 Anti-saccade Task
18     Cona, 2013 1.787737  28.994392     Go/No-Go Task

$`processing speed`
          author         yi          vi n1 n2 mean_age1 mean_age2
3  Balouch, 2013  1.5349718   0.9803923 28 29     24.64     67.86
8  Bernard, 2013  0.6710382 412.2254792 10 16      21.7      61.9
19  Gillis, 2013 -0.3854916   0.8525118 25 34     20.52     68.12
25   Kelly, 2013  2.2941662   0.1349244 48 48     18-24     60-79
26   Kelly, 2013  2.5073828   0.2895045 48 48     18-24     60-79

$shifting
          author         yi           vi              task
2  Balouch, 2013 -0.2578337 7.189572e-02 Trail Making Test
5  Barulli, 2013  2.7258066 3.450196e+04     Shifting Task
6  Bernard, 2013  1.2530391 1.939528e+00 Trail Making Test
10   Brand, 2013  0.7824283 5.386653e-02     Modified WCST
20  Gillis, 2013 -0.1181795 7.892916e-01 Trail Making Test

$updating
          author        yi         vi  n1 n2 mean_age1 mean_age2
1  Aguirre, 2014 0.8708623  2.2852193  48 48      20.5     68.31
4  Balouch, 2013 0.3963551  0.1139885  28 29     24.64     67.86
7  Bernard, 2013 2.0009349 82.3702214  10 16      21.7      61.9
9    Brand, 2013 0.4282817  0.1627862 251 46     24.55     70.35
11   Brand, 2013 0.7656311  2.4470096 251 46     24.55     70.35

The result is again a list of four. Each piece is labelled with $ and its name, like $inhibition. Notice that the pieces have different columns: only author, yi and vi appear in all four.

You forgot the ~ before the recipe. Without it, R tries to run head(.x, 5) straight away, before map() has filled in .x.

Q3 Try other functions inside map()

If the function needs nothing but the piece itself, you can just give its name, without ~ or .x. For example, colnames() lists a table’s column names:

Try these, one at a time. Run each and look at what comes back.

map(split_data, colnames)
map(split_data, as_tibble)
map(split_data, ~ select(.x, author, yi, vi))
$inhibition
[1] "author" "yi"     "vi"     "task"  

$`processing speed`
[1] "author"    "yi"        "vi"        "n1"        "n2"       
[6] "mean_age1" "mean_age2"

$shifting
[1] "author" "yi"     "vi"     "task"  

$updating
[1] "author"    "yi"        "vi"        "n1"        "n2"       
[6] "mean_age1" "mean_age2"

select() is new: it keeps only the columns you name. Where filter() picks rows, select() picks columns. We keep the three columns that every piece has.

Now combine the last two into one recipe. Inside the recipe, you can use a pipe, just like before:

map(
  split_data,
  ~ .x |>
    as_tibble() |>
    select(author, yi, vi)
)

map() is going to go through split_data, piece by piece. Spreading the code over several lines makes it easier to read; the brackets show where map( starts and ends.

For each piece: start with that piece, .x, and then…

…turn it into a tibble, and then…

…keep the three columns author, yi and vi. The ) on the last line closes map(. The result: a list of four tibbles, all with the same three columns. (Shortened to 3 rows each here.)

$inhibition
# A tibble: 289 × 3
  author         yi    vi
  <chr>       <dbl> <dbl>
1 Bugg, 2014A  1.96  272.
2 Bugg, 2014B  3.39  164.
3 Bugg, 2014C  3.41  164.
# ℹ 286 more rows

$`processing speed`
# A tibble: 270 × 3
  author            yi      vi
  <chr>          <dbl>   <dbl>
1 Balouch, 2013  1.53    0.980
2 Bernard, 2013  0.671 412.   
3 Gillis, 2013  -0.385   0.853
# ℹ 267 more rows

$shifting
# A tibble: 208 × 3
  author            yi         vi
  <chr>          <dbl>      <dbl>
1 Balouch, 2013 -0.258     0.0719
2 Barulli, 2013  2.73  34502.    
3 Bernard, 2013  1.25      1.94  
# ℹ 205 more rows

$updating
# A tibble: 501 × 3
  author           yi     vi
  <chr>         <dbl>  <dbl>
1 Aguirre, 2014 0.871  2.29 
2 Balouch, 2013 0.396  0.114
3 Bernard, 2013 2.00  82.4  
# ℹ 498 more rows

You selected a column that only some pieces have. map() applies the same recipe to every piece, so it must work for all of them. Check with map(split_data, colnames) and pick columns that appear everywhere.

Q4 Stack the pieces back together with bind_rows()

Now that every piece has the same columns, we can stack them on top of each other into one big table. bind_rows() does exactly that.

But if we just stack them, we lose track of which piece each row came from. .id = "domain" adds a new column, domain, that records the name of the piece.

map(
  split_data,
  ~ .x |>
    as_tibble() |>
    select(author, yi, vi)
) |>
  bind_rows(.id = "domain")

The list of four tibbles from Q3, each with the same three columns.

$inhibition
# A tibble: 289 × 3
  author         yi    vi
  <chr>       <dbl> <dbl>
1 Bugg, 2014A  1.96  272.
2 Bugg, 2014B  3.39  164.
3 Bugg, 2014C  3.41  164.
# ℹ 286 more rows

$`processing speed`
# A tibble: 270 × 3
  author            yi      vi
  <chr>          <dbl>   <dbl>
1 Balouch, 2013  1.53    0.980
2 Bernard, 2013  0.671 412.   
3 Gillis, 2013  -0.385   0.853
# ℹ 267 more rows

$shifting
# A tibble: 208 × 3
  author            yi         vi
  <chr>          <dbl>      <dbl>
1 Balouch, 2013 -0.258     0.0719
2 Barulli, 2013  2.73  34502.    
3 Bernard, 2013  1.25      1.94  
# ℹ 205 more rows

$updating
# A tibble: 501 × 3
  author           yi     vi
  <chr>         <dbl>  <dbl>
1 Aguirre, 2014 0.871  2.29 
2 Balouch, 2013 0.396  0.114
3 Bernard, 2013 2.00  82.4  
# ℹ 498 more rows

and then bind_rows() stacks them into one table of 1,268 rows. The new first column, domain, says which piece each row came from.

Type the code into the chunk under question 4 and run it. Check: the result should have 1,268 rows and 4 columns.

Q5 Bonus: both steps at once with map_dfr()

map_dfr() does map() and bind_rows() in one go. The dfr stands for “data frame, by rows”. It gives the same table as Q4.

map_dfr(
  split_data,
  ~ .x |>
    as_tibble() |>
    select(author, yi, vi),
  .id = "domain"
)
You may see map_dfr() described as “superseded”

That means there’s a newer way to do the same thing, map(...) |> list_rbind(), but the old one still works. You’ll see map_dfr() in a lot of code online.

Wrap-up

What you learned

Part 1 · dplyr count, mutate, group_by and summarise, chained with |>, cover most everyday summaries.

Part 2 · tidyr Tidy data has one variable per column and one observation per row. pivot_longer() and pivot_wider() switch between shapes.

Part 3 · purrr map() applies one recipe to every piece of a list. bind_rows() stacks the results.

One question to take away: think of the data you might use for your own project. Is it tidy? If not, which rule does it break?

Save your work

Press Ctrl + S in RStudio to save the notebook. Your code is saved in the .qmd file, so next time you can open it and run everything again.

Further reading

  • R for Data Science (2e), free online: the chapters on data transformation, data tidying and iteration cover today’s material in more depth.
  • Two-page cheat sheets for dplyr and tidyr, also under Help → Cheat Sheets in RStudio.

Next week: making graphs with ggplot.

Glossary

Underlined words on this page are explained in this list. Click or tap one anywhere on the page to see what it means and where it was first explained.

Word Meaning
argument An input to a function, written inside its brackets: round(3.14, digits = 1)
assign, <- Store a value under a name, so R remembers it
chunk A grey box of R code in a notebook
class The kind of object, e.g. "data.frame". Check with class()
comment A line starting with #. R ignores it: it’s a note for humans
console The pane where you can type R code line by line
CSV A plain-text table file where commas separate the columns
data frame R’s word for a table with rows and columns
documentation The help page for a function or dataset. Open it with ?, e.g. ?round
effect size A number for how strong a relationship is in a study
file extension The end of a file name after the dot, e.g. .csv or .qmd. It tells the computer what kind of file it is
function A tool that takes inputs and gives back a result. Always has brackets: mean()
list A container that can hold several objects, even whole tables
meta-analysis A study that combines the results of many other studies
NA A missing value
object Anything R has stored under a name
observation One case you measured, e.g. one country in one year; in tidy data, a row
package A bundle of extra functions written by others. Install once, load with library() every session
pane One of the four areas of the RStudio window
path Where a file is, written as folders and a file name: "data/wb-life-exp.csv"
pipe, |> “And then”: passes the result on the left into the function on the right
pipeline Several steps chained together with |>, read top to bottom
tibble A modern data frame that prints more neatly
tidy data One variable per column, one observation per row, one value per cell
tidyverse A family of packages that work together, loaded with library("tidyverse"). Includes dplyr, tidyr, readr, purrr and tibble
type The kind of value: number, text (character), TRUE/FALSE (logical)
variable Something measured; in tidy data, a column
vector Several values of the same type in a row, made with c()
working directory The folder R looks in when you give it a file name. For a notebook, the folder the notebook is in
zip file A folder squashed into one file for downloading. Unpack (extract) it before using what’s inside