Fewer dimensions: PCA and MCA
MY202 · Lab 7 · Self-paced guide
Before you start
This page walks you through Lab 7 one small step at a time. It assumes you have done Labs 1 to 5: you know notebooks, chunks, the pipe |>, ggplot, and Lab 5’s recipe() with step_normalize(). Everything new is explained the first time it appears, and there is a glossary at the end.
What’s new this week. This is the first unsupervised lab. Until now there was always an outcome to predict: life expectancy, sales, diabetes, PTSD, childcare prices. Today there is none. Instead we ask what structure the variables have among themselves, and we use it to replace many variables with a few:
- PCA (principal component analysis) does this for numbers.
- MCA (multiple correspondence analysis) does it for categories.
Each part goes in the same order: first the idea, with an everyday example you can work out in your head; then the code; then your turn in the notebook.
Your turn Something to do in the notebook. The box says which notebook lines to go to.
What you should see The real result R gives back. Compare it with yours.
Big idea A concept worth remembering beyond today.
Check yourself A quick question. Click an answer to see if you were right.
At the end of each step there is a Mark as done button. Your ticks are saved in this browser, so you can close the page and pick up where you left off.
There is no code to write this week. Every chunk in the student notebook (my202-07-lab-student.qmd) is ready to run. The work is in the notebook’s reflection questions: what does this plot say? So for every question, write your own answer into the notebook first, then open the “Compare with our answer” box. All line numbers on this page refer to the student notebook.
Part I (PCA) takes a little over an hour, Part II (MCA) about 45 minutes. Nothing runs for more than a few seconds.
Open the notebook and run the setup chunk
- Open the Lab 7 student notebook on Nuvolos.
- Run the chunk at lines 25–36. It loads eight packages.
| Package | What it gives you |
|---|---|
tidymodels, tidyverse |
As before. New today: step_pca(). |
psychTools |
The data for Part I, holzinger.swineford. |
reshape2 |
melt(), which reshapes a correlation matrix for plotting. |
FactoMineR |
MCA(), the function for Part II. |
factoextra |
Ready-made plots of an MCA: fviz_mca_biplot(), fviz_contrib(). |
corrplot, patchwork |
Loaded, but not needed today. |
Run install.packages(c("FactoMineR", "factoextra", "psychTools", "reshape2")) once. Part II reads data/wvs-wave-7-ethical-norms.csv, so the file from Moodle must sit in a folder called data next to the notebook.
The idea: why fewer dimensions?
Read the callout “The curse of dimensionality”, then come back.
A dimension is a variable: a column you could draw as an axis. A bank wants to find customers similar to one particular customer (the diamond). With one variable, income, that is easy: several customers are within £6k. Add a second variable, debt, and draw the same £6k circle:
Same 30 customers, one more variable, and the neighbourhood has emptied. Every variable you add pushes everyone further apart. With many variables something stranger happens: everyone ends up about equally far from everyone else.
With 2 variables, the farthest point is 32 times as far away as the nearest. With 1,000 it is 1.1 times: “nearest” has stopped meaning anything. That is bad news for any method that relies on similarity, like Lab 5’s k nearest neighbours. On top of that, models with many columns overfit more easily and run more slowly.
Dimensionality reduction squeezes many variables into a few new ones that keep most of the structure. It works because real variables overlap: 24 ability tests do not measure 24 separate things.
Part I: PCA, for numbers
A little over an hour. Compress 24 ability tests into a handful of components, and work out what each component means.
The idea: the school report
Forty pupils each have a maths mark and a physics mark. You may write only one number per pupil on the report. Which number?
Pupils who are good at maths tend to be good at physics, so the cloud is long and thin. The red line runs along the direction in which the pupils differ most. PCA calls it the first principal component, PC1. The dashed line, at a right angle, is PC2: what is left over.
Left: drop each pupil straight onto the red line. Their position along it, their “shadow”, is their PC1 score: one number, roughly “maths plus physics”, an overall mark. Right: the same pupils with the red line laid flat. Nobody has moved relative to anybody else; we have only turned the page.
If you keep PC1 and throw PC2 away, two columns have become one and you have lost 6% of the differences between pupils.
PCA rotates the data so that the first new axis points along the longest direction of the cloud, the second along the longest remaining direction, and so on. The share of the total spread along each axis is its variance explained.
The idea: reading the loadings
How is a component built from the original variables? Each variable gets a weight, called a loading. Take 200 pupils and four subjects:
Blue bars are positive weights, red bars negative. Read a component with three questions:
- Which bars are long? Short bars barely count.
- Same sign, or opposite?
- What do the long ones have in common?
- PC1 (62%): four bars, all positive, all about the same length. A high score means good marks across the board. Call it the overall level.
- PC2 (31%): maths and physics one way, English and history the other. Opposite signs mean a contrast: a high score means stronger at sciences than at humanities. A pupil with top marks everywhere scores about zero.
No. The sign of a whole component is arbitrary: flip every bar of PC2 and you get “humanities against sciences”, which is the same component. Only which variables sit at opposite ends matters.
PCA gives you the weights. The name is yours, and it can be wrong.
Two IQ tests are strongly correlated. What would you expect PCA to do with them? Decide, then answer below.
Two IQ tests are strongly correlated. What does PCA do with them?
It deletes one of them, since it adds nothing
That would be feature selection. PCA never drops a column: it builds new columns out of all the old ones.
It gives both a similar weight on the same component
Yes: maths and physics again. Tests that move together share a direction, so one component can describe both.
It puts each on its own component, to keep them apart
The opposite: variables that move together end up on the same component. Components are for things that don’t move together.
Nothing: PCA only works on uncorrelated variables
Backwards. Correlation is what PCA feeds on. With uncorrelated variables there is nothing to compress.
Meet the data
Run the chunk. It picks the columns and splits the data.
The data are a classic: in 1939, Holzinger and Swineford gave 301 pupils in two Chicago schools a battery of ability tests. The chunk keeps case (an id), school and 24 tests, named t01 to t24:
| Tests | What they ask | Examples |
|---|---|---|
t01–t04 |
Spatial | visual perception, cubes, form board, lozenges |
t05–t09 |
Verbal | general information, sentence completion, word meaning |
t10–t13 |
Speed | addition, code, counting dots |
t14–t19 |
Memory | recognising words, numbers and figures |
t20–t24 |
Reasoning and maths | deduction, number puzzles, series, arithmetic |
That grouping is the test designers’. Keep it in mind: we’ll see whether PCA finds it without being told.
The chunk then splits the pupils as in Labs 3 to 5: 240 to train, 61 to test. There is no outcome today, so why split? Because PCA is often a preparation step for a later model, and then the test set must not help decide the components. In this lab the test set is not used again.
Explore: do the tests overlap?
Run the chunk, then answer the reflection at line 97: what can you say about this heatmap?
Each tile is one pair of tests; red means the two move together. It is the school-report scatter for 24 subjects at once.
- Almost no blue. Nearly every pair of tests is positively correlated (0.26 on average): pupils who do well on one test tend to do well on most others. That suggests an overall level, which should show up as PC1.
- Darker blocks. The verbal tests
t05–t09form the strongest block (0.68 among themselves). The speed testst10–t13form a weaker one. The spatial testst01–t04correlate least with everything else. Blocks suggest that the next components will be contrasts between groups of tests.
Apply PCA
Run the chunk.
The PCA sits inside a recipe, like Lab 5’s normalising step.
recipe(school ~ ., data = train_data) |> update_role(case, new_role = "ID") |> step_zv(...) |> step_normalize(all_numeric_predictors()) |> step_pca(all_numeric_predictors(), num_comp = 5, keep_original_cols = TRUE) |> prep()
- Same units first. PCA looks for spread, so without this the test with the biggest numbers would become PC1 just because of its scale.
- The rotation. Finds the 24 directions, longest first.
- Keep the first five as new columns
PC1toPC5. - Compute it now, on the training set.
Three smaller things: school ~ . does not predict anything here, it only tells the recipe which columns are “predictors”; update_role() keeps the id column case out; step_zv() would drop a column that never varies (there is none). bake() then applies the recipe and adds the scores:
Each pupil now has five scores. Pupil 201 has PC1 = -3.1: their shadow on the longest direction, well below the average of 0.
Your turn: how many components?
Run both chunks, then answer the three reflection questions at lines 155–159.
The first line, pca_recipe$steps[[3]]$res, reaches into the recipe’s third step (the PCA) and takes out the fitted result. tidy(matrix = "eigenvalues") turns it into a table with one row per component:
percent is that component’s share of the total spread (0.307 is 30.7%). cumulative is the running total: what you keep if you stop there. The two plots draw those two columns:
- Why might five be reasonable? PC1 stands out with 31%. PC2 to PC5 add 10%, 8%, 7% and 4%; after that every component adds less than 4%. There is no sharp bend in the bars, so five is a judgement call, not a rule.
- How much do 2, 3 and 5 explain? 41%, 48% and 60%.
- Why is plotting only two a risk? Two fit on a screen, but they show only 41% of the variance. Structure in the other 59% is invisible in that plot.
Compare with the school report, where PC1 held over 90%: two marks were nearly one thing. Twenty-four tests are not one thing.
Your turn: the first two components
Run the scatter plot, then answer the three reflection questions at lines 172–174. For the second one, try cor(pca_data$PC1, pca_data$PC2) in the console.
- Clusters, outliers, a trend? One cloud, with no clear clusters. Pupils differ by degree, not by type.
- The correlation between PC1 and PC2 is exactly 0. That is by construction: PCA turned the page until no tilt was left. The missing trend says nothing about the pupils.
- What might it hide? 59% of the variance. Anything that only shows in PC3 to PC5 cannot be seen here.
Add colour = school inside aes():
Grant-White pupils sit towards the upper right (average 0.6 on PC1, against -0.5 for Pasteur). PCA never saw the school column, so the components have picked up something real about the pupils.
Your turn: what is each component?
Run the chunk, then answer the two reflection questions at lines 213–214. Use the three questions from the school report, and the table of tests above. Try to give each component a name.
One panel per component, one bar per test, the longest bars at the top.
- PC1: overall ability (31%). All 24 loadings are positive and similar in size, from 0.09 to 0.28. A high score means doing well across the board. This is the heatmap’s “almost no blue”; psychologists call it g, general ability.
- PC2: words against memory and speed (10%). At one end the verbal tests: sentence completion (
t07), general information (t05), word meaning (t09). At the other: object–number (t17), addition (t10), number recognition (t15), number–figure (t18). A high score means relatively stronger with words. - PC3: shapes against speed (8%). At one end number recognition (
t15), lozenges (t04), cubes (t02); at the other addition (t10), counting dots (t12), code (t11). This one is harder to name, and it explains little.
So 24 tests come down to roughly three ideas: a level and two contrasts. Both predictions from the heatmap held.
A pupil has a PC1 score of +4 and a PC2 score of 0. What do you know about them?
They do well overall, and are about equally strong at verbal tests and at memory and speed tests
Yes. PC1 is the level, PC2 the contrast. A contrast of 0 means neither side stands out.
They do well overall, but are poor at verbal tests
A PC2 of 0 is the average, not a low score: neither end of the contrast stands out.
They are average at everything
PC1 is the overall level, and +4 is far above the average of 0.
Part II: MCA, for categories
About 45 minutes. Map the answers to 19 moral questions from the World Values Survey.
The idea: why not PCA?
Think about both questions before reading on: why can’t you run PCA on categorical variables, and what might “distance” mean instead?
PCA needs arithmetic: means, variances, a longest direction. A test score of 60 is 10 above 50. But what is “TV minus newspaper”? What is the mean of “yes” and “no”? Categories have no spacing, so there is no cloud to rotate.
What you can still do with categories is count: how often are two answers given by the same people? MCA uses only that.
In MCA, two answers are close when mostly the same people give them. Close means “go together”, not “similar in value”. (The notebook’s name for this is chi-square distance.)
The idea: what do you watch?
Twelve people answer four yes/no questions about what they watch:
| Football | F1 | Bake Off | Strictly | |
| A | y | y | n | n |
| B | y | y | n | n |
| C | y | y | n | y |
| D | y | n | n | n |
| E | n | n | y | y |
| F | n | n | y | y |
| G | n | y | y | y |
| H | n | n | y | n |
| I | y | y | y | y |
| J | n | n | n | n |
| K | y | y | n | n |
| L | n | n | y | y |
Which answers tend to come from the same people? A, B and K watch both sports and neither show. E, F and L are the reverse. A few mix. MCA turns that table into a map with one point per answer: four questions with two answers each, so eight points.
Three rules for reading such a map:
- Close together: the same people.
football_ysits byf1_y;bakeoff_ybystrictly_y. - Each
_yfaces its own_n. With two answers per question, they land on opposite sides of the middle. Read one side of an axis and you know the other. - Name an axis by what its ends have in common, as with loadings. Dim 1 here: sport on one side, shows on the other.
The percentages on the axes are the same idea as variance explained: how much of the structure that dimension holds.
Load the data
Run the chunk.
The World Values Survey asked 94,278 people around the world about 19 acts: is this ever justifiable? Each answer runs from 1 (never) to 10 (always).
read_csv(...) |> mutate( across(everything(), ~ as.factor(if_else( .x > median(.x), "y", "n")))) ; colnames(norms) <- values
- Every column gets the same treatment.
- Above that question’s median: “y”. Otherwise “n”. A ten-point scale becomes two categories.
- Readable names replace
Q177toQ195.
For 12 of the 19 questions, the median answer is 1, “never justifiable”. For those questions, “y” means any answer other than “never”. So terrorism_y is “did not rule it out completely”, not “supports terrorism”. For a question like divorce, where the median is 5, “y” means a fairly permissive answer.
Apply MCA
Run the chunk (one line), then answer the reflection: why mark a variable as supplementary?
mca <- MCA(norms, quali.sup = 1, graph = FALSE)
- From the
FactoMineRpackage, not tidymodels: no recipe this time. - The table of factors: 19 columns of y and n.
- Column 1 is supplementary:
cheating_benefitsis left out while the axes are built. - Don’t draw the default plot. We draw our own next.
A supplementary variable is placed on the map after the axes have been built from the other 18 questions. It cannot shape the map. That makes it a test of the map: if the map means something, a question it has never seen should land where we expect. It is the closest thing today to a test set.
Prediction: cheating_benefits_y should land near the other dishonesty answers, such as fare dodging and cheating on taxes.
Your turn: read the biplot
Run the chunk, then answer the two reflection questions at lines 277–278. In RStudio, click Zoom above the plot to see it large. Use the three rules from the TV map.
One triangle per answer: 18 questions with two answers each in red, and the supplementary question in green.
- Dim 1 (37%): how much is ever justifiable. Every
_yis on the right and every_non the left. Someone on the right says “not never” to many things. Like PC1, it is a level. - Dim 2 (16%): which kind. Look at the right-hand side. At the top: violence against a spouse, terrorism, stealing, bribes. At the bottom: divorce, sex before marriage, homosexuality, abortion. Like PC2, it is a contrast: harm to others against private life.
- The supplementary question
cheating_benefits_ylands on the right and above the middle, next to fare dodging, as predicted, although it played no part in building the axes.
They are rare answers: about a quarter of respondents give them. On an MCA map, rare answers sit far from the middle, and common ones close to it.
Your turn: who builds Dim 1?
Run the chunk, then answer the reflection at line 291.
fviz_contrib() shows how much each answer contributes to a dimension: choice = "var" means the answers (not the people), axes = 1 means Dim 1, top = 10 the ten largest. The red dashed line is what each answer would contribute if all contributed equally (2.8%).
The top eight are all “y” answers: stealing, political violence, social violence, terrorism, bribes, suicide, cheating on taxes, violence against a spouse. What they share is harm and dishonesty, and rarity: few people give these answers, so they separate respondents the most. This matches the biplot, where these answers sit farthest to the right.
Contributions are MCA’s version of loadings: both answer “what is this axis made of?”.
On an MCA map, abortion_y and divorce_y sit close together. What does that tell you?
People rate abortion and divorce as about equally justifiable
The map knows nothing about how high the ratings were, only who gave which answer.
Mostly the same people gave both answers
Yes. Close means “chosen by the same people”.
Both answers are rare
Rare answers sit far from the middle. Closeness to each other is about being given by the same people.
Wrap-up
PCA or MCA?
| PCA | MCA | |
|---|---|---|
| Data | Numbers with meaningful spacing: test scores, incomes | Categories: yes/no, news source, age group |
| Built from | Correlations between variables | Counts of answers given together |
| Close means | Similar values | Chosen by the same people |
| What you read | Loadings: one bar per variable | The map and the contributions: one point per answer |
| Today | step_pca() in a recipe: 24 tests → 5 components |
MCA(): 19 questions → 2 dimensions |
Both labs ended with the same shape of answer: a first axis that is a level (overall ability; how much is justifiable) and a second that is a contrast (words against memory and speed; harm against private life). That is common when many variables measure related things.
What you learned
Why reduce dimensions With many variables, data are sparse, distances stop telling rows apart and models overfit. Real variables overlap, so a few new axes keep most of the structure.
PCA The school report: turn the cloud so the first axis runs along its longest direction. Each component has a share of variance (how much it keeps) and loadings (what it is made of).
Reading a component Which bars are long? Same sign (a level) or opposite (a contrast)? What do the long ones share? The name is yours.
MCA The TV map: one point per answer, close when the same people give them, each y opposite its n. Read the ends of each axis and the contributions.
One question to take away: two components show 41% of the test data. Before you trust a pattern in the PC1–PC2 scatter, what would you check?
Further reading
- An Introduction to Statistical Learning, free online: sections 12.1–12.2 (unsupervised learning, principal components).
- Tidy Modeling with R, free online: chapter 16 (dimensionality reduction).
- STHDA’s MCA guide: every
factoextraplot for MCA, with code.
Glossary
Underlined words on this page are explained in this list. Click or tap one anywhere on the page to see what it means and where it was first explained.
| Word | Meaning |
|---|---|
| biplot | The MCA map: one point per answer, placed so that answers given by the same people are close |
| contrast | A component with long loadings of opposite signs: a high score means more of one group of variables than of the other |
| contribution | In MCA, how much one answer accounts for a dimension. MCA’s version of a loading |
| dimension | A variable you could draw as an axis. In MCA, also the name for a new axis (Dim 1, Dim 2) |
| dimensionality reduction | Replacing many variables with a few new ones that keep most of the structure |
| loading | The weight of an original variable in a principal component |
| MCA | Multiple correspondence analysis: dimensionality reduction for categorical variables |
| normalise | Rescale a variable to mean 0 and standard deviation 1, so units don’t decide the result |
| PCA | Principal component analysis: dimensionality reduction for numeric variables |
| principal component | A new axis found by PCA: a weighted combination of the original variables. PC1 runs along the longest direction of the data |
| score | A row’s position on a component: its “shadow” on that axis |
| supplementary variable | A variable placed on the MCA map after the axes are built, so it cannot shape them |
| unsupervised | Learning without an outcome to predict: looking for structure in the variables themselves |
| variance explained | The share of the total spread in the data that lies along one component |