| subject | chocolate | happiness |
|---|---|---|
| 1 | 1 | 1 |
| 2 | 2 | 1 |
| 3 | 3 | 2 |
| 4 | 3 | 2 |
| 5 | 5 | 5 |
| 6 | 4 | 5 |
| 7 | 7 | 6 |
| 8 | 8 | 4 |
| 9 | 5 | 6 |
| 10 | 9 | 8 |
2 Correlation
Correlation does not equal causation.
Every Statistics Instructor Ever
Chapter 1 worked with one variable at a time: its center, fully formalized (mean, median, mode), and a first, blunt pass at its spread, the range. Most interesting questions aren’t about a single variable in isolation, though, they’re about whether two things move together. Does ozone rise on hotter days? Do wetter winters produce taller trees?
This chapter gets there in stages: first, what two related variables actually look like side by side, in a scatter plot. Partway through, we’ll pick up where the range left off: a proper measure of differentness, the one Chapter 1 promised and this chapter builds. That same measure, extended to two variables, gives us covariance, then Pearson’s \(r\), a first look at regression, and finally the specific ways a correlation can mislead you.
2.1 What data look like when one thing changes another
To use data for explanation, we need tools for recognizing when two things move together. Sometimes that reflects a real cause-and-effect relationship, and sometimes it doesn’t. The challenge is learning to tell the difference. In this chapter we’ll build intuition for what data look like when two variables are unrelated, when they move together, and when the pattern might mislead us.
2.1.1 More Chocolate = More Happiness?
Suppose a person’s chocolate supply has a causal influence on their happiness: more chocolate means more happiness, less chocolate means less. Because happiness is also caused by plenty of other things in a person’s life, we’d expect the relationship between chocolate and happiness to be real but imperfect, not a perfect one-to-one match.
Our first step is to collect some imaginary data from 100 people. We walk around and ask the first 100 people we meet to answer two questions:
- how much chocolate do you have? and
- how happy are you?
For convenience, both the scales will go from 0 to 100. For the chocolate scale, 0 means no chocolate, 100 means lifetime supply of chocolate. Any other number is somewhere in between. For the happiness scale, 0 means no happiness, 100 means all of the happiness, and in between means some amount in between.
Here is some sample data from the first 10 imaginary subjects.
We asked each subject two questions so there are two scores for each subject, one for their chocolate supply, and one for their level of happiness. You might already notice some relationships between amount of chocolate and level of happiness in the table. To make those relationships even more clear, let’s plot all of the data in a graph.
2.1.2 Scatter plots
When you have two measurements worth of data, you can always turn them into dots and plot them in a scatter plot. A scatter plot has a horizontal x-axis, and a vertical y-axis. You get to choose which measurement goes on which axis. Let’s put chocolate supply on the x-axis, and happiness level on the y-axis. Figure 2.1 shows 100 dots. Each dot is for one subject and represents both their happiness level and their chocolate supply.
Each dot has two coordinates, an x-coordinate for chocolate, and a y-coordinate for happiness. Looking at the scatter plot, we can see that the dots show a pattern. People with less chocolate tend to report lower happiness, while those with more chocolate tend to report higher happiness. It looks like the more chocolate you have the happier you will be, and vice-versa. This kind of relationship is called a positive correlation.
2.1.3 Positive, Negative, and No-Correlation
Two other patterns are worth seeing: a reversed relationship, where more chocolate means less happiness, and no relationship at all, where chocolate supply has nothing to do with happiness. Figure 2.2 shows all three side by side:
The first panel shows a negative correlation: happiness goes down as chocolate supply increases, more of X means less of Y. The second panel shows a positive correlation: happiness goes up as chocolate supply increases, more of X means more of Y. The third panel shows no correlation: there’s no obvious relationship between chocolate supply and happiness, changes in X don’t relate to changes in Y.
TipTip: Always plot first
If the scatter shows a U-shape or other curve, a single straight-line correlation will mislead. Fit a curved model (e.g., add a quadratic term) or transform variables instead of reporting r alone.
We’ve already covered central tendency: mean, median, mode. Variation comes next.
2.2 How much does one variable vary?
Before we can measure how two variables move together, we need the proper measure of differentness Chapter 1 promised: a number for how much a single variable moves at all. That turns out to be the same machinery needed for two variables, used twice, so we build it once here and then extend it in the section that follows.
2.2.1 Difference Scores
It would be nice to summarize the amount of differentness in the data. One idea is to compare every number to every other number: pick a first number, pick a second number, and subtract them. The result tells us exactly how different the two are: a large gap means the two numbers are far apart, a gap near zero means they’re close together. Do that across a whole dataset and you have a complete picture of how different everyone is from everyone else.
The count grows fast, though: 10 numbers means 10 × 10 = 100 comparisons, 500 numbers means 500 × 500 = 250,000. And even a full set of pairwise comparisons wouldn’t solve a problem you’re about to run into anyway: the positive and negative differences cancel out, so their average comes out to zero no matter what the data look like. These problems motivate a much simpler approach: compare each score to one reference point instead, the mean.
2.2.2 The Variance
We have used the words variability, variation, and variance a lot, and they all point at the same idea: numbers differ. The word variance does double duty, though. Loosely it means that general idea. Precisely it names a specific statistic, the mean of the squared deviations from the mean.
To calculate it, subtract the mean from each score to get its difference score (its deviation from the mean), square those differences, then average them. Squaring is the trick that stops positive and negative deviations from cancelling. In short:
\(variance = \frac{\text{Sum of squared difference scores}}{\text{Number of Scores}}\)
A little further on we’ll distinguish between dividing by \(N\) (when you have the entire population) and \(n-1\) (when you only have a sample, which is almost always the case). For now, just use \(N\).
Deviations from the mean.
Earlier we compared every number to every other number, which got unmanageable fast. The simpler approach is to compare each score to the mean instead: find the mean, then subtract it from every score. What comes back tells us both how well the mean represents the data and how much spread there is around it.
| scores | values | mean | Difference_from_Mean |
|---|---|---|---|
| 1 | 1 | 4.5 | -3.5 |
| 2 | 6 | 4.5 | 1.5 |
| 3 | 4 | 4.5 | -0.5 |
| 4 | 2 | 4.5 | -2.5 |
| 5 | 6 | 4.5 | 1.5 |
| 6 | 8 | 4.5 | 3.5 |
| Sums | 27 | 27 | 0 |
| Means | 4.5 | 4.5 | 0 |
The mean here is \(\frac{1+6+4+2+6+8}{6} = \frac{27}{6} = 4.5\). The third column just repeats it on every row, which looks redundant but makes the earlier point concrete: six copies of 4.5 add back to the original total of 27. The fourth column is the deviation from the mean, \(X_{i}-\bar{x}\), so the score of 1 sits 3.5 below and the score of 6 sits 1.5 above.
And now the problem. Those deviations sum to zero. The positives and negatives cancel exactly, which would suggest there is no variation at all, and that is obviously false. Squaring will fix it, but first it is worth seeing why the cancellation happens.
Mean as the Balancing Point.
The mean is the balancing point of the data. Imagine laying a ruler across your finger. The place where it balances is the point where the weight on each side is equal. Data works the same way: if we treat each value like a weight, the mean is where the total “mass” to the left and right cancel out.
This balancing property explains why the deviations always sum to zero. The values below the mean contribute negative deviations, the values above contribute positive ones, and together they cancel:
\(-x + x = 0\)
To see this more concretely, consider the numbers
\(X = (1,2,6,7,9)\)
Now imagine the number line as a teeter-totter. Only when the fulcrum is placed at 5 does the board balance:
Formally, this means the signed distances from the mean always cancel out:
\((1−5)+(2−5)+(6−5)+(7−5)+(9−5)=0\)
This makes the mean unique. It is the only value for which the sum of deviations equals zero:
\(\sum_{i=1}^{n}(x_{i}-a) = 0 \quad \Rightarrow \quad a = \bar{x}\)
The mean is not just a “typical” value, it is the one point where the data balance. And that balancing property is exactly why the raw deviations always sum to zero. To summarize variation, we need a way around this problem.
The squared deviations.
The standard trick is to square the deviations. Since \(2^2 = 4\) and \((-2)^2 = 4\), squaring turns every negative into a positive, so the deviations can no longer cancel. We call these squared deviations. Here is the same table with a column for them:
| scores | values | mean | Difference_from_Mean | Squared_Deviations |
|---|---|---|---|---|
| 1 | 1 | 4.5 | -3.5 | 12.25 |
| 2 | 6 | 4.5 | 1.5 | 2.25 |
| 3 | 4 | 4.5 | -0.5 | 0.25 |
| 4 | 2 | 4.5 | -2.5 | 6.25 |
| 5 | 6 | 4.5 | 1.5 | 2.25 |
| 6 | 8 | 4.5 | 3.5 | 12.25 |
| Sums | 27 | 27 | 0 | 35.5 |
| Means | 4.5 | 4.5 | 0 | 5.92 |
The first score of 1 sits 3.5 below the mean, so its deviation is \(-3.5\) and its squared deviation is \((-3.5)^2 = 12.25\). Now that nothing is negative we can add them up, and that total is the sum of squares (SS): take every score’s deviation from the mean, square it, and add up all those squared deviations.
\[SS = \sum_{i=1}^{N}(x_{i}-\bar{x})^2\]
You will meet this quantity again in the ANOVA chapter, but there is nothing more to it than the sum of those squared deviations.
Finally, the variance.
We have already computed the variance, one step at a time, without naming it. The goal was a single number summarizing spread, but deviations give us as many numbers as we have data points, so we want their average. Sum the squared deviations, divide by how many there are, and that is the variance:
\[variance = \frac{SS}{N}\]
For our data it comes to 5.917. On its own that number is hard to interpret, because squaring the deviations put it on a squared scale, so it is no longer in the same units as the data. The fix is to take the square root, \(\sqrt{5.917} \approx 2.43\).
2.2.3 The Standard Deviation
That square root is the standard deviation (SD), so we had already computed it a step earlier without naming it:
\[\text{standard deviation} = \sqrt{Variance} = \sqrt{\frac{SS}{N}} = \sqrt{\frac{\sum_{i}^{N}(x_{i}-\bar{x})^2}{N}}\]
The square root “unsquares” the variance, putting the measure of spread back on the original scale of the data. Here is the table one more time:
| scores | values | mean | Difference_from_Mean | Squared_Deviations |
|---|---|---|---|---|
| 1 | 1 | 4.5 | -3.5 | 12.25 |
| 2 | 6 | 4.5 | 1.5 | 2.25 |
| 3 | 4 | 4.5 | -0.5 | 0.25 |
| 4 | 2 | 4.5 | -2.5 | 6.25 |
| 5 | 6 | 4.5 | 1.5 | 2.25 |
| 6 | 8 | 4.5 | 3.5 | 12.25 |
| Sums | 27 | 27 | 0 | 35.5 |
| Means | 4.5 | 4.5 | 0 | 5.92 |
Our standard deviation, \(\sqrt{5.917} \approx 2.43\), makes far more sense than the variance did. A variance of 5.917 feels too large for this data because it is on the squared scale, but an SD of 2.43 lines up with the actual deviations, since most scores fall within about 2.4 of the mean of 4.5.
That is what makes the SD so useful. Told only that a dataset has a mean of 4.5 and an SD of 2.4, you already have a picture of it: values cluster around 4.5, not exactly at it, with a typical spread of two to three units either way. It gives you the typical deviation from the mean while staying in the original units.
2.2.4 Mean Absolute Deviation
So far we’ve handled the “differences cancel out” problem by squaring deviations. But there’s another simple option: take the absolute value of each deviation from the mean. That way, every difference becomes positive, and when we add them up we don’t end up at zero.
Here’s what that looks like for our data:
| scores | values | mean | Difference_from_Mean | Absolute_Deviations |
|---|---|---|---|---|
| 1 | 1 | 4.5 | -3.5 | 3.5 |
| 2 | 6 | 4.5 | 1.5 | 1.5 |
| 3 | 4 | 4.5 | -0.5 | 0.5 |
| 4 | 2 | 4.5 | -2.5 | 2.5 |
| 5 | 6 | 4.5 | 1.5 | 1.5 |
| 6 | 8 | 4.5 | 3.5 | 3.5 |
| Sums | 27 | 27 | 0 | 13 |
| Means | 4.5 | 4.5 | 0 | 2.17 |
The last column shows absolute deviations. If we take their mean, we get the mean absolute deviation (MAD). Notice this acts a lot like the standard deviation: it’s the average size of a deviation from the mean, only without squaring. Both approaches, squaring and taking absolute values, solve the same problem in slightly different ways. Squaring is more common because it leads to nice mathematical properties (like in regression and ANOVA), but the MAD can be more intuitive to interpret.
2.3 Covariance
We now have a number for how one variable varies. Covariance is the same idea for two: it tells us whether being above the mean on \(X\) tends to come with being above the mean on \(Y\) (positive), below the mean on \(Y\) (negative), or neither (near zero).
In our chocolate–happiness example, the scatter plot suggested that higher chocolate tends to come with higher happiness. That means the deviations from each mean often have the same sign, so their products are mostly positive: evidence of positive covariance. Put simply, the two variables vary together.
Covariance is important because it underlies correlation, regression, and many other statistical tools. It can feel abstract at first, so keep the core idea in mind: look at how each variable’s deviations from its mean behave together.
NoteNote: Covariance intuition
Think of a three-legged race. When partners move forward together, their steps align (positive covariance). When one moves forward as the other moves back, their steps oppose (negative covariance). If their steps are uncoordinated, there’s no consistent pattern (covariance near zero).
2.3.1 Turning the numbers into a measure of Covariance
To turn “chocolate and happiness vary together” into an actual number, we need a way to measure that relationship directly. Consider our chocolate and happiness scores.
| subject | chocolate | happiness | Chocolate_X_Happiness |
|---|---|---|---|
| 1 | 1 | 1 | 1 |
| 2 | 2 | 1 | 2 |
| 3 | 3 | 2 | 6 |
| 4 | 3 | 2 | 6 |
| 5 | 5 | 5 | 25 |
| 6 | 4 | 5 | 20 |
| 7 | 7 | 6 | 42 |
| 8 | 8 | 4 | 32 |
| 9 | 5 | 6 | 30 |
| 10 | 9 | 8 | 72 |
| Sums | 47 | 40 | 236 |
| Means | 4.7 | 4 | 23.6 |
We’ve added a new column called Chocolate_X_Happiness, which is simply each person’s chocolate score multiplied by their happiness score. Why multiply? Because the products reveal how the two values move together.
- Small × small → small product
- Large × large → large product
- Small × large → product in between
Now extend this idea from pairs to the whole dataset. If chocolate and happiness rise together (both above average at the same time), the summed products will be large. If one tends to rise while the other falls, the summed products will shrink. This is the basic logic behind covariance.
To see just how large or small those summed products can get, set chocolate and happiness aside for a moment and look at two idealized cases, using plain numbers instead of a real scenario. The first table pairs two variables, \(X\) and \(Y\), so every value lines up with its match: 1 with 1, 2 with 2, and so on. The second table uses a different pair, \(A\) and \(B\), holding the same ten numbers but paired in the exact opposite order: 1 with 10, 2 with 9, and so on, the mirror image of the first table.
| X | Y | XY |
|---|---|---|
| 1 | 1 | 1 |
| 2 | 2 | 4 |
| 3 | 3 | 9 |
| 4 | 4 | 16 |
| 5 | 5 | 25 |
| 6 | 6 | 36 |
| 7 | 7 | 49 |
| 8 | 8 | 64 |
| 9 | 9 | 81 |
| 10 | 10 | 100 |
| 55 | 55 | 385 |
| A | B | AB |
|---|---|---|
| 1 | 10 | 10 |
| 2 | 9 | 18 |
| 3 | 8 | 24 |
| 4 | 7 | 28 |
| 5 | 6 | 30 |
| 6 | 5 | 30 |
| 7 | 4 | 28 |
| 8 | 3 | 24 |
| 9 | 2 | 18 |
| 10 | 1 | 10 |
| 55 | 55 | 220 |
\(X\) and \(Y\) are perfectly aligned: \(1\) with \(1\), \(2\) with \(2\), up to \(10\) with \(10\). Their products add up to 385, the largest possible sum.
\(A\) and \(B\) are perfectly reversed: \(1\) with \(10\), \(2\) with \(9\), and so on. Their products add up to 220, the smallest possible sum.
Any other pairing of 1–10 with 1–10 will fall between these two extremes. Figure 2.5 shows the sums of products from 1000 random pairings. Every result falls between 220 (perfect negative alignment) and 385 (perfect positive alignment).
The sum of the products tells us about the direction of the relationship:
- Close to 385 → strong positive relationship.
- Close to 220 → strong negative relationship.
- Near the middle → little or no relationship.
This is the foundation of covariance: products of paired values reveal whether two variables rise together, fall together, or move in opposite directions.
2.3.2 Covariance, the measure
Earlier we saw that multiplying values from two variables can highlight how they vary together: large-with-large and small-with-small pairs push the product up, while mixed pairs keep it lower. That intuition is useful, but in statistics we define covariance more precisely.
Instead of multiplying raw values of \(X\) and \(Y\), we multiply their deviations from the mean. This centers each variable, so the product reflects how the two variables co-vary around their averages:
\(cov(X,Y) = \frac{\sum_{i}^{N}(x_{i}-\bar{x})(y_{i}-\bar{y})}{N}\)
In practice, the steps are straightforward: subtract the mean of \(X\) from each \(x_i\), subtract the mean of \(Y\) from each \(y_i\), multiply the deviations, and then average those products. A positive covariance means the two variables tend to move together (above or below their means at the same time). A negative covariance means they move in opposite directions. A covariance near zero suggests no consistent relationship.
The following table shows this process for our chocolate and happiness data. Cd and Hd are each person’s deviation from the chocolate mean and the happiness mean, and Cd_x_Hd is their product:
| subject | chocolate | happiness | Cd | Hd | Cd_x_Hd |
|---|---|---|---|---|---|
| 1 | 1 | 1 | -3.7 | -3 | 11.1 |
| 2 | 2 | 1 | -2.7 | -3 | 8.1 |
| 3 | 3 | 2 | -1.7 | -2 | 3.4 |
| 4 | 3 | 2 | -1.7 | -2 | 3.4 |
| 5 | 5 | 5 | 0.3 | 1 | 0.3 |
| 6 | 4 | 5 | -0.7 | 1 | -0.7 |
| 7 | 7 | 6 | 2.3 | 2 | 4.6 |
| 8 | 8 | 4 | 3.3 | 0 | 0 |
| 9 | 5 | 6 | 0.3 | 2 | 0.6 |
| 10 | 9 | 8 | 4.3 | 4 | 17.2 |
| Sums | 47 | 40 | 0 | 0 | 48 |
| Means | 4.7 | 4 | 0 | 0 | 4.8 |
The mean of those products (in the bottom right corner of the table) is the official covariance.
One note before moving on. The formula above divides by \(N\), which is the population version, and that is what the table computes. R’s cov() and sd() divide by \(n-1\) instead, for the same reason the sample standard deviation did earlier in this chapter. The two differ only by a factor of \(n/(n-1)\), and that factor cancels completely in Pearson’s \(r\) below, so \(r\) comes out identical either way. If you compute the covariance itself in R, though, expect a slightly larger number than the one in this table.
Covariance is a powerful idea, but it has a major drawback: its magnitude is hard to interpret. Just as variance is tied to squared units, covariance is tied to the scales of both variables, so a covariance of 3.22 isn’t inherently “large” or “small” the way a correlation is. All we can read directly from it is the sign, not the strength. To compare strength across variables with different units or scales, we need to normalize covariance, which leads to Pearson’s \(r\).
2.3.3 Pearson’s r
We often want a relationship measure that is directional, unitless, and bounded. Pearson’s \(r\) does exactly this by rescaling covariance to lie between \(-1\) and \(1\):
- \(r=1\): perfect positive linear relationship
- \(r=0\): no linear relationship
- \(r=-1\): perfect negative linear relationship
Values closer to \(\pm1\) indicate stronger linear association; values near \(0\) indicate weak or no linear association.
Let’s take a look at a formula for Pearson’s \(r\):
\(r = \frac{cov(X,Y)}{SD_{X}SD_{Y}}\)
\(r\) is the covariance of X and Y divided by the product of their standard deviations. Dividing by the standard deviations normalizes the covariance, constraining \(r\) to fall between -1 and 1, which makes correlation values comparable across variables measured in different units or on different scales.
NoteNote: Computing r
You’ll see several algebraically equivalent formulas for Pearson’s r, some easier for hand calculation, others more compact. They all produce the same result. In practice, you’ll compute \(r\) with software, so the priority here is understanding what \(r\) means and how to interpret it. Working through one example by hand still helps: it shows where \(r\) actually comes from, which is why lecture walks through a full hand calculation before you run it in R during lab.
To confirm \(r\) really does stay between -1 and 1, Figure 2.6 shows a simulation: we randomly ordered the numbers 1 to 10 for both an X measure and a Y measure, computed Pearson’s \(r\) between them, and repeated the process 1000 times. Every dot falls between -1 and 1.
2.4 Regression: A mini intro
This section adds one more piece to our understanding of correlation: linear regression, at an introductory level.
Figure 2.7 shows the same three scatter plots as before, now with a fitted line added to each.
2.4.1 The best fit line
Notice the red lines in these panels. Each is a best fit line: the single straight line that best summarizes the overall pattern of dots.
With one variable, we use the mean to describe central tendency in one dimension. With two variables plotted together, we need a two-dimensional equivalent. A line serves that purpose, capturing the average relationship across the cloud of points.
Many lines could be drawn through a scatter of points, but most would miss the pattern. The best fit line is the one with the least error, defined precisely below.
Figure 2.8 shows what that error looks like: a line runs through the dots, and short vertical lines drop from each dot to the line. These are residuals, the vertical differences between the observed data and the fitted line, showing how far off the line’s prediction is for each point. Since not all dots fall exactly on the line, some error is inevitable; the best-fit line is the one that minimizes it overall.
Figure 2.8 shows a scatter plot of two variables \(X\) and \(Y\). Each black dot represents observed data; the blue line is the regression line, and the white dots show its predicted values. The red lines are residuals: they drop vertically from each observed value to its predicted value on the line, showing how far the prediction differs from what was actually observed.
The regression line is chosen to minimize the total size of these residuals. In practice, we compute the sum of squared deviations: square each residual, then add them together. The blue line produces the smallest possible total squared error; any other line would yield a larger one.
Figure 2.9 animates this directly, comparing the best fit line (blue) to other possible lines (black) as the black line moves up and down. The red lines show the error between the black line and the data points at each position, and the shaded grey area shrinks to its minimum exactly when the black line reaches the best fit line, then expands again as it moves away.
Whenever the black line does not overlap with the blue line, it fits the data less well: the blue regression line is the one that minimizes the overall error, not too high, not too low, but the line with the least total deviation.
Figure 2.10 makes that error concrete: for each position of the black line, we square the length of every red line from the animation above and add them up, giving the total sum of squared deviations at that position. The dots start high on the left, swoop down to a minimum in the middle, and rise again on the right. That minimum point is the best fit regression line.
This graph shows how the total error behaves as the line’s y-intercept moves up and down: the error rises whether we shift the line down (0 to -5) or up (0 to +5), and is smallest right in the middle, at the original best fit line.
2.4.2 Computing the best fit line
How do we find the line that minimizes the sum of squared deviations? In principle, you could try many possible lines and compare their errors, but that’s not efficient. In practice, we let software identify the line with the smallest total error, the same lm() fit behind every regression line drawn in this chapter’s figures.
Figure 2.11 checks that against a small example: seven \(x,y\) pairs, plotted with their fitted line.
We’ll return to these same formulas in Week 11, computing a line’s slope and intercept by hand and using those values to compute and interpret residuals for individual points.
2.5 Interpreting Correlations: why caution is warranted
You’ve heard the warning that correlation does not equal causation. Interpreting correlations requires care, because the same number can arise from very different situations: random chance, confounding influences, or nonlinear patterns. In this section, we’ll look at why correlations can be unreliable and how to recognize their limits.
2.5.1 Spurious correlation: chance alone can look convincing
Correlations can be produced by random chance. You can find a positive or negative correlation between two measures that have nothing to do with one another, and you will, often enough that it matters. Unrelated measures produce these spurious correlations by chance alone.
Let’s demonstrate how correlations can appear by chance when there is no causal connection between two measures. Imagine two independent participants, one at the North Pole and one at the South Pole. Each has a lottery machine containing balls numbered 1 through 10. With replacement, each participant randomly draws 10 balls and records the numbers. Because the draws occur independently and half a world apart, there is no possible causal link between the two sets of numbers. The outcomes are determined by chance alone.
Here is what the numbers on each ball could look like for each participant:
| Ball | North_pole | South_pole |
|---|---|---|
| 1 | 2 | 8 |
| 2 | 8 | 4 |
| 3 | 4 | 2 |
| 4 | 3 | 3 |
| 5 | 8 | 4 |
| 6 | 10 | 8 |
| 7 | 10 | 1 |
| 8 | 8 | 7 |
| 9 | 2 | 7 |
| 10 | 5 | 9 |
In this one case, if we computed Pearson’s \(r\), we would find that \(r =\) -0.2. But, we already know that this value does not tell us anything about the relationship between the balls chosen in the north and south pole. We know that relationship should be completely random, because that is how we set up the game.
The better question is what random chance alone can do: if we ran this game thousands of times, each time drawing new balls and computing the correlation, the \(r\) value would fluctuate across the whole range. Looking at the shape of that fluctuation tells us what kinds of correlations chance can produce.
Monte-carlo: what chance produces.
We can use a computer to repeat the lottery game as many times as we like. This process, repeated random sampling to explore possible outcomes, is called a Monte Carlo simulation.
Running the game 1,000 times, each time generating 10 random numbers for both the North Pole and South Pole participants and computing the correlation between them, gives 1,000 different values of Pearson’s \(r\), each produced under conditions where we know no real relationship exists.
Figure 2.12 shows the outcome. Each dot is one simulated correlation. All of them fall between –1 and 1, but the values scatter widely: sometimes strongly positive, sometimes strongly negative, often near zero. The point is that chance alone can generate correlations of any size. Finding a correlation in your data does not by itself guarantee a meaningful relationship.
To visualize this process more concretely, we can animate a single run. In Figure 2.13, each frame shows two sets of 10 random values plotted against each other, along with the fitted regression line. Because the data are random, we expect no consistent pattern. Yet across repetitions, the line slopes upward, then downward, then occasionally flat. This is what spurious correlation looks like, and how quickly random samples can mimic a real pattern.
This leaves us with a real problem: how do we know whether the correlation we observed is meaningful or just chance? One way forward is to look at the whole distribution of outcomes chance produces, rather than at a single outcome. By simulating 1,000 random correlations and plotting them in a histogram, we can see which values occur frequently and which are rare.
Figure 2.14 shows that most chance correlations cluster near zero. Moderate correlations (around \(\pm 0.5\)) occur less frequently, and near-perfect correlations (\(\pm 0.95\)) are almost absent. This distribution defines a “window of chance.” When we compute a correlation in real data, we can ask: is our observed \(r\) consistent with what chance alone could plausibly generate?
For example, if you observed \(r=0.1\), the histogram shows that such values are common in random data, so chance could easily explain it. An observed \(r=0.5\) is less common but still possible by chance. An observed \(r=0.95\), however, falls outside what chance typically produces, suggesting that something more systematic is driving the relationship.
2.5.2 Sample size and stability
So far, our lottery game has used 10 draws per participant. With such small samples, chance produces a wide range of correlations: sometimes strongly positive, sometimes strongly negative. But what happens if we increase the sample size?
We can repeat the simulation under four conditions: each participant draws 10, 50, 100, or 1000 numbers. In each case, we run 1000 simulations and record the correlation values.
Figure 2.15 shows the resulting four histograms of Pearson \(r\) values.
The pattern is clear:
- With n=10, correlations spread widely across the range –1 to 1. Chance alone frequently produces values that would look “strong” in real data.
- With n=50 or 100, the spread narrows, but moderate correlations still appear.
- With n=1000, almost all simulated correlations fall close to zero.
The takeaway is that larger samples reduce the influence of chance. Small samples are highly unstable, making it difficult to distinguish signal from noise. As sample size increases, the distribution of chance correlations collapses toward zero, so observing a large \(r\) becomes far less likely under randomness alone.
This illustrates why researchers place so much emphasis on collecting sufficient data. With too few observations, even strong-looking correlations may be spurious. With larger samples, a correlation of the same size is much harder to attribute to chance, and therefore more likely to reflect a real underlying relationship.
TipTip: Sample size
Bigger n narrows the “window of chance.” If you expect a modest linear effect (e.g., |r| ≈ 0.2–0.3), plan for larger samples so random swings are less likely to mimic or mask the effect.
Visualizing the effect of sample size.
Animations make it easier to see how sample size influences the stability of correlation estimates. When the sample is very small, chance variation can create apparent patterns in any direction. As the sample size increases, those chance fluctuations diminish.
When there is no true correlation
In Figure 2.16, each panel shows two random variables sampled with no real relationship. The panels differ only in sample size: 10, 50, 100, or 1000 observations. Each frame of the animation re-samples the data, plots them, and fits a line.
With \(n=10\), the line swings dramatically from positive to negative slopes. At \(n=50\) and \(n=100\), the line still shifts but is more restrained. At \(n=1000\), the line is consistently flat, as expected when no correlation exists.
The line from the sample with 1000 observations is the most reliable. It stays nearly flat across repetitions, as we would expect when no true correlation exists. The line from a sample of only 10 observations swings wildly. Claims about correlation are only as trustworthy as the sample size behind them.
When there is a true correlation
In Figure 2.17, the simulation is repeated with a real positive correlation built into the data. Again, we vary the sample size from 10 to 1000.
With \(n=10\), the regression line jumps around and occasionally even points downward, despite the underlying positive relationship. As the sample size grows, the line becomes more stable and consistently reflects the true positive slope. Sampling error still introduces some noise, but with larger samples the real correlation is much easier to detect.
Even when a real positive correlation exists, a small sample might miss it, or even suggest the opposite, simply due to chance. Because we usually only have one sample, we can never know for certain whether we were “lucky” or “unlucky.” The best protection is to collect more data.
2.5.3 Nonlinearity: when r misses real relationships
Consider a snake plant. Like most plants, it needs water to stay alive, but also needs the right amount of water. Imagine we grow 1000 snake plants, each receiving a different amount of water per day, ranging from no water to extreme overwatering (say 1000 teaspoons a day), and measure their weekly growth. Water is clearly causally related to plant growth, so we’d expect to see a correlation.
Plants given no water at all will eventually die, showing little or no weekly growth. With only a few teaspoons per day, plants may survive but grow only modestly. In a scatter plot, those cases would appear near the bottom left (low water, low growth) and rise upward as water increases. Growth continues to improve until a threshold is reached, beyond which excess water harms the plants. Severely overwatered plants also fail to grow and eventually die, just as those given no water do.
The scatter plot would form an inverted U or V: as water increases, growth rises to a peak and then declines with overwatering. A relationship like this can produce a Pearson’s \(r\) close to zero, even though water clearly affects growth. Figure 2.18 illustrates this pattern.
There is clearly a relationship between watering and snake plant growth. But, the correlation isn’t in one direction. As a result, when we compute the correlation in terms of Pearson’s r, we get a value very close to zero, suggesting no relationship.
#> [1] 0.009793996What this really means is there is no relationship that can be described by a single straight line. When we need lines or curves going in more than one direction, we have a nonlinear relationship.
This example shows why correlations can be tricky to interpret. Water is clearly necessary for plant growth, yet the overall correlation can be misleading. The first half of the data suggests a positive relationship, the second half a negative one, and the full dataset little or none. Even with a real causal link, a single correlation coefficient may fail to capture it. This is one reason plotting matters: if you see an upside-down U in the scatter, correlation is probably not the right analysis for your data.
2.5.4 Anscombe’s Quartet: same statistics, different data
Everything so far has compressed a relationship into a number: a covariance, an \(r\), a slope. Those numbers are summaries, and summaries lose things. The most famous demonstration of how much they can lose is a set of four datasets built to share almost every statistic you would think to compute.
Francis Anscombe (Anscombe (1973)) made this point with a famous example, now called Anscombe’s Quartet. Each panel in Figure 2.19 shows pairs of \(x\) and \(y\) values as a scatterplot.
Visually, the four panels are very different: one looks linear, one curved, one is mostly linear except for an outlier, and one forms a near-vertical line except for one point.
Yet if we summarize each quartet with just its descriptive statistics:
| quartet | mean_x | var_x | mean_y | var_y | cor_xy |
|---|---|---|---|---|---|
| 1 | 9 | 11 | 7.5 | 4.13 | 0.82 |
| 2 | 9 | 11 | 7.5 | 4.13 | 0.82 |
| 3 | 9 | 11 | 7.5 | 4.12 | 0.82 |
| 4 | 9 | 11 | 7.5 | 4.12 | 0.82 |
Rounded to two decimals, all four datasets have the same descriptive statistics:
same mean and variance for \(x\)
same mean and variance for \(y\)
same correlation between \(x\) and \(y\)
Yet the scatterplots could not look more different. (Anscombe constructed the four sets to agree to two decimal places, so they differ very slightly beyond that. Carry more decimals and the variances of \(y\) separate in the third one.)
2.5.5 Confounding variables (third-variable problem)
Correlations can arise because of a third variable: something not directly measured that influences both variables of interest. Returning to the snake plant example: imagine we measure water and plant growth, but ignore sunlight. If plants in sunnier windows both receive more water and grow faster, we might attribute the effect to water alone, when in fact sunlight is doing much of the work.
The same issue appears in human data. Suppose we find that, across an elementary school, children with bigger feet read better. Shoe size doesn’t drive reading ability; age independently makes feet bigger and makes reading better. Comparing shoe size and reading ability without accounting for age would credit shoe size with an effect that age is actually driving.
2.6 Chapter Summary
Why it matters. Correlation is how we quantify whether two variables move together. It’s essential for describing patterns (e.g., temperature and canopy cover, chocolate and happiness) and for deciding whether a relationship is worth modeling or acting on.
Core ideas
- Scatter first, always. A scatter plot shows direction (up, down, none), form (straight vs curved), and outliers. If the pattern is curved, don’t stop at r: fit a suitable model (e.g., add a quadratic term) or transform a variable instead.
- Covariance → correlation. Covariance measures whether deviations from each mean tend to align. Pearson’s r rescales covariance by the standard deviations so the result is unitless and bounded ([-1, 1]).
- Interpreting r. Sign indicates direction; magnitude indicates strength of a linear relationship. r near 0 can still hide a strong nonlinear pattern.
- Regression link. The best-fit line summarizes the average relationship; residuals are the vertical gaps between data and the line. The line is chosen to minimize the sum of squared residuals.
- Chance alone can manufacture a correlation. Small samples can produce large |r| by accident, and larger samples shrink the “chance window” toward 0. Always report r alongside its sample size (n) and, when possible, a confidence interval. If a result hinges on a small sample, say so and treat the conclusion as tentative.
- Nonlinearity hides real effects. Curved relationships (e.g., too little or too much water harming plant growth) can yield r ≈ 0 even when the underlying effect is strong.
- Confounding. A third variable can create or inflate a correlation (e.g., sunlight affecting both watering patterns and plant growth). This is why the honest phrasing is “associated with,” not “caused by.”