What is the goal of multiple comparisons?
We want to move from:
“Is there any difference between groups?”
to:
“Which specific groups are different?”
Why do we need multiple comparisons?
ANOVA only tells you:
- whether a difference exists somewhere
It does not tell you:
- which groups differ
- how many groups differ
- where the difference is
So we need a second step.
What is the full ANOVA process?
Step 1: Run ANOVA
- tests all groups at once
Step 2: Run multiple comparisons
- identifies specific group differences
What does ANOVA actually test?
It tests:
“Are all group means equal?”
- If p > .05 → no differences
- If p < .05 → at least one difference exists
Key idea:
ANOVA tells you that a difference exists, not where it is.
What are multiple comparisons doing?
They answer:
“Which groups differ from each other?”
They break down the overall ANOVA result into specific comparisons.
Two types of comparisons
Post hoc (pairwise)
Question:
“Are any of the groups different from each other?”
Answer:
Compare every group with every other group
- A vs B
- A vs C
- B vs C
Used when:
- no specific hypothesis
- exploratory analysis
Planned (a priori)
Question:
“Do these specific groups differ in a way I predicted?”
Answer:
Compare selected groups only
- A vs B
- (A + B) vs C
Used when:
- comparisons are decided before analysis
- hypothesis-driven
What exactly are we comparing?
Group means (averages)
Not categories or labels, only numerical outcomes.
What is a contrast?
The difference between group means
Examples:
- A − B
- (A + B)/2 − C
Key idea:
All comparisons are forms of subtraction.
Why do we use weights?
We assign weights (contrast coefficients) to tell the model what to compare.
You are turning a comparison into a mathematical equation the program can understand.
What do weights mean?
- Positive → included on one side
- Negative → included on the other side
- Zero → ignored
How does a simple comparison work?
You subtract one group’s mean from another group.
Example: Mickey vs Batman
- Mickey = +1
- Batman = −1
- Superman = 0
Represents:
Mickey − Batman
How do we combine groups?
When groups represent the same idea in the hypothesis we add them together
Example:
(Superman + Batman) vs Mickey
Weights:
- Superman = +0.5
- Batman = +0.5
- Mickey = −1
Represents:
(Superman + Batman)/2 − Mickey
What rules do weights follow?
- Must sum to 0
- Define the structure of the comparison
- Positive → positive-weighted group is higher
- Negative → negative-weighted group is higher
Does it matter which group is positive or negative?
No, only the direction changes
- magnitude stays the same
- p-value stays the same
- significance stays the same
Only the wording changes.
What is the t-statistic?
The value that tells us whether the difference between groups is meaningful.
Conceptually:
t = difference between means ÷ variability
- Large difference + low variability → large t → significant
- Small difference or high variability → small t → not significant
Where does variability come from?
It comes from the Mean Square Within (MSwithin) from ANOVA
This Represents:
- variation within groups
- random differences
What does the p value tell us?
The likelihood that the observed difference happened by chance
- p < .05 → significant difference
- p > .05 → not significant
What is the problem with running multiple comparisons?
- Every time you run a test you have have a 5% error rate
- The more tests you run, the higher the error rate becomes
- Each test increases the chance of false positives
This leads to:
family-wise error rate
What is family-wise error rate?
The probability of making at least one Type I error across multiple tests.
More comparisons:
- higher error risk
How do we fix this?
Use correction methods such as the Bonferroni.
What is the Bonferroni correction?
A method to make significance stricter.
Method 1: Adjust alpha
Adjusted alpha = 0.05 ÷ number of comparisons
Example:
3 comparisons → 0.0167
Method 2: Adjust p-values
Adjusted p = original p × number of comparisons
Compare to:
0.05
What is the effect of Bonferroni?
- reduces false positives
- increases strictness
- may miss real effects
When should we apply corrections?
Post hoc comparisons
- many comparisons
- higher risk
Correction is required.
Planned comparisons
- fewer comparisons
- hypothesis-driven
Correction may or may not be used.
