Research Context of Statistics

We don’t conduct statistical analyses for no reason. We do it for research, so it’s important to understand the context behind the data you are analysing.

Questions to ask:

• What is the Research Question?
• What is the sample & population?
• What is the unit of measurement?

What Is Inferential Statistics?

• Once we understand the research context, we then conduct inferential statistics.

• Inferential statistics = using data from a sample to infer (make predictions) about the larger population

• We can’t usually test the whole population, so we infer conclusions from a representative smaller group.

• This process is called hypothesis testing.

Key Statistical Details

Concept	What It Means	Simple Example
H₀ (Null Hypothesis)	Assumes no effect, no difference, or no relationship. Presumed true unless evidence suggests otherwise.	“There is no difference between Group A and Group B.”
H₁ (Alternative Hypothesis)	Claims there is an effect, difference, or relationship.	“There is a difference between Group A and Group B.”
α (Alpha Level)	The significance level. The cutoff for deciding if results are ‘extreme’ under H₀. Usually 0.05 (5%).	If p < .05 → reject H₀
Test Statistic & p-value	Test statistic = calculated value (e.g., t, z, χ²). p-value = probability of observing results this extreme if H₀ were true.	Small p-value → evidence against H₀

$$

Broader Research Process Steps

1. Make observation
2. Review literature and identify related theories
3. Generate aims, RQs, hypotheses
4. Design stufy
5. Get ethical approval for stufy
6. Run study & collect data
7. Analyse data
8. Write report on findings

Statistical Analyses Covered In The Unit

Recap Of Important Statistics Concepts

Statistical Significance vs Effect Size

After running statistical analyses you will get:

• A test statistics with degrees of freedom
• A p-value

What do p-values tell us?

• Whether the effect observed is statistically significant or likely to due to chance
• We use the alpha value of 0.05
• If p < 0.05, we reject the null hypothesis and the effect is statistically significant
• e.g., if p=.912 there is a 91.2% chance we could see the same pattern by pure chance. Therefore the result is not statistically significant
• The test statistic and the df determine the p-value
• p-values are affected by sample size

How does sample size impact the p-value?

• Sample size (n) is used to calculate the test statistic
• Larger n = larger test statistic –> smaller p = small effects become significant
• Larger sample sizes increase statistical power, reducing random error and allowing small effects to be detected

What is the standardised effect size?

• Value that measures the magnitude of the effect independent of the sample size
• How big was the effect?

Point Estimate vs Interval Estimate

• We use samples to estimate population parameters
• Population parameters = fixed values that describe a characteristic of the population (mean, SD)

Feature	Point Estimate	Interval Estimate
Definition	A single value used to estimate a population parameter	A range of values used to estimate a population parameter
Example (Stats)	Sample mean (x̄), difference between means, sample correlation (r)	95% Confidence Interval (e.g., 3.2 to 5.6)
Example (Everyday)	“My cat wakes me at 4am.”	“My cat wakes me between 3–5am.”
Precision	More precise (one number)	Less precise (range of values)
Accuracy	Less reliable across samples (varies each time)	More informative because it accounts for sampling variability
Sampling Variation	Changes every time we take a new sample	Captures uncertainty caused by sampling
Common Interpretation	Best single guess of parameter	“We are 95% confident the true population parameter lies within this range.”
Trade-off	Precise but ignores uncertainty	More accurate but less precise

Key Ideas

1. Point and Interval Estimates Are Related

• A point estimate is the single value you calculate from your sample.
• An interval estimate is built around that point estimate.
• Confidence intervals use the point estimate as the centre.

2. Confidence Intervals (CI)

• A CI = point estimate ± margin of error
• Margin of error depends on:
  • Sample size (n)
  • Variability (standard error)
  • Critical value (linked to confidence level, e.g., 95%)

3. Effect of Sample Size (n)

• Larger n → Narrower CI
• Smaller n → Wider CI
• Why? Larger samples give more stable estimates.

4. Precision vs Accuracy

• Narrow CI → more precision
• Wider CI → more uncertainty acknowledged
• With equal n:
  • Narrower CI = more precision
  • Wider CI = less precision

5. 95% Confidence Interval Means:

If we repeated the study many times:

• 95% of the calculated intervals would contain the true population parameter.
• If the range crosses zero –> test is NOT statistically significant
• If range doesn’t cross zero –> test IS statistically significant
• STATA command for 95% confidence interval = ci2 var1 var2, corr

Sample Size Effect On Statistical Significance

Effect on p-values & test statistics

• Sometimes if you have very large sample sizes, the p-value and effect size can tell different stories
• Larger sample sizes mean smaller effects (p-values) are more likely to be significant, even if there is no real effect happening
• If you were to take a random smaller subset of the sample and run the same test, you might see the p-value increase and no longer read as statistically significant

Effect on assumptions testing

• With a large sample size, the formal assumption test might appear violated/ not met
• Important to do a visual test as well (a histogram might show the data actually does look pretty normally distributed)
• Don’t just rely on formal inferential assumptions tests

Larger n >> larger test statistic, smaller p-value, narrower CI

Smaller n >> smaller test statistic, larger p-value, wider CI

The Significance Fallacy

• Don’t just focus on the p-value to determine the significance of the effect because sample sizes can influence the result
• Important to also check standardised effect size values which will not be impacted by the n

Type 1 & Type 2 Error

Decision ↓ / Reality →	Null Hypothesis TRUE	Null Hypothesis FALSE
Reject H₀	Type I Error (False Positive)	Correct Decision (True Positive)
Fail to Reject H₀	Correct Decision (True Negative)	Type II Error (False Negative)

Statistical Power

What is Power?

• Power = 1 − β
• It is the probability of correctly rejecting a false null hypothesis.
• In simple terms: the ability to detect a real effect if it exists.

If power = 0.80:

• There is an 80% chance of detecting a true effect.

What Increases Statistical Power?

1. Higher α (alpha level)
   • Larger significance level → more power
   • (But increases risk of Type I error)
2. Larger Effect Size
   • Bigger true effects are easier to detect.
3. Larger Sample Size (n)
   • Reduces sampling variability.
   • Makes estimates more stable.
   • Increases power.

Key Relationships

• Low power → higher chance of Type II error (β).
• High power → more likely to detect real effects.

Factors Affecting Statistical Power

1. Research Design

• Some designs are more powerful than others.
• Within-subjects designs are usually more powerful than between-subjects.
  • Each participant acts as their own control.
  • Reduces variance from individual differences.

2. Measurement Decisions

• Keeping a variable continuous (numeric) increases power.
• Categorising (e.g., turning scores into “high/low”) reduces power.
  • You lose information.

3. Model Complexity

• Fewer unnecessary parameters (fewer df used up) can increase power.
• Simpler, efficient models tend to be more powerful.