Sample Size Calculations and Systematic Reviews

Author

Daniel Sun, Amanda Ng

Introduction

Welcome to the Sample Size Calculations module. By the end of this workshop, you will be able to:

Explain why sample size matters and what happens when a study is underpowered or overpowered
Define Type I error (alpha), Type II error (beta), and statistical power
Calculate Cohen’s d as a standardized measure of effect size
Use the sample size formula for a two-sample comparison
Interpret how alpha, power, and effect size each influence the required sample size
Perform a sensitivity analysis showing sample size across a range of assumptions

This module uses the Wisconsin Breast Cancer dataset (n=569), which contains measurements from fine needle aspirate (FNA) images of breast masses, each classified as malignant (M) or benign (B).

Part 1: Sample Size Calculations

Section 1: Why Sample Size Matters

The problem with underpowered studies

Imagine you design a study to test whether a new imaging biomarker can distinguish malignant from benign breast tumours. You recruit 20 patients. Even if the biomarker truly works, a sample of 20 may not give you enough statistical evidence to detect the difference reliably. This is an underpowered study: it has a high chance of returning a non-significant result even when an effect genuinely exists.

The opposite problem also exists. If you recruit 50,000 patients to detect a trivially small and clinically irrelevant difference, you will almost certainly find a significant result, but it tells you nothing practically useful. This is an overpowered study: it wastes resources and may lead to overinterpreting noise.

Determining sample size in advance forces you to be explicit about: 1. What size of effect is clinically meaningful? 2. How much uncertainty are you willing to accept?

The Wisconsin dataset as a motivating example

The Wisconsin dataset was collected by Dr. William Wolberg at the University of Wisconsin to support computer-aided diagnosis of breast FNA samples. It contains 30 features computed from digitised cell nuclei images (radius, texture, perimeter, area, etc.) for 569 patients: 212 malignant and 357 benign.

We will use radius_mean (mean radius of cell nuclei) as our primary outcome, comparing malignant vs benign tumours.

1.1 Explore the data

Run the code below to load the dataset and compare radius_mean between the two diagnostic groups.

viewof mcq1 = Inputs.radio(
  ["Recruiting as many patients as possible to be safe.",
   "Deciding in advance how many participants are needed based on the expected effect and acceptable error rates.",
   "Running the study and stopping when p < 0.05.",
   "Using the same sample size as the most recent published study on the topic."],
  {label: html`<b>Question: What is the purpose of an a priori sample size calculation?</b>`}
)

{
  if (mcq1 === null) return html`<span></span>`;
  if (mcq1 === "Deciding in advance how many participants are needed based on the expected effect and acceptable error rates.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> An a priori sample size calculation is done before data collection. It forces you to specify the effect size you want to detect, your acceptable Type I error rate, and your desired power.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Sample size calculations should happen before the study begins, based on explicit assumptions about effect size and error rates.
  </div>`;
}

Section 2: Error Types and Power

2.1 Type I and Type II errors

Every statistical test can make two kinds of mistakes:

Decision	Truth: No effect	Truth: Effect exists
Reject H0 (significant)	Type I error (false positive)	Correct (true positive)
Fail to reject H0	Correct (true negative)	Type II error (false negative)

Type I error rate (alpha): The probability of claiming an effect when none exists. Conventionally set at 0.05 (5%).
Type II error rate (beta): The probability of missing a real effect. Conventionally 0.10 or 0.20.
Power (1 - beta): The probability of correctly detecting a real effect when it exists. Conventionally 80% or 90%.

Tip

Think of it this way: alpha is the false alarm rate, and power is the hit rate. You want the false alarm rate low and the hit rate high. Unfortunately, for a fixed sample size, reducing one tends to worsen the other. The only way to improve both simultaneously is to increase n.

viewof mcq2 = Inputs.radio(
  ["The study found a significant result even though no real effect exists.",
   "The study missed a real effect because it was underpowered.",
   "The study's alpha level was set too high.",
   "The study recruited too many participants."],
  {label: html`<b>Question: A study comparing tumour radius in malignant vs benign FNA samples fails to reach p < 0.05, but the true difference is 5 mm. This is an example of:</b>`}
)

{
  if (mcq2 === null) return html`<span></span>`;
  if (mcq2 === "The study missed a real effect because it was underpowered.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> When a real effect exists but the study fails to detect it, that is a Type II error. It commonly happens when the sample size is too small relative to the effect size.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> The study had a real effect but did not detect it. Which type of error does that describe?
  </div>`;
}

2.2 Visualizing power

The plot below shows how power changes as sample size increases, for three different effect sizes. Notice that larger effects require fewer participants to achieve the same power.

View code for plot

power_curve <- function(n_vec, d, alpha = 0.05) {
  sapply(n_vec, function(n) {
    delta <- d * sqrt(n / 2)
    z_a   <- qnorm(1 - alpha / 2)
    pnorm(delta - z_a) + pnorm(-delta - z_a)
  })
}
n_seq <- seq(5, 120, by = 1)
df <- rbind(
  data.frame(n=n_seq, power=power_curve(n_seq,0.5),  effect="Small (d=0.5)"),
  data.frame(n=n_seq, power=power_curve(n_seq,0.8),  effect="Medium (d=0.8)"),
  data.frame(n=n_seq, power=power_curve(n_seq,2.05), effect="Large (d=2.05)")
)
ggplot(df, aes(x=n, y=power, colour=effect)) +
  geom_line(linewidth=1.1) +
  geom_hline(yintercept=c(0.80,0.90), linetype=c("dashed","dotted"), colour="grey40") +
  scale_y_continuous(labels=scales::percent_format(), limits=c(0,1)) +
  labs(title="Power curves by effect size", x="Sample size per group", y="Power") +
  theme_bw()

View code for plot

from scipy import stats
def power_curve(n_vec, d, alpha=0.05):
    z_a = stats.norm.ppf(1 - alpha / 2)
    delta = d * np.sqrt(np.array(n_vec) / 2)
    return stats.norm.cdf(delta - z_a) + stats.norm.cdf(-delta - z_a)

n_seq = np.arange(5, 121)
for d, label, col in [(0.5,"Small","tomato"),(0.8,"Medium","steelblue"),(2.05,"Large","forestgreen")]:
    plt.plot(n_seq, power_curve(n_seq, d), label=label, color=col, linewidth=1.8)
plt.axhline(0.80, linestyle="--", color="grey"); plt.axhline(0.90, linestyle=":", color="grey")
plt.xlabel("Sample size per group"); plt.ylabel("Power")
plt.title("Power curves by effect size"); plt.legend(); plt.tight_layout(); plt.show()

viewof mcq3 = Inputs.radio(
  ["Larger effects need more participants to detect.",
   "All three effect sizes require the same sample size for 80% power.",
   "Smaller effects need fewer participants because they are easier to measure.",
   "Larger effects need fewer participants to achieve the same power."],
  {label: html`<b>Question: What does the power curve plot tell you about the relationship between effect size and required sample size?</b>`}
)

{
  if (mcq3 === null) return html`<span></span>`;
  if (mcq3 === "Larger effects need fewer participants to achieve the same power.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> A large effect size is easier to detect statistically, so you need fewer participants to achieve the same power. The Wisconsin dataset's d = 2.05 means even a small study could detect the radius difference reliably.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Look at where each curve crosses the 80% power line. Which effect size gets there at the smallest n?
  </div>`;
}

Section 3: Effect Size (Cohen’s d)

3.1 What is Cohen’s d?

When comparing two group means, Cohen’s d expresses the difference in units of the pooled standard deviation:

\[d = \frac{\mu_1 - \mu_2}{s_{\text{pooled}}}\]

where the pooled standard deviation is:

\[s_{\text{pooled}} = \sqrt{\frac{s_1^2 + s_2^2}{2}}\]

Cohen’s d is a standardized effect size: it has no units, so it can be compared across studies and outcomes. Cohen’s conventional benchmarks are:

d	Interpretation
0.2	Small
0.5	Medium
0.8	Large
> 1.0	Very large

3.2 Exercise: Compute Cohen’s d for radius_mean

Solution

pooled_sd <- sqrt((sd_M^2 + sd_B^2) / 2)
cohens_d  <- (mean_M - mean_B) / pooled_sd
cat("Pooled SD:  ", round(pooled_sd, 3), "\n")
cat("Cohen's d:  ", round(cohens_d, 3), "\n")
cat("Interpretation: very large effect (d > 1.0)\n")

Solution

pooled_sd = np.sqrt((sd_M**2 + sd_B**2) / 2)
cohens_d  = (mean_M - mean_B) / pooled_sd
print(f"Pooled SD:  {pooled_sd:.3f}")
print(f"Cohen's d:  {cohens_d:.3f}")
print("Interpretation: very large effect (d > 1.0)")

viewof mcq4 = Inputs.radio(
  ["0.2 — small", "0.5 — medium", "0.8 — large", "Greater than 2 — very large"],
  {label: html`<b>Question: Based on Cohen's conventional benchmarks, how would you classify d = 2.05?</b>`}
)

{
  if (mcq4 === null) return html`<span></span>`;
  if (mcq4 === "Greater than 2 — very large")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> Cohen's benchmarks go up to 0.8 for "large", but d = 2.05 is far beyond that. This reflects the dramatic biological difference in cell nucleus size between malignant and benign tumours.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Compare d = 2.05 against Cohen's benchmarks: small = 0.2, medium = 0.5, large = 0.8. Where does 2.05 sit?
  </div>`;
}

Section 4: The Sample Size Formula

4.1 Derivation

For a two-sided, two-sample t-test, the number of participants required per group is:

\[n = \frac{2(z_{\alpha/2} + z_{\beta})^2}{d^2}\]

where:

\(z_{\alpha/2}\) is the critical z-value for the chosen alpha (1.96 for alpha = 0.05)
\(z_{\beta}\) is the z-value corresponding to the desired power (0.842 for 80% power; 1.282 for 90% power)
\(d\) is Cohen’s d

Note

This formula assumes equal group sizes. If your groups are unequal (like the Wisconsin dataset’s 212 malignant and 357 benign), a correction factor is needed, but this version is a widely used starting point.

4.2 Exercise: Compute required sample size

Solution

cohens_d  <- (mean_M - mean_B) / sqrt((sd_M^2 + sd_B^2) / 2)
z_alpha   <- qnorm(1 - 0.05 / 2)
z_beta_80 <- qnorm(0.80)
z_beta_90 <- qnorm(0.90)
n_80 <- ceiling(2 * (z_alpha + z_beta_80)^2 / cohens_d^2)
n_90 <- ceiling(2 * (z_alpha + z_beta_90)^2 / cohens_d^2)
cat("Cohen's d:              ", round(cohens_d, 3), "\n")
cat("Required n per group (80% power):", n_80, "\n")
cat("Required n per group (90% power):", n_90, "\n")
cat("Total n (80% power, equal groups):", n_80 * 2, "\n")
cat("Total n (90% power, equal groups):", n_90 * 2, "\n")

Solution

cohens_d  = (mean_M - mean_B) / np.sqrt((sd_M**2 + sd_B**2) / 2)
z_alpha   = stats.norm.ppf(1 - 0.05 / 2)
z_beta_80 = stats.norm.ppf(0.80)
z_beta_90 = stats.norm.ppf(0.90)
n_80 = math.ceil(2 * (z_alpha + z_beta_80)**2 / cohens_d**2)
n_90 = math.ceil(2 * (z_alpha + z_beta_90)**2 / cohens_d**2)
print(f"Cohen's d:                        {cohens_d:.3f}")
print(f"Required n per group (80% power): {n_80}")
print(f"Required n per group (90% power): {n_90}")
print(f"Total n (80% power, equal groups): {n_80 * 2}")
print(f"Total n (90% power, equal groups): {n_90 * 2}")

viewof mcq5 = Inputs.radio(
  ["A much larger sample is needed because the effect is harder to measure.",
   "A much smaller sample is needed because the effect is easier to detect statistically.",
   "The same sample size is always needed regardless of effect size.",
   "Sample size only depends on alpha, not effect size."],
  {label: html`<b>Question: For d = 2.05 compared to d = 0.5, what happens to the required sample size?</b>`}
)

{
  if (mcq5 === null) return html`<span></span>`;
  if (mcq5 === "A much smaller sample is needed because the effect is easier to detect statistically.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> Because d appears in the denominator of the sample size formula (d²), a larger effect size dramatically reduces the required n. A medium effect (d=0.5) needs around 64 per group at 80% power, while d=2.05 needs only about 4.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Look at the formula: n = 2(z_alpha + z_beta)² / d². What happens to n as d increases?
  </div>`;
}

Section 5: Sensitivity Analysis

A sensitivity analysis shows how your required sample size changes as you vary your assumptions. This is essential in practice because your pre-study estimates of effect size and acceptable error rates are rarely perfect.

Exercise: Build a sensitivity analysis table

Solution

z_alpha <- qnorm(1 - 0.05 / 2)
d_values     <- c(0.2, 0.5, 0.8, 1.0, 2.05)
power_levels <- c(0.80, 0.90)
results <- data.frame()
for (d in d_values) {
  for (pwr in power_levels) {
    z_beta <- qnorm(pwr)
    n_per_group <- ceiling(2 * (z_alpha + z_beta)^2 / d^2)
    results <- rbind(results, data.frame(
      Cohen_d=d, Power=paste0(pwr*100,"%"),
      n_per_group=n_per_group, total_n=n_per_group*2
    ))
  }
}
print(results)

Solution

z_alpha = stats.norm.ppf(1 - 0.05 / 2)
d_values = [0.2, 0.5, 0.8, 1.0, 2.05]
power_levels = [0.80, 0.90]
rows = []
for d in d_values:
    for pwr in power_levels:
        z_beta = stats.norm.ppf(pwr)
        n_per_group = math.ceil(2 * (z_alpha + z_beta)**2 / d**2)
        rows.append({"Cohen_d":d, "Power":f"{int(pwr*100)}%",
                     "n_per_group":n_per_group, "total_n":n_per_group*2})
results = pd.DataFrame(rows)
print(results.to_string(index=False))

Now let’s visualize the sensitivity analysis as a heatmap.

View code for plot

z_alpha <- qnorm(1 - 0.05/2)
d_values <- c(0.2, 0.3, 0.5, 0.8, 1.0, 1.5, 2.05)
power_levels <- c(0.70, 0.80, 0.90, 0.95)
grid <- expand.grid(d=d_values, power=power_levels)
grid$n <- ceiling(2*(z_alpha+qnorm(grid$power))^2/grid$d^2)
grid$n_label <- ifelse(grid$n>999,">999",as.character(grid$n))
grid$power_label <- paste0(grid$power*100,"%")
ggplot(grid, aes(x=factor(d), y=factor(power_label), fill=pmin(n,300))) +
  geom_tile(colour="white") + geom_text(aes(label=n_label), size=3.5) +
  scale_fill_gradient(low="forestgreen", high="tomato") +
  labs(title="Sample size per group", x="Cohen's d", y="Power") + theme_bw()

View code for plot

z_alpha = stats.norm.ppf(1 - 0.05/2)
d_values = [0.2, 0.3, 0.5, 0.8, 1.0, 1.5, 2.05]
power_levels = [0.70, 0.80, 0.90, 0.95]
n_matrix = np.zeros((len(power_levels), len(d_values)))
for i, pwr in enumerate(power_levels):
    for j, d in enumerate(d_values):
        n_matrix[i, j] = math.ceil(2 * (z_alpha + stats.norm.ppf(pwr))**2 / d**2)
fig, ax = plt.subplots(figsize=(9, 4))
im = ax.imshow(np.minimum(n_matrix, 300), cmap="RdYlGn_r", aspect="auto")
plt.colorbar(im, ax=ax, label="n per group (capped at 300)")
for i in range(len(power_levels)):
    for j in range(len(d_values)):
        ax.text(j, i, str(int(n_matrix[i,j])) if n_matrix[i,j]<=999 else ">999", ha="center", va="center", fontsize=9)
ax.set_xticks(range(len(d_values))); ax.set_xticklabels(d_values)
ax.set_yticks(range(len(power_levels))); ax.set_yticklabels([f"{int(p*100)}%" for p in power_levels])
ax.set_xlabel("Cohen's d"); ax.set_ylabel("Power")
ax.set_title("Sample size per group (two-sided t-test, alpha = 0.05)")
plt.tight_layout(); plt.show()

viewof mcq6 = Inputs.radio(
  ["Increasing power from 80% to 90% always doubles the required sample size.",
   "A small effect size (d=0.2) requires far more participants than a large effect size (d=0.8) for the same power.",
   "Sample size is independent of effect size.",
   "Increasing alpha always reduces the required sample size."],
  {label: html`<b>Question: What is the main practical lesson from the sensitivity analysis heatmap?</b>`}
)

{
  if (mcq6 === null) return html`<span></span>`;
  if (mcq6 === "A small effect size (d=0.2) requires far more participants than a large effect size (d=0.8) for the same power.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> Effect size is the dominant driver of sample size. At 80% power, d=0.2 needs around 197 per group while d=0.8 needs only 26. Getting a good estimate of effect size before planning a study is therefore the most important step in a sample size calculation.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Look at the colour gradient across the columns of the heatmap. What changes most dramatically as d changes?
  </div>`;
}

Part 2: Systematic Reviews and Forest Plots

The second half of this module shifts from planning a single study to synthesising evidence across multiple studies. You will work through the logic of a meta-analysis: computing odds ratios, pooling them across cohorts, testing for heterogeneity, and producing a forest plot.

Learning outcomes for Part 2:

Articulate what makes a systematic review different from an informal literature review
Compute an odds ratio and its 95% confidence interval from a 2x2 table
Explain when to prefer odds ratios over relative risk
Read every element of a forest plot: squares, lines, diamond, and the null line
Calculate Cochran’s Q and I² and use them to judge heterogeneity
Choose between fixed-effect and random-effects models based on the evidence
Write the code to draw a publication-ready forest plot

Data. The same Wisconsin Breast Cancer dataset (n=569) is used, now split into six simulated study cohorts to replicate the multi-study structure of a real meta-analysis.

Section 7: Systematic Reviews

7.1 What is a systematic review?

A systematic review is a structured synthesis of all available evidence on a clearly defined research question. Unlike a narrative review (where the author selects studies based on their own judgement), a systematic review uses:

A pre-specified, reproducible search strategy across multiple databases
Explicit inclusion and exclusion criteria
Standardized data extraction
A formal assessment of study quality (risk of bias)
Statistical pooling of results (meta-analysis) when appropriate

PRISMA guidelines

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement provides a checklist and flow diagram for transparently reporting systematic reviews. Most journals require PRISMA compliance for submitted systematic reviews.

7.2 Why pool studies?

Individual studies are often underpowered to detect modest effects or to estimate them precisely. A meta-analysis pools results across studies, increasing effective sample size and producing a more precise overall estimate. The pooled estimate is displayed as the diamond at the bottom of a forest plot.

viewof mcq8 = Inputs.radio(
  ["A review where the author selects whichever studies they find most convincing.",
   "A structured synthesis using pre-specified search and inclusion criteria, with formal statistical pooling.",
   "A review that only includes randomised controlled trials.",
   "A single large study that replicates previous findings."],
  {label: html`<b>Question: Which of the following best describes a systematic review?</b>`}
)

{
  if (mcq8 === null) return html`<span></span>`;
  if (mcq8 === "A structured synthesis using pre-specified search and inclusion criteria, with formal statistical pooling.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> The defining features are pre-specification (to avoid cherry-picking), explicit criteria (for reproducibility), and formal pooling (to quantify the combined evidence).
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Think about what distinguishes a systematic review from a narrative review. The key is pre-specification and reproducibility.
  </div>`;
}

Section 8: Odds Ratios

2.1 What is an odds ratio?

An odds ratio compares the odds of an outcome in one group to the odds in another. Given a 2x2 table:

	Outcome present	Outcome absent
Exposed	a	b
Unexposed	c	d

\[\text{OR} = \frac{a/b}{c/d} = \frac{ad}{bc}\]

For our Wisconsin dataset, we define: - Exposure: concavity_mean above the median (high concavity) - Outcome: malignant diagnosis

	Malignant	Benign
High concavity	198	87
Low concavity	14	270

OR vs Relative Risk

The relative risk (RR) is the ratio of probabilities: P(outcome | exposed) / P(outcome | unexposed). The OR uses odds instead of probabilities. For rare outcomes, OR approximates RR. For common outcomes, OR exaggerates the association compared to RR. In case-control studies, only OR (not RR) can be calculated because sampling is done on outcome, not exposure.

2.2 Interpreting the OR

OR = 1: no association
OR > 1: the exposure is associated with increased odds of the outcome
OR < 1: the exposure is associated with decreased odds of the outcome
The 95% CI tells you the precision of the estimate. If it crosses 1, the result is not statistically significant at alpha = 0.05.

2.3 Exercise: Compute the odds ratio

Solution

OR <- (a_overall * d_overall) / (b_overall * c_overall)
cat("Overall Odds Ratio:", round(OR, 2), "\n")
cat("Interpretation: patients with high concavity have", round(OR, 1),
    "times the odds of malignancy compared to those with low concavity.\n")

Solution

OR = (a_overall * d_overall) / (b_overall * c_overall)
print(f"Overall Odds Ratio: {OR:.2f}")
print(f"Interpretation: patients with high concavity have {OR:.1f}x the odds of malignancy.")

viewof mcq9 = Inputs.radio(
  ["OR = 1: high concavity doubles the odds of malignancy.",
   "OR > 1: high concavity is associated with increased odds of malignancy.",
   "OR < 1: high concavity is protective against malignancy.",
   "OR cannot be interpreted without knowing the p-value."],
  {label: html`<b>Question: OR = 43.9. Which interpretation is correct?</b>`}
)

{
  if (mcq9 === null) return html`<span></span>`;
  if (mcq9 === "OR > 1: high concavity is associated with increased odds of malignancy.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> An OR greater than 1 means the exposed group has higher odds of the outcome. OR = 43.9 means the odds of malignancy are 43.9 times higher in the high-concavity group.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> An OR of 43.9 is much greater than 1. What does OR > 1 mean for the association between exposure and outcome?
  </div>`;
}

Section 9: Confidence Intervals for the OR

3.1 The log-OR and its standard error

OR confidence intervals are computed on the log scale (because the log-OR is approximately normally distributed), then back-transformed:

\[\text{SE}(\ln \text{OR}) = \sqrt{\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}}\]

\[95\%\ \text{CI} = \exp\left(\ln\text{OR} \pm 1.96 \times \text{SE}(\ln\text{OR})\right)\]

3.2 Exercise: Compute the 95% CI

Solution

OR_overall <- (a_overall * d_overall) / (b_overall * c_overall)
log_OR    <- log(OR_overall)
se_log_OR <- sqrt(1/a_overall + 1/b_overall + 1/c_overall + 1/d_overall)
ci_lo     <- exp(log_OR - 1.96 * se_log_OR)
ci_hi     <- exp(log_OR + 1.96 * se_log_OR)
cat("OR:         ", round(exp(log_OR), 2), "\n")
cat("95% CI:     [", round(ci_lo, 2), ",", round(ci_hi, 2), "]\n")

Solution

log_OR    = math.log(OR_overall)
se_log_OR = math.sqrt(1/a_overall + 1/b_overall + 1/c_overall + 1/d_overall)
ci_lo     = math.exp(log_OR - 1.96 * se_log_OR)
ci_hi     = math.exp(log_OR + 1.96 * se_log_OR)
print(f"OR: {OR_overall:.2f}, 95% CI: [{ci_lo:.2f}, {ci_hi:.2f}]")

viewof mcq10 = Inputs.radio(
  ["The 95% CI crosses 1, so the result is not statistically significant.",
   "The 95% CI is entirely above 1, confirming a statistically significant positive association.",
   "The width of the CI tells us the OR is unreliable.",
   "CI calculations are only valid for randomised controlled trials."],
  {label: html`<b>Question: The 95% CI for OR is [18.7, 102.4]. What can you conclude?</b>`}
)

{
  if (mcq10 === null) return html`<span></span>`;
  if (mcq10 === "The 95% CI is entirely above 1, confirming a statistically significant positive association.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> When the entire 95% CI is above 1, the null hypothesis (OR = 1, no association) is rejected at alpha = 0.05. The association is statistically significant and in the positive direction.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Check whether the CI includes the value 1 (the null value for an OR). If it does not include 1, what does that imply for statistical significance?
  </div>`;
}

Section 10: The Forest Plot

4.1 Reading a forest plot

A forest plot displays the results of multiple studies alongside a pooled estimate. Each study is represented by:

A square whose position on the x-axis is the point estimate (OR)
Horizontal lines extending from the square that represent the 95% CI
The size of the square represents the study’s weight in the meta-analysis (larger studies get more weight)
A diamond at the bottom showing the pooled estimate and its CI

The vertical line at OR = 1 is the line of no effect. Studies whose CI crosses this line are not individually significant.

4.2 Exercise: Compute per-study ORs and CIs

Solution

studies$log_or <- log(studies$a * studies$d / (studies$b * studies$cc))
studies$se     <- sqrt(1/studies$a + 1/studies$b + 1/studies$cc + 1/studies$d)
studies$or     <- exp(studies$log_or)
studies$ci_lo  <- exp(studies$log_or - 1.96 * studies$se)
studies$ci_hi  <- exp(studies$log_or + 1.96 * studies$se)
studies$weight <- 1 / studies$se^2
print(studies[, c("study","n")])
print(round(studies[, c("or","ci_lo","ci_hi","weight")], 2))

Solution

studies["log_or"] = np.log(studies["a"] * studies["d"] / (studies["b"] * studies["c"]))
studies["se"]     = np.sqrt(1/studies["a"] + 1/studies["b"] + 1/studies["c"] + 1/studies["d"])
studies["or"]     = np.exp(studies["log_or"])
studies["ci_lo"]  = np.exp(studies["log_or"] - 1.96 * studies["se"])
studies["ci_hi"]  = np.exp(studies["log_or"] + 1.96 * studies["se"])
studies["weight"] = 1 / studies["se"]**2
print(studies[["study","n","or","ci_lo","ci_hi","weight"]].round(2).to_string(index=False))

viewof mcq11 = Inputs.radio(
  ["The study with the smallest OR.",
   "The study with the narrowest confidence interval, because it has more precision.",
   "The study with the widest confidence interval.",
   "All studies receive equal weight regardless of sample size."],
  {label: html`<b>Question: In inverse-variance weighting, which study gets the most weight in the pooled estimate?</b>`}
)

{
  if (mcq11 === null) return html`<span></span>`;
  if (mcq11 === "The study with the narrowest confidence interval, because it has more precision.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> Inverse-variance weighting gives more weight to studies with smaller variance (smaller SE), which corresponds to narrower CIs. Larger studies tend to have smaller SEs and thus receive more weight.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> The weight is 1/SE². A smaller SE means a larger weight. What does a smaller SE correspond to in terms of precision?
  </div>`;
}

Section 11: Heterogeneity

5.1 What is heterogeneity?

Heterogeneity refers to variation in the true effect across studies beyond what would be expected from sampling variability alone. Sources include differences in study populations, exposure definitions, outcome ascertainment, and confounders.

Two key statistics measure heterogeneity:

Cochran’s Q: A chi-squared test of whether observed variation exceeds chance. Calculated as the weighted sum of squared deviations of each study’s log-OR from the pooled log-OR.
I²: The proportion of total variation that is due to between-study heterogeneity (rather than sampling error). Ranges from 0% to 100%.

\[Q = \sum_i w_i (\ln\text{OR}_i - \ln\text{OR}_{\text{pooled}})^2\]

\[I^2 = \max\left(0, \frac{Q - (k-1)}{Q}\right) \times 100\%\]

where k is the number of studies.

I²	Interpretation
0-25%	Low heterogeneity
25-50%	Moderate heterogeneity
50-75%	Substantial heterogeneity
75-100%	Considerable heterogeneity

5.2 Fixed-effect vs random-effects models

Fixed-effect model: Assumes all studies estimate the same true effect. Appropriate when heterogeneity is low (I² < 25%) and studies are methodologically homogeneous.
Random-effects model: Assumes each study estimates a slightly different true effect drawn from a distribution of true effects. More conservative (wider CIs) and appropriate when heterogeneity is present.

5.3 Exercise: Compute heterogeneity statistics

Solution

studies$log_or <- log(studies$a * studies$d / (studies$b * studies$cc))
studies$se     <- sqrt(1/studies$a + 1/studies$b + 1/studies$cc + 1/studies$d)
studies$weight <- 1 / studies$se^2
pooled_log_or <- sum(studies$weight * studies$log_or) / sum(studies$weight)
Q  <- sum(studies$weight * (studies$log_or - pooled_log_or)^2)
k  <- nrow(studies)
I2 <- max(0, (Q - (k - 1)) / Q) * 100
cat("Pooled OR (fixed-effect):", round(exp(pooled_log_or), 2), "\n")
cat("Cochran's Q:            ", round(Q, 2), "\n")
cat("I-squared:              ", round(I2, 1), "%\n")

Solution

studies["log_or"] = np.log(studies["a"] * studies["d"] / (studies["b"] * studies["c"]))
studies["se"]     = np.sqrt(1/studies["a"] + 1/studies["b"] + 1/studies["c"] + 1/studies["d"])
studies["weight"] = 1 / studies["se"]**2
pooled_log_or = np.sum(studies["weight"] * studies["log_or"]) / np.sum(studies["weight"])
Q  = np.sum(studies["weight"] * (studies["log_or"] - pooled_log_or)**2)
k  = len(studies)
I2 = max(0, (Q - (k - 1)) / Q) * 100
print(f"Pooled OR: {np.exp(pooled_log_or):.2f}, Q: {Q:.2f}, I²: {I2:.1f}%")

viewof mcq12 = Inputs.radio(
  ["Use a fixed-effect model because all studies are from the same dataset.",
   "Use a random-effects model regardless of heterogeneity.",
   "Choose based on I²: low heterogeneity supports fixed-effect; substantial heterogeneity supports random-effects.",
   "I² does not influence model choice."],
  {label: html`<b>Question: How should I² guide the choice between fixed-effect and random-effects meta-analysis?</b>`}
)

{
  if (mcq12 === null) return html`<span></span>`;
  if (mcq12 === "Choose based on I²: low heterogeneity supports fixed-effect; substantial heterogeneity supports random-effects.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> I² quantifies how much of the observed variation reflects genuine between-study differences. When that proportion is large, assuming all studies estimate the same true effect (fixed-effect) is implausible, and a random-effects model is more appropriate.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> I² tells you how much between-study variation there is. High I² means the studies may not all be estimating the same effect.
  </div>`;
}

Section 12: Drawing the Forest Plot

6.1 Components of a forest plot

The forest plot below displays all six study cohorts plus the pooled estimate. Notice:

The x-axis uses a log scale because ORs are multiplicative (OR = 2 and OR = 0.5 are equidistant from OR = 1 on the log scale)
Study squares are sized proportionally to their inverse-variance weight
The pooled estimate is shown as a diamond whose width spans the 95% CI

View code for plot

# Full code in the autorun chunk above
# Key elements:
# - geom_segment() for CI lines
# - geom_point(shape=15) for study squares sized by weight
# - geom_point(shape=18) for pooled diamond
# - scale_x_log10() for log scale x-axis
# - geom_vline(xintercept=1) for line of no effect

View code for plot

# Key elements of forest plot construction:
# - ax.plot([ci_lo, ci_hi], [y, y]) for CI lines
# - ax.scatter(or, y, marker="s", s=weight_scaled) for study squares
# - ax.scatter(pooled_or, y, marker="D") for pooled diamond
# - ax.set_xscale("log") for log scale
# - ax.axvline(1) for line of no effect

viewof mcq13 = Inputs.radio(
  ["A study with very wide confidence intervals, because wide CIs indicate a large OR.",
   "A study with a very narrow confidence interval, because it has more precision and thus more weight.",
   "The study with the largest point estimate.",
   "All studies contribute equally."],
  {label: html`<b>Question: In the forest plot, which study has the largest square, and why?</b>`}
)

{
  if (mcq13 === null) return html`<span></span>`;
  if (mcq13 === "A study with a very narrow confidence interval, because it has more precision and thus more weight.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> Square size reflects inverse-variance weight (1/SE²). A narrower CI corresponds to a smaller SE and thus a larger weight. In the Wisconsin cohorts, Study 1 and Study 2 tend to have the largest squares because their cell counts are more balanced, giving smaller SEs.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Square size reflects weight = 1/SE². What does a small SE correspond to visually in the CI?
  </div>`;
}

Summary

Part 1: Sample Size Calculations

Understood why sample size must be pinned down before recruitment, not after.

Defined Type I error, Type II error, and power and traced how they relate.

Computed Cohen’s d from group means and standard deviations.

Applied the two-sample t-test sample size formula to a real clinical dataset.

Built a sensitivity table showing how n shifts across a grid of effect sizes and power levels.

Part 2: Systematic Reviews and Forest Plots

Distinguished systematic reviews from narrative reviews and identified what makes the former reproducible.

Computed an odds ratio directly from a 2x2 table.

Derived a 95% CI for an OR using the log-scale method.

Calculated inverse-variance weights and understood why larger studies dominate the pooled estimate.

Computed Cochran’s Q and I² and used them to assess whether pooling is appropriate.

Chose between fixed-effect and random-effects models based on the heterogeneity evidence.

Constructed a forest plot from scratch and interpreted every element of it.

viewof mcq7 = Inputs.radio(
  ["Use a smaller alpha to increase power.",
   "Increase the sample size, since power rises with n for any fixed effect size.",
   "Use a one-sided test, which always gives more power.",
   "Reduce the effect size threshold to make the study easier to power."],
  {label: html`<b>Final question (Part 1): A researcher's power calculation shows only 60% power for their planned study. What is the most principled way to increase power without changing the significance threshold?</b>`}
)

{
  if (mcq7 === null) return html`<span></span>`;
  if (mcq7 === "Increase the sample size, since power rises with n for any fixed effect size.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> Power is a direct function of sample size (for fixed alpha and effect size). Recruiting more participants is the primary lever available to researchers once the study design is set.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Think about which quantities in the power formula are under the researcher's control after the hypothesis and alpha are fixed.
  </div>`;
}

viewof mcq14 = Inputs.radio(
  ["They always have the same true effect, so pooling is always valid.",
   "The CI of the pooled estimate is always wider than any individual study.",
   "When the CI of the pooled estimate crosses 1, the association is not significant at alpha = 0.05.",
   "The pooled estimate is always the simple average of all study ORs."],
  {label: html`<b>Final question (Part 2): Which statement about the pooled estimate in a fixed-effect meta-analysis is correct?</b>`}
)

{
  if (mcq14 === null) return html`<span></span>`;
  if (mcq14 === "When the CI of the pooled estimate crosses 1, the association is not significant at alpha = 0.05.")
    return html`<div style="padding:8px;background:#d4edda;border-left:4px solid #28a745;border-radius:4px;margin-top:4px">
      ✅ <strong>Correct!</strong> OR = 1 is the null value (no association). If the 95% CI of the pooled estimate includes 1, the pooled result is not statistically significant at the 5% level. This is directly analogous to the confidence interval for any other statistic.
    </div>`;
  return html`<div style="padding:8px;background:#f8d7da;border-left:4px solid #dc3545;border-radius:4px;margin-top:4px">
    ❌ <strong>Not quite.</strong> Think about what OR = 1 means and what the CI tells you about statistical significance.
  </div>`;
}

Key takeaway

radius_mean in the Wisconsin data has a Cohen’s d of 2.05, meaning even a very small study would be adequately powered to detect the malignant vs benign difference. Most real clinical research has much smaller effect sizes (d = 0.2-0.5), making rigorous sample size planning essential. The sensitivity table is the deliverable that belongs in grant applications and ethics submissions.

For the meta-analysis, pooling six cohorts on concavity_mean produced a fixed-effect OR of approximately 44 for malignancy. Individual ORs ranged from roughly 19 to 100, a reminder that between-study variation is the norm. The forest plot and heterogeneity statistics are what allow readers to judge whether that pooled number is meaningful or whether the studies are too different to pool at all.

Return to Learning Hub Homepage

Learning Hub Homepage