Guidance on Conducting Sample
Size and Power Calculations
Applied Statistics Seminar Series
Northwestern University
Department of Preventive Medicine
Presented by: Lutfiyya N. Muhammad, PhD MPH
Assistant Professor
Department of Preventive Medicine – Biostatistics
Biostatistics Collaboration Center (BCC)
Northwestern University, Feinberg School of Medicine
Presented on: June 21, 2022
Who has taken a course that was focused on sampling
methods and sample size calculations?
Outline
• Hypothesis Testing
• Power Analysis
• Required Information for Power Analysis
• Different Types of Effect Sizes
• Power Analysis via Software
• Power Analysis via Simulations
• Resources
• Summary
My General Process for Sample Size or Power Calculations
Examine aims,
hypotheses,
and key
variables.
Think about
how the key
variables are
collected.
Develop an
analysis plan.
Determine
values needed
for power
analysis and
inform
collaborators.
Conduct the
power
analysis.
Hypothesis Testing
• Hypotheses are stated clearly.
- Null Hypothesis (e.g. no difference among groups/arms)
- Alternative Hypothesis (e.g. difference among groups)
• Compute test statistic from data.
• Compare test statistic to a critical value.
• Reject or fail to reject null hypothesis.
Hypothesis Testing
(Kim, 2015)
Power Analysis
• Power analysis is the calculation that is used to determine the minimum sample
size needed for a research study.
• Power analysis is conducted before the study begins.
• To compute the power or sample size, you will need:
- Null and alternative hypotheses
- The statistical method that will be used to test the null hypothesis
- Effect size or variability
- Statistical significance level (α = 0.05)
- Either sample size or power (opposite of what you are solving for)
• Power is traditionally set at 80%
Keep in mind…
• Grant proposals includes several hypotheses depending on the number of aims.
- Calculations for the sample size or power are based on the primary hypothesis.
• You can include a sample size calculation or power analysis for secondary
hypotheses.
- Secondary hypothesis may only be tested using a subset of the sample or a
different sample from the primary hypothesis.
• Sample size can be dependent on the budget of the proposed study.
- How many study participants can we have based on the budget?
• There is not one simple formula for computing a power analysis.
Obtaining the Values Required for a Power Analysis
• Investigators should provide statisticians with the values required for the power
analysis.
• Values can come from:
- Pilot Study
- Literature Review
- Cohen’s recommendations
Things to consider…
• Are the values given for the power analysis consistent with how the primary
outcome and primary variable(s) will be measured?
• You may need to utilize a meta-analysis method to get an overall effect size.
• Does the power analysis need to account for dropouts?
Common Effect Sizes
(Sullivan & Feinn, 2012)
Cohen’s Other Effect Sizes
• Cohen’s w – effect size for Chi-Squared tests
• Cohen’s h – effect size for comparing two independent proportions
• Cohen’s
2
– effect size for F-test from ANOVA or multiple regression
Power Calculations via Software
• Power Analysis & Sample Size (PASS)
• R: pwr package
• SAS: PROC POWER & PROC GLMPOWER
• SPSS
• G*Power
• nQuery+nTerim4.0
• Stata: sampsi, fpower, powerreg, aipe
Example 1 – Finding the Sample Size
• Research Objective - To clinically validate the use of a biomarker that correctly
identifies patients with pain.
- Biomarker based on white blood cells count
- Pain measured through conditioned pain modulation (CPM)
• Hypotheses
- Null: Sensitivity = 60%
- Alternative: Sensitivity = 80%
• Statistical analysis plan included computing the sensitivity between the two
measures.
Example 1 – Finding the Sample Size
• Effect size for sensitivity is determined by the values of the prevalence of pain
and by the values of sensitivity under the hypotheses.
• I came across a peer-reviewed publication (Bujang & Adnan, 2016) that
computed multiple sample size calculation scenarios using PASS.
- Power ≥80%
- Various levels of alpha, prevalence, and sensitivity under the hypotheses
Bujang & Adnan, 2016
Example 1 – Finding the Sample Size
Prevalence of
Pain
Sensitivity –
Under the Null
Hypothesis
Sensitivity- Under
the Alternative
Hypothesis
Minimum Sample
Size for Cases of
Pain
Minimum Total
Sample Size
20% 60%
80%
45 225
70% 155 775
30% 60% 45 150
70% 155 517
Power Calculations via Simulations
• Power analyses conducted using options from standard software may not be
suitable for complex study designs
• Power calculations via simulations can be useful when conducting a power
calculation that is not standard.
General Steps for Power Calculations via Simulations
• Step 1 – Generate random data from the distribution with the properties that
you expect.
• Step 2 – Conduct the statistical analysis that you have planned to do.
• Step 3 – Store the p-value for the statistical test of interest (β=0).
• Step 4 – Repeat steps 1-3 many times (e.g. 1000).
• Step 5 – Compute power by averaging the number of times the p-value ≤ 0.05.
Resources – Power Analysis for Animal Studies
Resources – RNA-Seq Power Analysis
Resources – Mediation Analysis with Longitudinal Data
Summary
• Research questions and hypotheses must be well defined.
• Power analyses are based on the primary hypothesis.
• Sample size can be dependent on the budget of the proposed study.
• Many factors that can impact power.
References
Bujang MA, Adnan TH. Requirements for Minimum Sample Size for Sensitivity and Specificity Analysis. J Clin Diagn
Res. 2016 Oct;10(10):YE01-YE06. doi: 10.7860/JCDR/2016/18129.8744. Epub 2016 Oct 1. PMID: 27891446; PMCID:
PMC5121784.
Effect Size. https://en.wikipedia.org/wiki/Effect_size. Accessed June 13, 2022.
Kim HY. Statistical notes for clinical researchers: Type I and type II errors in statistical decision. Restor Dent Endod.
2015;40(3):249-252. doi:10.5395/rde.2015.40.3.249
Lee J. Sample Size and Power Calculation Using Simulation. Applied Statistics Seminar; May 9, 2017, 2017;
Northwestern University.
Sullivan GM, Feinn R. Using Effect Size-or Why the P Value Is Not Enough. J Grad Med Educ. 2012;4(3):279-282.
doi:10.4300/JGME-D-12-00156.1
Arifin WN, Zahiruddin WM. Sample Size Calculation in Animal Studies Using Resource Equation Approach. Malays J
Med Sci. 2017;24(5):101-105. doi:10.21315/mjms2017.24.5.11
References
Charan J, Kantharia ND. How to calculate sample size in animal studies?. J Pharmacol Pharmacother. 2013;4(4):303-
306. doi:10.4103/0976-500X.119726
Pan, H., Liu, S., Miao, D. et al. Sample size determination for mediation analysis of longitudinal data. BMC Med Res
Methodol 18, 32 (2018).
https://doi.org/10.1186/s12874-018-0473-2
Thank You!
Email: lutfiyya.muhammad@northwestern.edu
