When two people arguing about a study, I often hear one of them talk about how a study automatically flawed and can’t be trusted. However, studies with small sample sizes regularly appear in meta-analyses. Why aren’t they automatically considered useless?
In: Mathematics
In observational analysis, there are many ways to approach small sample sizes while producing a useful result. The widely-used t-distribution is explicitly made for dealing with double-digit sample sizes under ~70 where using the standard normal distribution tends to be unhelpful. One might also try quasi-experimental methods — matching, propensity scores, regression discontinuity design, synthetic control, etc. — in which one attempts to coerce the statistical underpinnings of experimental analysis on a dataset *post-hoc*.
In *actual* experiments, sample size is still important, but the size needed for conventionally-useful findings varies widely by discipline and subject of interest. Social science, for example, has much less stringent conventions concerning the statistical significance of hypothesis testing results (i.e. the p < .05, or “95% chance the true population effect we estimated is inside the confidence interval for our sample”, which for most things is interpreted as “a 1/20 chance that we observed something due to randomness rather than a systemic process”) compared to something like high-energy particle physics (i.e. the infamous five-sigma criterion, or “roughly 1 in 3.5 million chance the true effect is *not* in our CI”).
There is also an epistemic/ontological (and practical, besides) concern discussed quite a bit by the statistics community in recent years regarding the tendency of applied disciplines — especially social sciences — to rely on hard cutoffs for results to be considered meaningful (e.g. the p < 0.05 above). [This](https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913) piece from Wasserstein et al. in 2019 gives a good and candid discussion of this issue and how we might work to reframe hypothesis testing.
So, depending on context, disciplinary standards, and what statistical tools are amenable to the data, you can still extract useful information from small samples.
Latest Answers