When two people arguing about a study, I often hear one of them talk about how a study automatically flawed and can’t be trusted. However, studies with small sample sizes regularly appear in meta-analyses. Why aren’t they automatically considered useless?
In: Mathematics
Because you can measure how “truthful” a study is with something called statistical significance.
Let’s say you suspect your six sided die is loaded to always land on six. You throw it three times and get three sixes. Does that mean the die is loaded? Well not necessarily, because you absolutely can get three sixes in a row with a normal die too.
So you throw it ten times and get eight sixes and two fives. NOW are you sure the die is loaded? You’re far more sure than after rolling it three times, but there is STILL a possibility to get these results naturally with an unloaded die – it’s just extremely unlikely.
How “sure” you are that the results are because of the measured factor rather than random chance is “statistical significance”.
Scientists typically aim at statistical significance level of at least 95%. The exact value is always in the study, or even in the abstract. Generally speaking the statistical significance is one of the easiest things to check because it’s just high school level math. You can be certain that every published study is statistically significant, i.e. that the sample size is “big enough” – if it wasn’t, it wouldn’t be published.
Which is why dismissing studies because they have “small sample sizes” is a very amateur move and done mostly by people who are desperate to deny their results.
Latest Answers