Hello! There are many ELI about statistical power, but I didn’t see one asking quite what I’m wondering about – the basic question is:
If power is related to the number of people required to detect a difference between groups, is it important that a trial was underpowered if it found a statistically significant difference between groups?
It seems like maybe not(?) since at that point you can no longer have made a Type 2 error, only Type 1? But being underpowered is often discussed, so I’m curious why.
In: Mathematics
I’m not sure what studies you’re reading, but I agree that it would be odd for someone to say “Our trial rejected the null, but it was underpowered, so we can’t come to a clear conclusion.” In fact, it seems more likely they would say “Our trial rejected the null despite being underpowered, so we likely found a very strong relationship.”
That said, there’s a different reason why power can be important when the null is rejected. If your sample is so large that you have enough power to detect and reject on even tiny differences between treatment and control, then it’s possible to report a statistically significant difference that is of little consequence. For example, we usually don’t have the power to detect that an experimental treatment increases survival rates by 0.000001%. Supposing we somehow did, it’s unlikely that treatment would be worthwhile, even though its effect was “significant.” This is rarely an issue in clinical trials but can come up when people are studying large datasets like the Census.
Latest Answers