I am a grad student desperately trying to analyze her data. I am having a hard time understanding why correcting for the amount of tests I’m doing (Bonferoni and Tukey) is taking away my significance. I have 4 factors across 3 timepoints and when I run stats on each factor across the timepoints, they are significant. When I put them all together on one graph (all four factors across all 3 timepoints), they are no longer significant. I understand how Bonferoni works, what I am asking is why does it feel like I am being punished with stricter p-values when I am being more thorough? I feel like this correction encourages people to break down their data in order to get significance, which feels icky. Im wishing I would have studied just one of the factors across the timepoints instead of all 4.
In: 4
It’s worth noting that Bonferroni is a *really* conservative (erring on the side of non-significance) correction. It’s an easy option but generally not a great one — there are other procedures, probably available in whatever stats software you’re using or just out there for you to look up, that can be perfectly appropriate to use.
Also, there are cases when it’s not really necessary to perform multiple testing corrections at all, or to perform your corrections in smaller groups… which you noted can be *icky* if it comes from a motivation of just p-fishing, but is much more defensible when you do it in a way that aligns with your actual hypotheses (for instance, correcting the timepoint-wise comparisons separately per factor). But opinions about that differ a lot and what is appropriate to one statistician may not be to another.
Latest Answers