I am a grad student desperately trying to analyze her data. I am having a hard time understanding why correcting for the amount of tests I’m doing (Bonferoni and Tukey) is taking away my significance. I have 4 factors across 3 timepoints and when I run stats on each factor across the timepoints, they are significant. When I put them all together on one graph (all four factors across all 3 timepoints), they are no longer significant. I understand how Bonferoni works, what I am asking is why does it feel like I am being punished with stricter p-values when I am being more thorough? I feel like this correction encourages people to break down their data in order to get significance, which feels icky. Im wishing I would have studied just one of the factors across the timepoints instead of all 4.
In: 4
The more distinct tests you do, the more likely you are to find patterns that aren’t there. You realized this instinctively when you said that breaking the data up to increase significance is icky.
A great example of why is this xkcd https://xkcd.com/882/
If you run 20 different tests at a 5% significance level, you would expect one of the tests to find a pattern even if none is there
Latest Answers