I did a biology undergraduate degree and often did reports where would statistically analyse our results. P value of less than 0.05 shows that the results are statistically significant. How do these tests actually know the data is significant? For example we might look at correlation and get a significant positive correlation between two variables. Given that variables can be literally anything in question, how does doing a few statistical calculations determine it is significant? I always thought there must be more nuance as the actual variables can be so many different things. It might show me a significant relationship for two sociological variables and also for two mathematical, when those variables are so different?
In: Mathematics
The root problem is that “significant” has a common-language definition and a stats definition that aren’t the same. In your post, you’re using it like the common usage meaning “substantial” or “important”.
In stats-language, “significant” just means “less than ___% chance this result is a random coincidence”, where ___% is *whatever P value threshold you choose* to use.
If you decide to use P of 0.05 as your cutoff, that means a 95% chance the effect/result is real, 5% chance that it’s a random coincidence. So if you get P < 0.05, it means “less than 5% chance this is random coincidence”.
But you could just as easily choose to use P = 0.4 as your cutoff. Then let’s say you do the calculation and find some effect has P = 0.3. That P is smaller than your chosen “threshold of significance”, so by definition that effect is “significant” (by the stats definition) *even though you just showed it has a 30% chance of being a random coincidence*.
>How do these tests actually know the data is significant
They don’t – that’s a misuse of the common meaning of “significant”. **Calculating P only tells you a probability of an observed effect being due to random coincidence, and you decide (by your choice of critical P, often 0.05 by convention) at what probability of false alarm you’re willing to call the result “significant”.**
Latest Answers