I did a biology undergraduate degree and often did reports where would statistically analyse our results. P value of less than 0.05 shows that the results are statistically significant. How do these tests actually know the data is significant? For example we might look at correlation and get a significant positive correlation between two variables. Given that variables can be literally anything in question, how does doing a few statistical calculations determine it is significant? I always thought there must be more nuance as the actual variables can be so many different things. It might show me a significant relationship for two sociological variables and also for two mathematical, when those variables are so different?
In: Mathematics
statistical test cannot “prove” significance. In fact, you cannot prove statistical significance at all, you can only measure it. There are many techniques to measure it, but then you usually get a couple of numbers. Usually, most important number is the p-value, (other numbers include size effects).
p-value stands for probability. It measures the probability that nothing happened, but you got good results anyway due to luck.
For example. Let’s say I claim to be a psychic who could control chaos magic like Wanda and determine the result of a coin toss. How many heads in a row is enough to convince you that I am a psychic?
If I throw 2 heads in a row, you might just call me lucky. If I throw 5 heads in a row, you might think that I might be up to something. If I throw like 20 heads in a row, I will definitely get your interest. Either I got an excellent throwing techniques, or there are tricks in the coin, or I’m a real psychic, but you would be pretty sure it is not up to chance.
So maybe for you the limit is somewhere between 5 and 20 coin tosses. If you do the statistical test, the P-value of 5-head is 0.03125, while for 20-head is 9.5e-7.
Now, the same with biology, let’s say testing if a medicine is working. How do we know if a medicine is working, or is just up to luck?
*************************************
Well, you wanna find out the p-value. to do that, you use one of the many statistical tests. These are tools that people can misuse and abuse. In fact, it is kinda hard to get it right.
And then you get a p-value. Different fields have different standard. It seems that you are familiar with p<0.05, which is 1 in 20 chance that it is luck. Other fields use p< 5sigmas, which translates to 1 in a million chance.
https://news.mit.edu/2012/explained-sigma-0209
Latest Answers