What exactly does it mean when a result is statistically significant vs. insignificant? When we compare, for example, a t-stat and the critical t-value, I know we either reject or fail to reject the null hypothesis based on whether the t-stat is less than or greater than the t-value. What exactly does it mean when the t-stat is greater than the critical t-value? What even is the “t-stat” and “critical t-value” in layman terms?
After doing enough problems, I’m sure I’ll get it, but I don’t like _not_ being able to explain this to myself simply – which indicates that I haven’t understood it well enough. Can someone please dumb all of this down for me and truly explain it to me like I’m a child?
In: Mathematics
Statistical significance means that **if the null hypothesis were true**, the chances of observing a difference as (or more) extreme as what we’re seeing are so low that we’re pretty sure we can correctly reject the null.
Say you’re testing a die to see if it’s fair or loaded. The null hypothesis is that it’s fair, so you would expect to see the numbers 1-6 show up in roughly equal proportions across multiple rolls.
As an extreme example, say you roll it 100 times and get a 6 every time. Obviously the chances of rolling the same number 100/100 times on a fair die are ridiculously low, so we can confidently reject the null hypothesis that this is a fair die.
But what if you couldn’t roll the die 100 times? What if you could only roll it 10 times? This is analogous to doing a study with limited resources and only being able to recruit a sample size of so many people.
If you rolled a six 10/10 times, you could still be pretty sure it’s loaded, but not as sure as if it were 100/100 times. If you only had 5 rolls and rolled a six every time, you might suspect it’s loaded, but you probably wouldn’t bet your life on it. At two rolls, it’s impossible to draw a conclusion, since you can very easily roll the same number twice in a row by chance.
The critical value, aka the significance threshold, is just a mathematical way of establishing when you would feel confident rejecting the null, based on the number of rolls you have and the outcome you observed.
For any outcome we observe over a number of rolls, we can calculate the probability of getting that outcome on a fair die, since we know the mean and SD of a fair die (mean = 3.5 and SD = 1.7). **This is the test statistic/t-stat**.
We can then set our **critical value/significance threshold**, being the probability of an outcome occurring beyond which we would be confident in rejecting the null. In science, this is most often 95%. It’s an arbitrary value, but is commonly used by convention. So in this case, we can say that if the outcome we observed had less than a 5% probability of occurring by chance on a fair die, it’s statistically significant, and we’ll reject the null.
As mentioned, our ability to draw these conclusions depends on the sample size. For example, if you only had two rolls, any possible of outcomes would lie within that 95% range. In order to reach statistical significance, you would have to observe a mean value between the two rolls that is either less than 1 or greater than 6; both of which are impossible on a 6-sided die. Therefore there is no possibility of reaching statistical significance with a sample that small. As you increase the sample size, you increase the power of the study to reach statistical significance.
Latest Answers