What exactly does it mean when a result is statistically significant vs. insignificant? When we compare, for example, a t-stat and the critical t-value, I know we either reject or fail to reject the null hypothesis based on whether the t-stat is less than or greater than the t-value. What exactly does it mean when the t-stat is greater than the critical t-value? What even is the “t-stat” and “critical t-value” in layman terms?
After doing enough problems, I’m sure I’ll get it, but I don’t like _not_ being able to explain this to myself simply – which indicates that I haven’t understood it well enough. Can someone please dumb all of this down for me and truly explain it to me like I’m a child?
In: Mathematics
tl;dr: the critical value is a benchmark, and the test statistic is the number you compare to the benchmark. the test statistic is calculated from the data you observed, and you’re assuming the data comes from some assumed distribution. you’re comparing the test statistic to the bench mark to see if there is evidence that your assumption is false. “statistical significance” is then related to the benchmark you choose.
——————–
To answer the questions of describing these concepts in layman’s terms, assuming you’ve done a few introductory lessons already:
* The “t-stat” is a number that’s the result of putting your data into an equation based on the **t-distribution**.
* The “critical t-value” is a value that is your benchmark for a certain probability that you would observe the data **if** your null hypothesis was true based on the **t-distribution**
If you reject the null hypothesis this is what you’re saying:
> The probability of observing the data, if we assume the data follows the t distribution, is *so low* that we feel our assumption is false. We have evidence that the data follows some *other* distribution.
——————————–
Analogy/example to hopefully clear up the *methodology* of this stuff: Let’s say you’re a basketball coach, and you’re scouting a player. Someone told you this particular player is a great ball-handler, so that’s your initial assumption when watching. Now you have to choose a benchmark to decide whether or not that assumption is wrong or not. Let’s say our benchmark is “losing the ball 3 times”. If he loses the ball 4 times, your assumption was wrong, your scout lied to you and this guy is not a great ball-handler.
Why 3? No profound reason, really. You think it’s a fair number. This could be 2 or 4 or 5. It’s whatever you want it to be (this is analogous to choosing your “significance level”). You probably wouldn’t choose 1 –maybe the player would just be unlucky on a play. But you feel 3 is a fair number so you don’t write off the player too soon. If by the end of the game, the player only loses the ball once, that would be statistically insignificant — you still think he’s a great ball handler. If he loses the ball 10 times, yeah that’s some *statistically significant* evidence that he’s not as great as you thought.
the above example was off the cuff and just to provide a different context for these basic concepts that maybe you wouldn’t have seen yet.
——————————————–
Now going past introductory statistics classes, it’s a recent development to move away from *having* to reject or fail to reject the null hypothesis due to rampant abuse and misunderstanding.
You don’t *have* to reject or fail to reject anything. Practically, is there a difference between a p-value of 0.499999999999999 and 0.500000001? I’d argue, in most cases, that’s a flat out no.
Latest Answers