I’m hearing this phrase a lot in my meetings and from what I gather, it’s basically the number an analysis needs in order to reach a calculated conclusion. Basically more data = closer to stat sig?

In: Mathematics

I’m not sure if this is exactly what it means for your situation but with psychology research statistical significance means if the difference between the numbers actually means something or if its just a difference due to chance. So most papers will use “p<0.05” which means the results showing a difference are less than 5% likely to be due to chance. If the p value is over 0.1 then there’s more than 10% likelihood it’s random and you can’t really say something if meaningful or effective if its just chance.

So let’s imagine we flip coin ten times and count the number of heads and tails. Now we know what the probability of a heads or tails in a single flip of a totally fair coin is 50%, so ten flips should lead to 5 heads and 5 tails. However let’s say we get 8 heads and 2 tails. Now this seems like an interesting data set, because we went from a theoretical expected count of 50% heads to a actual count of 80% heads in our sample. However we can’t be sure is this result is just due to randomness, or if it is actually telling us something about this coin, like it is weighted so heads is more likely. The point of this example is to show that in real life, there are numerous observations we can make and ask the question, “Did this occur due to pure chance or is there a specific cause?”

Essentially, we use the term statistically significant to indicate that a certain result that has been observed is extremely unlikely to have occured due to chance, and has a specific cause.

Also, more data doesn’t necessarily mean it’s statistically significant. Depending on how the data was collected, there could be biases that lead to a result that may not be accurate.

If I roll a dice 1 time and it comes up showing a 6, can we say it is an unfair weighted die that shows 6 more than statistically reasonable? No we can not, we don’t have enough information. So we roll it 10 more times, and it comes up showing 6 twice more, out of those 10. Can we say it is an unfair weighted die now? Not really, even though that’s still showing 6 slightly more often than we would expect. There’s simply not enough rolls to feel strongly about the results. Getting 6 3 times in 11 rolls is a lot, but not super improbable. It’s not statistically significant enough.

If we roll it 1,000 times and it rolls a 6 300 times, now can we say it’s weighted? Yes we probably can. That’s enough rolls where it should have distributed more evenly than that. It’s a significant statistical anomaly.

So, there will be a particular deviation level (usually 2 standard deviations, or 95% of probable distribution) where we would consider it to be significant and unlikely to happen by chance except in the most outlandish of instances, to the point where we feel comfortable in the resulting conclusions.