I’ve noticed most U.S. polls query around 1,000 people and sometimes even less. Somehow that qualifies for headlines like “Americans say…” or “Most Americans…” How is it acceptable that 0.0002% of the population is accepted as representative?
In: Other
Adding onto what others have mentioned, I believe a visual representation is a valuable tool for understanding. If you search for “sample size and margin of error” in your browser, you’ll come across numerous graphs illustrating a consistent trend: as the sample size increases, the margin of error decreases. However, you’ll also observe that the reduction in error becomes less significant with larger samples. In other words, there’s a substantial difference in error between a sample of, for instance, 10 people and 100 people, but not a substantial difference between a sample of 1000 and 2000 people. Like this one: [https://ihopejournalofophthalmology.com/content/132/2022/1/1/img/IHOPEJO-1-009-g001.png](https://ihopejournalofophthalmology.com/content/132/2022/1/1/img/IHOPEJO-1-009-g001.png)
Why does this happen? The margin of error is calculated using the formula: Margin of Error = (Variation in sample / square root of sample size) * Z score.
I don’t know how good your mathematical intuition is, but notice **the diminishing returns in terms of sample size.** As we add more individuals, each new addition has a diminishing impact on reducing the error. This phenomenon arises from the nature of divisions and can be shown on the following numerical example: Assume we have a sample size of 1, 2 and 4 people respectively, and for simplicity I’ve ignored squaring everything. Then we get that 1/1 equals 1, 1/2 equals 0.5, and 1/4 equals 0.25. While 0.25 is undoubtedly smaller than 0.5, the difference between 1.00 and 0.5 is greater than that between 0.5 and 0.25. This mathematical principle results in diminishing returns from increasing sample sizes. We achieve smaller errors, but the rate of decrease in errors itself diminishes. Consequently, at a sufficiently large sample size, the difference becomes so negligible that it may as well be considered equal to zero.
**This insight explains why we accept “small” sample sizes, such as 0.0002% of the population. Eventually,** **the effort required to increase the sample size becomes disproportionate to the marginal reduction in error.**
Finally, there’s a crucial aspect not purely evident in mathematics but highlighted by other Redditors in this thread. **It’s not just about the sample size; the quality of the sample matters significantly.** The well-known example of the “The Literary Digest presidential poll” illustrates this point. According to Wikipedia, “The magnitude of the magazine’s error – 19.54% for the popular vote for Roosevelt vs. Landon, and even more in some states – destroyed the magazine’s credibility, and it folded within 18 months of the election. In hindsight, the polling techniques employed by the magazine were faulty. They failed to capture a representative sample of the electorate and **disproportionately polled higher-income voters, who were more likely to support Landon.** Although it had polled ten million individuals (of whom **2.27 million responded, an astronomical total for any opinion poll**), it had surveyed its own readers first […]”
This example demonstrates that while sample size is crucial, it’s not the sole determining factor. Having a correct sampling technique is equally, if not more, important. Other Redditors have delved into this aspect in more detail in this thread. I hope this clarifies things for you!
I’m not sure how to explain this in a way that a 5 year old would understand, but I’m a statistics major so this is up my alley.
Basically, there was a guy who worked for a beer company under the pseudonym “student” and he developed a table of numbers called a Student’s T Table which allows statisticians such as poll workers to make broader estimates of a larger population from a small randomized sample. He developed it so that he could test small samples of beer for quality without testing the entire batch.
The numbers generally work to some degree because most data follows what’s called a “normal distribution.” You’ve probably heard of this referred to as a “bell curve.” It’s essentially a random distribution of numbers that form around an average value in the center. Because most data follows this trend, we can estimate the population normal distribution (with a pre-determined level of confidence) relatively accurately from the normal distribution of a small random sample of that population.
Take a course in statistics and you will get the real answer. A statistical sample is not random in the way that you can be led to believe. Random polling by voice call preselects:
1. People who will answer a phone call from a stranger
2. People who want to talk
3. People who tend to be older
4. People within more populous geographic areas
5. People with a landline
Random voice voting excludes:
1. People with cellular phones with call filtering
2. People with prepaid cell phones
3. People with limited mobility who cannot reach the phone in time
4. People who do not want to talk
It goes on and on. Significant time is spent in class talking about how to select populations for statistical analysis. When someone puts together a national opinion poll they now have to stitch several methods together to reach the statistically valid sample. They need to hit a cross section of regions, ages, marital situations, educational attainment, and perhaps political affiliation or lack thereof.
Imagine I have a santa-sized sack of marbles of all sorts of colors.
Lets say we have somewhere like one million marbles.
I give you the task of telling me how many of them are red.
How many are you going to want to dump out and count before you’d feel confident giving me an answer?
All of them? Half of them? How correct do you want to be? Do you NEED to be exact?
Depending on what you’re trying to achieve, statistics has a lot of power for telling us what we need to know.
While you’re busy spending the [next week or two](https://nowiknow.com/how-long-would-it-take-to-count-to-a-million/) dumping them all out over the floor of a gymnasium and counting every single one and putting them back in the bag, I’m going to stick my arm in, stir them up for a minute, and grab a few handfuls. Maybe 20 marbles. An amount that I can count out in a minute or less. I find 2 red marbles in 20. I toss them back in and repeat. This time I find 3 in 20. Then I repeat again and find 2 in 20 again.
I’ll do that 10 times, and come to the conclusion that there are on average 2.3 red marbles in 20, or 115,000 total red marbles in the sack.
Then I’m going to spend 10 minutes with some statistical calculations, use the standard deviation of the sample results, and use the formulas to determine a 95% and 99% confidence level.
E.g. this might be “I am 95% confident that there are 115,000 +/- 3000 red marbles” and “I am 99% confident that there are 115,000 +/- 8000 red marbles”
The samples and those results can mathematically tell me that there’s only a 1% chance that my sampling was wrong outside of a range of 107,000 to 123,000.
My test was done in under an hour with a printed report, whereas counting any meaningful fraction of marbles will take much longer.
What my test relies on is that my sampling was sufficiently random, i.e. the marbles were well mixed before and between sampling.
So when surveying people, ideally we want to randomly sample from the target population. That’s actually very hard to do, and it’s a valid reason these studies are flawed.
E.g. If you wanted to sample random people, you could stand on a street corner and interview passers by. But your sampling will be skewed towards people who walk to work. If you’re sampling at 2pm in the afternoon, you’re skewing away from people who work 9-5 office jobs. Almost any method of sampling in person has a location bias.
One of the best ways to sample is to get a list of phone numbers of county residents, use a random number generator to pick 1000 of them at random, and then start calling. The best data list is probably a list of registered voters if it includes phone numbers. Of course, you’re then skewed based on time of day, and towards people who actually have the patience to answer your annoying questions.
Because there are so many ways to accidentally bias your sampling, a well designed study will also ask demographic questions like ethnicity, address, gender, age, etc. These may be useful for making headline conclusions like “People over 60… “, but they’re also useful for just checking biases. E.g. you can use census data for a county to find out that 40% of residents are over the age of 60. If you run your survey and it turns out that 70% of the respondents are over 60 yrs of age, that’s an indication that you may not have had a sufficiently random sample, and you need to overhaul your sampling technique (e.g. maybe your phone list includes cell phones and landlines, and older citizens are likely to have cell phones AND landlines, making them twice as likely for one of their numbers to get selected).
[removed]
Statistician here. It surprises a lot of people to learn that you don’t necessarily need a very large sample to get good information about a group. In fact, assuming an infinitely-sized population, it’s possible to get meaningful results with a sample as small as 385!
At it’s core, statistics is about getting information about a large group (the population) by examining a smaller group (the sample). It’s assumed that the sample is an accurate reflection of the population. This is likely to be true because of probability. At a certain point, it’s mathematically improbable for the sample to be too different from the population.
Let’s say that you have a jar full of 500 green marbles and 500 orange marbles (50% of each). You randomly pull out 100 marbles. Mathematically, you’re probably going to end up with 50 green marbles and 50 orange marbles, or at least pretty close to that. It may be technically possible to get lucky and pull out 100 green marbles, but that’s so unlikely that it’s not much of a concern. Sampling is the same way. At a certain point, it’s extremely unlikely that the sample will be radically different from the population.
There is a catch here–the sample has to be taken randomly. If I were to look for green marbles to pull out of the jar, it’s not going to accurately represent the jar’s contents. Likewise, samples need to selected randomly from the population to be valid. In practice, this is almost never the case, especially for large samples. You will always have a sampling method that excludes people or a number of people who don’t respond after they’re selected. However, surveys still have to assume that the sampling was random. That’s why it’s important to look at the sampling method used by the poll, and pay attention to multiple polls to make up for the errors in individual surveys.
If it helps, here’s a [sample size calculator](https://www.calculator.net/sample-size-calculator.html) that you can use to look at how big the sample needs to be for a survey. As stated earlier, using an infinite population, a confidence level of 95%, and a margin of error of 5%, you would only need a sample of 385. That’s on the higher end of acceptability though, but you can play around with other numbers to find out a better sample size.
It’s the make up adverts on tv here in the UK that get me. 76% out of 120 people surveyed etc. I’m a data analyst as a job and this just screams adding one more until they can prove their point until they hit a believable percentage.
Just the world we live in unfortunately
If the USA has 341 million people, the question to ask is for a question with say four choices are you going to get 341 million different responses?
So the question then arises, given a random set of individuals, how many do you need to get close to the national ratio of the four possible choices?
1000 is chosen as the upper limit as it is not an unmanageable number to poll and gives about a 3% margin of error.
If the variation within that group is small, and it’s a diverse sample, then you can be confident. Unfortunately both those things need to be true
Surveys have been done with many more people, but once abut 1000 are polled the numbers seem to correlate very highly to polls with more data.
Latest Answers