What does “Polls Say” in an Election even mean? How are they updated? No one has ever asked me my thoughts. Who are they asking and who is actually answering?
In: Other
Ok, so to do this, it will be useful to look at an actual polls. You can go find a bunch here:
https://projects.fivethirtyeight.com/polls/national/
I’m going to use this poll for my example, which is the top poll for July 16:
https://www.ipsos.com/sites/default/files/ct/news/documents/2024-07/Reuters%20Ipsos%20Post%20Trump%20Assasionation%20Attempt%20Poll%20Topline.pdf
but what I’m going to say works for all of them.
** Part 1: What a poll is**
A poll is nothing more than someone asking questions. That’s it. In a way, all of /r/AskReddit is free-response polling, because it’s just asking a question and getting a bunch of answers. And shitty “news” sites use it like that, and report the answers like something meaningful.
*Political* polls are more than just questions about political subjects. They are groups of questions structured in a certain way, in hopes of getting answers that are *predictive* of something, usually public sentiment about an elected leader or insight into the potential outcomes of an election.
Polls are made in accordance with the rules of statistics, in particular [rules for eliminating types of bias that can make them unreliable](https://www.surveymonkey.com/learn/survey-best-practices/how-to-avoid-common-types-survey-bias/). Pollsters do a bunch of things, but for our purposes they try to ask as many people as possible, they try to select who they ask as randomly as possible, they try to ask questions as neutrally as possible, and they they try to ask questions that can only be answered in certain narrow ways.
So for example, if I ask the residents of a college dorm which college football team is the most fun to watch, I’m probably just going to get a bunch of answers saying that school’s team. That’s not helpful for measuring anything but school spirit. If I ask the valedictorian of every graduating class in the land what team is the most fun to watch, I’ll get a different answer, but it may not be a very reliable answer because there’s no connection between that population and the answer. But if I ask every head coach and wide receivers coach on every P5 football team who the most skilled all-around wide receiver is out of a group of 10 pre-selected names, I will in fact probably get a useful answer.
But it’s usually impossible to ask a question of everyone (a population), so instead, you usually pick a smaller group (a sample) that is as representative as possible of the broader population. We can’t ask every person in France which is better, red wine or white wine, because it would take forever and cost a fortune. So instead, we want to ask a *representative sample* of French people that question, as a way to gauge the overall population’s view.
Political polls sample 3 different types of populations: all adults (the largest and least representative population), registered voters (a smaller and somewhat more representative population), and likely voters (the smallest and most accurate population). If you click on that first link I provided, you’ll see these as. A, RV, and LV beside the poll.
**Part 2: How polls are designed**
In order to be representative, polls have to eliminate as many biases as possible, and sample as a representative a population as possible. In practice, this means polling anywhere from 500 to 5000 people, depending on how well-funded the pollster is, and how hot they are about the reliability of the answer. A quick “who won last night’s debate” poll might only ask 100 people, while I imagine private polls being run by the Democratic party right now about whether or not Biden should step down are probably asking 3000 people or more.
It also matters who is running the poll. A dedicated well-funded outfit run by statisticians interested in accurately gauging the state of a race will produce one type of poll, while a group looking to actively influence the race might run another type of poll. Compare the first poll with these two:
https://drive.google.com/file/d/1yWjppGZ3zxiJqYmJimDLlscGbRldgM95/view
https://survey.mrxsurveys.com/orc/pollingresults/Big-Village-Political-Poll-07.14.24.pdf
And then look up the pollster ratings, and you can tell at a glance that some polls are just dodgier than others.
Here’s a full list of pollsters, with quality ratings based on the historical accuracy of their polls:
https://projects.fivethirtyeight.com/pollster-ratings/
If you look at the link for my example poll:
https://www.ipsos.com/sites/default/files/ct/news/documents/2024-07/Reuters%20Ipsos%20Post%20Trump%20Assasionation%20Attempt%20Poll%20Topline.pdf
you’ll see that it run by Ipsos, who is a solid outfit. They sample 992 registered voters. So it’s a big sample size. Put all that together, and we get a historically sound pollster, running a big poll, and it’s RV. So this is probably a pretty decent result.
I’m also going to compare that to this poll: https://emersoncollegepolling.com/july-2024-swing-state-polls/
**Part 3: HOW the people being polled are contacted**
Historically, pollsters conducted polls in all sorts of ways – going door to door, by mail, and calling people at home. For a long time, these methods were pretty reliable, especially phone surveys. But since ell phones have come out and phone books aren’t a thing, there’s a big divide in how people use their phones – basically, no one under about 40 answers calls from unknown numbers anymore, if at all. So since about 2012, there’s been a gradual shift in how polling works, [as pollsters try to find other methods, particularly online polling](https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/).
The problem is, online polling and phone polling tend to give divergent results.
If we look at the Ipsos poll, which was conducted online, it shows a dead heat. But is it a representative sample of registered voters? Or of people who are willing to take a survey online. Because those aren’t the same thing.
Similarly, if we look at the Emerson poll, it was “administered by contacting respondents’ cell phones via MMS-to-web and landlines via Interactive Voice Response,” and shows Trump up by 6. So the question is, is the second poll actually a representative sample of registered voters? Or is it a representative sample of people who are willing to answer their phone? Because those are very much not identical populations. Once upon a time they might have been, but that time is long past.
Pollsters offset these issues in two ways: 1) by running polls of both types, and 2) by running a LOT of polls, and then averaging them all. It kinda/sorta works for races that aren’t close, but it’s not super helpful in very tight races.
**Part 4: How polls are used**
And then there is also the problem of, what is the poll being used for. **National** polling is predictive of the national popular vote, but that doesn’t actually determine any electoral outcomes. Those happen at the state level, because state popular vote determines how electoral votes are cast, and obviously House and Senate races happen at the state level as well.
The problem is, there’s lots and lots of national polling, because one national poll is fairly easy to run. But there’s very little state polling before about September. For example, if you go back to the list of polls:
https://projects.fivethirtyeight.com/polls/national/
You can see that there have been something on the order of 100 national polls run since July 1. And when you average them all out and offset for quality of pollster, we get an average of Trump being up 3-3.5 points. If that held true until election day, we would expect him to win the popular vote by around 4.5 million votes. Of course, Biden was up 8.8 points this time in 2016 and he only won by 4.5 points, and Hillary was up 3.2 points on this day in 2016 and she won the popular vote by 2.1 points while losing the election, so we can see that polls this far out aren’t really predictive of much.
And the reason is, the paucity of state polling. There have only been 7 polls run in Michigan in the same period. It’s a crucial swing state. We can average those 7 polls and say Trump is up 1.7, but we can’t put a lot of weight on that number.
**Part 5: Conclusion**
So: put all that together, and what you get is this:
1. What polls say is a function of how the poll is written, but it usually means that someone is understood to be leading in a race by a certain number of percentage points.
2. They are updated by running a new poll. Major outfits usually run somewhere between several per week to a couple per month.
3. They are asking whoever they can get to answer the phone or to respond to online surveys. You haven’t been contacted because they’re randomly pulling names and numbers off lists, and your name hasn’t come up.
4. It’s an open question who is actually answering.
Sorry, I know this was long but I hope it helps!
Latest Answers