Eli5: How do studies and surveys avoid (or control for) self-selection or non-reporting bias?


If we are trying to collect data about a population, we usually want to randomly select participants to control for other factors. However, who decides to respond to such a survey may be biased towards those who have time or feel strongly about it.

How do experimenters in psychology (in particular) get around the fact that many people may not choose to respond or participate and could bias their results?

In: 1

It depends on the size of the survey, but there are two common methods:

1. Acknowledge the problem but don’t try to correct it at all; and,
2. Acknowledge the problem, but normalize the numbers based on the data you do have.

“Normalizing” is a complicated process where you adjust the numbers you have so that you can compare them to each other. For example, you might ignore the overall numbers and just compare percentages and say something like “25% of men and 30% of women …”

Being aware of this is a very good thing. So many things can influence the results. Even how the question is asked is makes a difference. The funny example is “Do you still beat your wife?” No matter how you answer it, the result is biased. I recently received one that came through what I thought was a legit source: “Is President Biden doing a bad job?” But using the word “bad” made me rethink that maybe the study was bias to begin with so I ignored it. The last five years has shown us that everyone seems to twist things so much that nothing can be believed. I was shown in a statistics class years ago that even good stats can be made to look bad.