What is p-hacking, how does it work, and what does it mean for science in general?

Question

I’ve been reading about how some studies that we assume provide us with information about the world actually don’t teach us anything because of statistical manipulation that makes them look more relevant than they are. How does this work?

In: Other

Answer 1

p-value in science refers to how likely (probability) your conclusion is wrong, based on a set of data. Most science field have generally accepted 5% is the threshold for something worthy to publish.  Here is an example of p-value. Let’s say you want to prove me a certain coin is unfair at flipping, say 100% head. You then flip it for 5 times and it’s all head. Then your have a claim with p<0.05. Because the coin can still be fair (50-50 head to tail), but you’re just lucky to get 5 head (probability is 1/2^5 = 0.031). So your chance of being wrong is 3.1% (p=0.031). 

P-hacking refers to people do seemingly legit things to bring p under 0.05. If we keep using the example above, say you flip the coin (which is actually fair) five times every day and record it. And then only show me the recording of the day you get 5 heads and claims it’s a 100-0 coin with p<0.05. 

You might think this is dumb, but in reality it can be really hard to detect. Some time the author even commits it subconsciously. In a lots of cases experiments are conducted in series with continuous improvement of the details. Say you’re testing a new drug in mouse, and failed, and then think: oh hey this batch of mice seems a bit skinnier the usual, let try again with actually well-fed mice. It failed again, and you think, oh maybe I should give the drug with food, so they don’t get stressed by force feeding the drug. And then and then… And finally one day worked. You report the last run, and in methods section you detail all the seemingly unsuspicious things you do about administrating the drug (keep them fed, don’t stress them, ask for God’s forgiveness etc.) Even though in reality none of these actually matters for the drugs efficacy. You’re p-hacking. 

Note for math: for the coin flipping example, the math only works if we are living in a world where a coin can only be 100-0 or 50-50, nothing in between.