Shouldn’t there be a neutral group that gets to decide what data is tossed out before arriving at conclusions? Case in point: Ansel Keys Seven Nations Study (he tossed out data from countries that didn’t support the conclusion that saturated fats were bad). I mean, OK if funding sources, and pre-stated hypothesis creates bias, shouldn’t the publication process vet the data better?
In:
A person reading the paper can evaluate whether excluding certain data is valid or not. But generally, poor data is excluded. If you have 10 things saying A but the 10 things are biased or measured wrong, and 1 thing saying B which was bias-free and measured well, then you trust the 1 thing rather than the 10 things just because the evidence supporting A is poorer.
https://www.truehealthinitiative.org/wp-content/uploads/2017/07/SCS-White-Paper.THI_.8-1-17.pdf
In this white paper, pages 5-8 outline rationales for excluding certain data sets which include:
> While it is true that selection of cohorts was non-random, selection was based on
practicality and dietary variation.
1–4 Allegations suggesting that SCS researchers
chose locations where they already knew the outcomes are clearly false based on
review of primary source material, the relevant timelines, and direct questioning
of investigators.
and
> Critics attest that France was excluded because researchers were aware of the
“French Paradox”, but this concept represents an anachronism, since the
information and associations being used today to allege bias were simply not
available at the time the SCS was being designed and implemented.
That is the purpose of *peer review*. The neutral party is *every other scientist in that field of study*. Respectable, scholarly papers always include sections detailing their experiments and methodology along with the experimental results. This is to ensure that anyone reading the paper knows *exactly* how they gathered their data and can recreate the experiment precisely. If they get the same results, that indicates that the paper’s authors have a well-designed experiment. That also means that readers can gather the same data without being worried about anything getting left out or the bias of the experimenters. Then, the second experimenters can draw their own conclusions based on that data.
The paper will also almost always have a section discussing potential flaws in the data or biases, examining them in detail, and explaining how the author accounts for them. This may include a line that explains that they got bad data and discarded it, and *why* they think they got bad data and *why* they chose to discard it.
There is no single publication process because there is no single publisher of scientific research. In general, papers do have to go through a review process with that journal where qualified scientists on the review board for that journal read through the paper and consider the experimental design, results, and conclusion. The paper is also passed along to other scientists around the world as part of prepublication for additional review. Objections are noted and passed along. If the experiment isn’t too burdensome to perform, some scientists may repeat the experiment right then and check to see if they get the same data. Even if someone doesn’t do the experiment, they can read through the experimental design and reference other papers with similar experiments to make sure it aligns with what is expected.
Once the paper has gone through all of that, it is officially published. Even after publication, the paper is often reviewed and the experiments repeated. If other scientists show that the paper is flawed it can be retracted and the journal will remove it.
Not all journals have such a rigorous prepublication process, and consequently not all scientific journals have the same reputation for publishing good, scientifically sound papers. Consider the difference between an article published in the New York Times or Washington Post versus an article published on Buzzfeed or the National Enquirer. Can Buzzfeed or National Enquirer publish articles that are factually accurate? Yes. Can the NYT or WP screw up? Yes. But which would you *generally* trust to be correct? NYT and WP have a much more rigorous vetting process.
Part of *your* job as a consumer of information is to do your due diligence and vet the paper *yourself*, which means understanding at least in part the scientific process. It means paying attention to where a paper is published and knowing how reputable that journal is. You have to be able to understand how to read a scientific paper so that even if you don’t understand the concepts themselves you can at least discern the methodology and decide if it’s sound.
Latest Answers