The world of Research is full of PhDs making claims, some claims are backed by credibility, some are overhyped and sensationalized. As a new researcher, I sometimes find it very hard to tell apart a good study from a bad one. Two of my recent favorite videos that shed light on this issue are “How to Detect Baloney the Carl Sagan Way” by Micheal Shermer (Big Think), and “Top 4 Tips to Spot Bad Science Reporting” by Myles Bess (Above the Noise)
Michael Shermer teaches a critical thinking course to freshman at Colorado University called Skepticism 101, which is basically preparing students to call out BS when they see it. He uses his teaching experiences from this class, combined with ideas by Carl Segan from his book ‘The Demon-Haunted World: Science as a Candle in the Dark’ in a chapter titled 'The Fine Art of Baloney Detection' - to share ideas on how to separate BS from the truth.
Shermer suggests following questions we must ask ourselves in order to detect BS
- SOURCE
How reliable is the Source of the claim?
Who said that? Click-bait website or credible journal?
- EVIDENCE
What’s the evidence for the claim?
What’s the quality of the evidence?
Where does the evidence come from?
- DISPROVE
What do people who don’t agree with this claim, think?
Has anyone tried to disprove the theory by running experiments?
Karl Popper - Principle of Falsification “For something to be scientific it must be be able to be proven false”
- CLAIMANT
Does the claimant play by the rules of science/ by the rules of their field?
Do the claimant’s personal beliefs affect the claim?
- TEST
Can the theory/claim be tested by others?
Myles Bess (host of the series ‘Above the Noise’ airing on KQED - an NPR and PBS affiliate in San Francisco, CA), suggests using the acronym GLAD to call out BS in Science Reporting.
GLAD stands for
- Get past the clickbait
Eye catching, sensationalized phrases like "revolutionary" "Life altering"
- Look out for crazy claims
“get rich now” “lose 50 lbs in 2 weeks”
Or in educational research context “Studies say digital educational games make children 10x smarter”
- Analyze sources
How credible is the source?
What’s the reputation of the source?
- Determine outside expert opinions
Experts not involved in the study - what do they think?
Interpreting these questions in terms of Academia, here is my own funny acronym version of the PhD BS detection kit : PHDBS
P- Peer Review - Is the research reviewed and validated through the feedback of peers?
H- Hypothesis - Is it tested and proved by evidence?
D- Define the scope of Research - Is it taking knowledge further or just paraphrasing past research? What else can be studied? What interesting questions does this open? Is it scalable?
B- Body of Work in this field - What does previous research say?
S- Sources of Information - Are the sources of information credible?
Originally published at World Campus GA Fellows Blog http://sites.psu.edu/wcfellows/2018/08/30/phd-bs-detection-kit/
Sources :
How to Detect Baloney the Carl Sagan Way” by Micheal Shermer (Big Think), and “Top 4 Tips to Spot Bad Science Reporting”by Myles Bess (Above the Noise)
Greater Good: 10 Questions to ask about Scientific Studies:
Forbes: 10 questions to distinguish real from fake science:
