Commentary on a Stanford University Study

Abstract: The Stanford University History Education Group issued a non-peer reviewed study in October 2017 investigating how individuals determine the credibility of digital information.  This commentary challenges the conclusions of their study, Lateral Reading, and notes the irony of their findings.  Two of the participant groups that scrutinized the study texts reached very different conclusions from the one group that ignored the text and sought opinions on the Internet.  This study’s conclusions are tainted by the obvious and admitted bias of its authors.

Investigators Sam Wineburg and Sarah McGrew from the Stanford University History Education Group sought to investigate how people determine the credibility of digital information.  They released their findings on their own Stanford University website as a “Working Paper,” titled, Lateral Reading: reading less and learning more when evaluating digital information. [1] This was not a peer-reviewed publication.

Participants

The study sampled 45 individuals: 10 Ph.D. historians, 10 professional fact checkers, and 25 Stanford University undergraduates.  The historians were doctorate history faculty recruited from 4-year colleges in California or Washington state.  The “fact checkers” were individuals “employed at news and political fact-checking organizations.”  The students were recruited and self-selected via ad posters on the Stanford campus.  These participants were asked to evaluate the “trustworthiness” of the online statements addressing bullying in schools, minimum wage policy and teacher tenure.  This commentary will focus on the  School Bullying policy statements found on the websites of two professional pediatric organizations, the American College of Pediatricians (the College) [2] and the American Academy of Pediatrics (the Academy). [3]

Participant’s approach

The participants were given up to 5 minutes to evaluate each statement on the organization’s website.  The investigators recorded the participant’s online actions.  The three participant groups approached the task very differently.  The students and historians tended to evaluate the statement based upon its content and context.  Historians spent “almost three times as long” viewing the statement page as the fact checkers, and most of the students did the same.  The fact checkers, however, tended to quickly go off site to discover what the Internet had to say about the organization, often before reading the entire bullying statement.  Consequently, in their Internet  searching, the fact checkers were influenced by non-objective sources such as  the  Southern Poverty Law Center and American Civil Liberties Union, or the open-sourced, Internet encyclopedia Wikipedia.  The students and historian were primarily focused on the text and references of the statement, as well as other papers on the respective websites.

In reality, the so-called “fact checkers” were only “opinion checkers,” failing to focus on the text of the statements and, instead, relying heavily upon each professional organization’s Internet reputation, which, of course, is not necessarily rooted in fact.  Some of these checkers spent only “8 seconds” on the statement page before going off site, with most making their final decision within 2 minutes.  Surprisingly, the study authors praised this hasty approach claiming, “Fact checkers arrived at more warranted conclusions in a fraction of the time.” Apparently, the veracity of the content of the statement and the validity of the scientific references upon which it was based are of trivial concern to the authors when assessing the trustworthiness of digital information.

Results

The reported results of the study were not surprising from an objective perspective, but were apparently disappointing to the study designers, who were openly biased against the College statement.  Those participants who spent the most time reading the actual statements (historians and students) concluded that the College statement was the most professional and reliable.  Fully 80 % of students chose the College to be more, or at least equally, trustworthy when compared with the Academy statement, with 50% of the historians responding the same.  The “fact checkers” were united in preferring the Academy statement after reading negative comments about the College (not its bullying statement) on the Internet.  These participants fell victim to the ad hominem rhetoric on the Internet, where the messenger (the College) is maligned when the message (College statements) is irrefutable.  This is one of the many risks to truth-seekers who look solely to the Internet for analysis.  The real fact checkers were the individuals who actually spent time investigating the content of the statements (and their references) and made objective judgments that were untainted by the apparent biased opinions of others.

Investigator bias?

Was this “study” intended as an attempt to discredit the College and were the investigators biased?  The investigators claim that the students and historian reached unwarranted conclusions by ranking the College statement equal to or more reliable than the Academy.

They further claim that the College bullying statement is unscientific and anti-gay for its recommendation that “no group of students be singled out for special treatment.”  Yet, the College’s position is defended by referenced research, and even further confirmed by recent research demonstrating that the two most effective character education programs (Olweus and Steps to Respect) promote a culture of kindness in schools without singling out particular groups of students. [4]

Could it be that they have abandoned science to blindly accept the ad hominem attacks found on the Internet?  While it is reasonable to point out to readers that the College is much smaller, younger, and less lucrative than the Academy, the investigators were more emotional in their further descriptions, calling the College “a splinter group,” “virulently anti-gay,” and “incendiary.”  It would appear that the investigators were frustrated with the objectivity of the students and historians, since this approach led them to a very different conclusion than they preferred.

Fake?

In their opening remarks, the investigators identify the problem of “fake news” on the Internet and its allure to readers.  Clearly, there is nothing “fake” about the professional organization of the American College of Pediatricians whose members are Board-certified pediatricians nationwide who subscribe to its common mission: “To enable all children to reach their optimal physical and emotional health and well-being.” Likewise, there is nothing “fake” about the research-referenced statement issued by the College on the topic of bullying at school.  The students and historians were obviously convinced of the professionalism and trustworthiness of this statement which opens with, “Bullying on the school campus is never acceptable in a society where all individuals are entitled to respectful treatment.”

Conclusions

In an objective profession, such as medicine, facts are not a matter of opinion or popularity.  Statements must be evaluated from a factual, scientific perspective in order to have value in the field.  This study and its authors wrongly accuse the students of naivety by stating, “Yet, even the most critical thinkers are susceptible to cognitive biases that steer them in the wrong direction.”  We note instead that the students (and historians) displayed more cognitive and critical assessment skills than the fact-checkers who were persuaded by less than reliable Internet ramblings.  These fact-checkers were in reality only “opinion-checkers,” opening themselves up to the influence of unsupported statements published by opinionated people on the Internet.

True fact-checking would involve scrutinizing the text and the references that support the text. That’s what the students and historians mostly did. The fact-checkers were more likely to have been influenced by the all-too-common ad hominem attacks found on the Internet, and perhaps persuaded by the views of their professional associations with “news and political organizations.”

While we concede that the Internet is replete with questionable information, this study did not demonstrate this with respect to the two bullying statements.  In fact, the students and historians are to be commended for their honest quest to fairly and fully evaluate the statements presented them.  On the contrary, the fact-checkers failed to check the facts (statements and references) and, instead, (as would be predicted by their career training) relied upon the opinion of others rather than conducting their own evaluation of the evidence.

We, therefore, are justly critical of this study’s investigators for revering the participant group (fact-checkers) who reached the least science-based conclusions, while condemning the students and historians who arrived at objective conclusions based on analysis of the statements.  Stanford’s reputation for exacting science is sullied by this unscientific report.

November 13, 2017
Primary author: Den A. Trumbull, MD, FCP

References: 

[1] Wineburg S & McGrew S. Lateral Reading: Reading Less and Learning More When Evaluating Digital Information (October 6, 2017). Stanford History Education Group Working Paper No. 2017-A1. Available at SSRN: https://ssrn.com/abstract=3048994

[2] The American College of Pediatricians.  Accessed at: https://www.acpeds.org/the-college-speaks/position-statements/societal-issues/bullying-at-school-never-acceptable.

[3] The American Academy of Pediatrics.  Accessed at: https://www.aap.org/en-us/about-the-aap/aap-press-room/Pages/Stigma-At-the-Root-of-Ostracism-and-Bullying.aspx.

[4] Lickona T. Bullying in Schools: A Critical Problem in Need of a New Solution. Psychology and Education Journal. 2016. Vol 35. Number 1-2.  Found at: http://www.psychologyandeducation.net/pae/2016/04/26/bullying-schools-critical-problem-need-new-approach-thomas-lickona/