Reasoning from Evidence: A Call for Education
Rather than choose a personal example of a change in mind, I reflect on instances in which my field, medicine, has apparently changed "its" mind based on changes in evidence. In my experience major reversals in belief (as opposed to simply progressions, or changes in course) typically arise when there are serious flaws in evaluation of evidence or inference leading to the old view, the new view, or both.
To be committed to a view based on facts, and later find the view wrong, either the facts had to be wrong or the interpretation of them had to extend beyond what the facts actually implied. The "facts" can be wrong in a range of settings: academic fraud, selective documentation of research methods, and selective publication of favorable results — among many. But in my experience more often it is the interpretation of the facts that is amiss.
Hormone replacement therapy ("HRT") is a case in point. You may recall HRT was widely hailed as slashing heart disease and dementia risk in women. After all, in observational studies — with large samples --women who took HRT had lower rates of heart disease and Alzheimer's than women who did not.
I was not among those advising patients that HRT had the benefits alleged. Women who took HRT (indeed any preventive medication) differed from those who did not. These differences include characteristics that might be expected to produce the appearance of a protective association, through "confounding." For instance, people who get preventive health measures have better education and higher income — which predict less dementia and better health, irrespective of any actual effect of the treatment. (Efforts to adjust for such factors can never be trusted to capture differences sufficiently.) These disparities made it impossible to infer from such "observational" data alone whether the "true" causal relationship of hormone replacement to brain function and heart events was favorable, neutral, or adverse.
When controlled trials were finally conducted — with random allocation to HRT or placebo providing that the compared groups were actually otherwise similar — HRT was found rather to increase rates of heart-related events and dementia. But the lessons from that experience have not been well learned, and new, similarly flawed "findings" continue to be published — without suitable caveats.
It is tempting to provide a raft of other examples in which a range of errors in reasoning from evidence were present, recognition of which should have curbed enthusiasm for a conclusion, but I will spare the reader.
Stunningly, there is little mandated training in evaluation of evidence and inference in medical school — nor indeed in graduate school in the sciences. (Nor are the medical practice guidelines on which your care is grounded generated by people chosen for this expertise.)
Even available elective coursework is piecemeal. Thus, statistics and probability courses each cover some domains of relevance such as study "power," or distinguishing posterior from a priori probabilities: thus it may be that commonly if a wife who was beaten is murdered, the spouse is the culprit (a posteriori), even if it is uncommon for wife beaters to murder their spouse (a priori). Epidemiology-methods courses address confounding and many species of bias.Yet instruction in logical fallacies, for instance, was absent completely from the available course armamentarium. Each of these domains, and others, are critical to sound reasoning from evidence.
Preventive treatments should mandate a high standard of evidence. But throughout the "real world" decisions are required despite incomplete evidence. At a level of information that should not propel a strong evidence-driven "belief," a decision may nonetheless be called for: How to allocate limited resources; whom to vote for; and whether to launch a program, incentive, or law. Even in these domains, where the luxury of tightly controlled convergent evidence is unattainable — or perhaps especially in these domains — understanding which evidence has what implications remains key, and may propel better decisions, and restrain unintended consequences.
The haphazard approach to training in reasoning from evidence in our society is, in its way, astounding — and merits a call to action. Better reasoning from evidence is substantially teachable but seldom directly taught, much less required. It should be central to curriculum requirements, at both graduate (advanced) and undergraduate (basic) levels — if not sooner.
After all, sound reasoning from evidence is (or ought to be) fundamental for persons in each arena in which decisions should be rendered, or inferences drawn, on evidence: not just doctors and scientists, but journalists, policy makers — and indeed citizens, whose determinations affect not solely their own lives but others', each time they parent, serve on juries, and vote.
