2017 : WHAT SCIENTIFIC TERM OR CONCEPT OUGHT TO BE MORE WIDELY KNOWN?

A few years ago, a reporter at a leading financial daily called with an intriguing question: “We’re doing a story about decision making, and asking researchers whether they follow their own advice.” I must have chuckled because she continued: “I just talked with one man who said, ‘For heaven’s sake, you don’t think I actually use this stuff, do you? My decisions are as bad as everyone else’s!’”

I suspect my colleague was being ironic, or perhaps tweaking the reporter. I gave a few examples I have found useful—sunk cost fallacy, regression toward the mean, and more—but then focused on one concept that everyone should understand: Type I and Type II errors, or false positives and false negatives.

The basic idea is straightforward: We should want to go ahead with a course of action when it will succeed and refrain when it will fail; accept a hypothesis when it is true and reject when it is false; convict a defendant who is guilty and free one who is innocent. Naturally, we spend time and effort to improve the likelihood of making the right decision. But since we cannot know for sure we will be correct, we also have to consider the possibility of error. Type I and Type II thinking forces us to identify the ways we can err, and to ask which error we prefer.

In scientific research, we want to accept new results only when they are demonstrably correct. We wish to avoid accepting a false result, or minimize the chance of a Type I error, even if that means we commit a Type II error and fail to accept results that turn out to be true. That’s why we insist that claims are supported by evidence that is statistically significant, often set (by convention) as the probability an observation could be due to random effects is less than one in twenty (p < .05) or less than one in a hundred (p < .01). (How we know the probability distribution in the first place leads us into the debate between frequentists and Bayesians, an exceedingly interesting question but beyond the scope of this note.)

Similarly, in criminal trials we want to convict a defendant only when we are very certain of guilt. Here again a Type II error is far preferable to Type I error, a view expressed in Blackstone’s law, which says it is better to let ten guilty men go free than to convict an innocent man, since the severity of a false positive is not only great but perhaps irreversible. The preference for Type II error is reflected in cornerstones of Anglo-Saxon law such as the presumption of innocence and burden of proof resting with the prosecution.

In other settings the greater danger is to commit a Type II error. Consider competition in business, where rival firms seek payoffs (revenues and profits) that accrue disproportionately to a few, or perhaps are winner-take-all. Although bold action may not lead to success, inaction almost inevitably leads to failure—because there is a high likelihood that some rival will act and succeed. Hence the dictum made famous by Intel’s Andy Grove: Only the paranoid survive. Grove did not say that taking risky action predictably leads to success; rather, he observed that in situations of intense competition, those who survive will have taken actions that involved high risk. They will have understood when it comes to technological breakthroughs or launching of new products, it is better to act and fail (Type I error) than fail to act (Type II error).  

To summarize, a first point is that we should not only consider the outcome we desire, but also the errors we wish to avoid. A corollary is that different kinds of decisions favor one or the other. A next point is that for some kinds of decision, our preference may shift over time. A young adult who is considering marriage may prefer a Type II error, finding it more prudent to wait rather than to marry too soon. As the years pass, he or she may be more willing to marry even if the circumstances do not seem perfect—a Type I error—rather than never marry—a Type II error.

There is a final point, as well: Discussion of Type I and Type II errors can reveal different preferences among parties involved in a decision. To illustrate, expeditions to the summit of Mount Everest are very risky in the best of times. Of course, climbers want to push on if they can reach the summit safely, and will want to turn back if pressing on results in death. That’s the easy part. More important is to discuss in advance preferences for error: Would they rather turn back when they could have made it (Type II) or keep going and die in the effort (Type I)? If the choice seems obvious to you, it may be because you have not invested the time and money to get into the position of fulfilling a dream—nor in a state of exhaustion and oxygen deprivation when the moment of decision arrives. As past Everest expeditions have shown, sometimes tragically, members of a team—leaders, guides, and climbers—should do more than focus on the outcomes they most desire. They should also discuss the consequences of error.