2017 : WHAT SCIENTIFIC TERM OR CONCEPT OUGHT TO BE MORE WIDELY KNOWN?
Professor of Journalism, New York University; Former Journalist, Science Magazine; Author, Hawking Hawking
The Texas Sharpshooter
A city-slicker statistician was driving through the backwoods of rural Texas, so the story goes, when she slammed on the brakes. There, right by the side of the road, was a barn that bore witness to a nigh-impossible feat of marksmanship. The barn was covered with hundreds of neat little white bullseyes, each of which was perforated with a single bullet hole in the dead center.
The incredulous statistician got out of the car and examined the barn, muttering about Gaussian distributions and probability density functions. She didn't even notice that Old Joe had sidled up to her, ancient Winchester slung over his shoulder.
Joe cleared his throat. With a start, the statistician looked at Joe with amazement. "Four hundred and twelve targets, every single one of which is hit in the center with less than 2 percent deviation every time... the odds against that are astronomical! You must be the most accurate rifleman in history. How do you do it?"
Without a word, Joe walked ten paces away from the barn, spun round, raised his rifle, and fired. A slug thunked dully into the wood siding. Joe casually pulled a piece of chalk out of his overalls as he walked back toward the barn, and, after finding the hole where his bullet had just struck, drew a neat little bullseye around it.
There are far too many scientists who have adopted the Texas Sharpshooter's methods, and we're beginning to feel the effects. For example, there's a drug on the market—just approved—to treat Duchenne muscular dystrophy, an incurable disease. Too bad it doesn't seem to work.
The drug, eteplirsen, received a lot of fanfare when researchers announced that the drug had hit two clinical bullseyes: It increased the amount of a certain protein in patients' muscle fibers, and patients did better on a certain measure known as the six-minute walk test (6MWT). The drug was effective! Or so it seemed, if you didn't know that the 6MWT bullseye was painted on the wall well after the study was underway. Another bullseye that was drawn before the study started, the number of certain white blood cells in muscle tissue, was hastily erased. That's almost certainly the sign of a missed target.
Looking at all the data and all the scientists' prognostications makes it fairly clear that the drug didn't behave the way researchers hoped. Eteplirsen's effectiveness is highly questionable, to put it mildly. Yet it was trumpeted as a big breakthrough and approved by the FDA. Patients can now buy a year's supply for about $300,000.
Texas-style sharpshooting—moving the goalposts and cherry-picking data so that results seem significant and important when they're not—is extremely common; check out any clinical trials registry and you'll see just how frequently endpoints are tinkered with. It goes almost without saying that a good number of these changes effectively turn the sow's ear of a negative or ambiguous result into the silk purse of a scientific finding worthy of publication. No wonder then that many branches of science are mired in replicability crises; there's no replicating a finding that's the result of a bullseye changing positions instead of reflecting nature's laws.
The Texas Sharpshooter problem should be more widely known not just by scientists, so that we can move toward a world with more transparency about changing protocols and unfixed endpoints, but also by the public. Maybe, just maybe, that will make us a little less impressed by the never-ending procession of supposed scientific marksmen—researchers whose results are little more permanent than chalk marks on the side of a barn.
