charles_seife's picture
Professor of Journalism, New York University; Former Journalist, Science Magazine; Author, Hawking Hawking
The Conquest Of Human Scale

It was just one among dozens and dozens of revelations about the National Security Agency, barely enough to cause a stir in the papers. Yet it is a herald of a new era.

In May, 2014, journalists revealed that the NSA was recording and archiving every single cell phone conversation that took place in the Bahamas. Now, the Bahamas isn't a very big place, with only a few hundred thousand people on the islands at any given time. Nor is it very often the source of international headlines. So, at first glance, the NSA's achievemment might not look like much. But in capturing—and storing—all of the Bahamas' cell phone conversations in real time, the NSA has managed to transform a significant proportion of the day-to-day interactions of a society into data: into information that can be analyzed and transformed and correlated and used to understand the people who produced that information. This was something unthinkable even a decade or two ago, yet, almost unnoticed, the processing power of computers, the scale of their memory banks, and the cheapness and ubiquity of their sensors, are making civilization-scale data gathering almost routine.

NSA's collection program in the Bahamas—codenamed SOMALGET—was a small part of a larger operation, which, itself, is just a tiny fraction of the NSA's global surveillance system. Whistleblowers and leaked documents have revealed that NSA has been gathering and storing e-mails, phone calls, and other records on a global scale, and, apparently to capture entire nations' communications outputs and store it for later study. And the NSA isn't the only entity with such ambition. Other agencies and companies around the world have been collecting and creating datasets that capture one entire facet of the behavior of millions or even billions of people. The city of New York can now analyze each taxi ride taken in the five boroughs over the past several years. Google has stored every single character that anyone has entered into its search engine for more than a decade. It's all in there, taking up much less room in memory than you might think.

A medical researcher can now download and analyze all the drug prescriptions filled in the United Kingdom, an epidemiologist can view all the deaths recorded in the United States, a civil engineer can view all airline flights taken anywhere in the world at any time in recent history. Personal genomics companies are now performing cut-rate genomic analysis of more than one million customers; at this point, it's just cost and desire that prevents us from capturing the entire genome of every individual on Earth. And as digital cameras, microphones, and other sensors are woven into every aspect of the fabric of our society, we are not far from the point from when we're able to capture the movements and utterances of every single macroscopic creature in the places we inhabit.

Pretty much anything that can be digitized or digitally collected, and numbers in the billions or trillions or quadrillions, can now be archived and analyzed. All our communications, all our purchases, our travels and our daily routines are all now, to at least some degree, sitting on banks of computer memory. We no longer have to guess, to sample, to model; it's all there for the taking. It's only as this data begins to shine light into every corner of our society that we will recognize how much of our existence has been in darkness—or how different life will be in a world without shade.