ELIZA'S WORLD

Conversation : CULTURE

ELIZA'S WORLD

Nicholas G. Carr [4.3.08]

What is the compelling urgency of the machine that it can so intrude itself into the very stuff out of which man builds his world?

JOSEPH WEIZENBAUM

1923 – 2008

A former executive editor of the Harvard Business Review, NICHOLAS CARR writes regularly for the The Guardian as well as his blog, Rough Type. He is the author of the recently published The Big Switch: Rewiring the World, from Edison to Google, examines the rise of "cloud computing" and its implications for business, media and society.

Nicholas Carr's Edge Bio Page

THE REALITY CLUB: Jaron Lanier, Kevin Kelly

 

ELIZA'S WORLD

What is the compelling urgency of the machine that it can so intrude itself into the very stuff out of which man builds his world?

— Joseph Weizenbaum

Somehow I managed to miss, until just a few days ago, the news that Joseph Weizenbaum had died. He died of cancer on March 5, in his native Germany, at the age of 85. Coincidentally, I was in Germany that same day, giving a talk at the CeBIT technology show, and — strange but true — one of the books I had taken along on the trip was Weizenbaum's Computer Power and Human Reason.

Born in 1923, Weizenbaum left Germany with his family in 1936, to escape the Nazis, and came to America. After earning a degree in mathematics and working on programming some of the earliest mainframes, he spent most of his career as a professor of computer science at MIT. He became — to his chagrin — something of a celebrity in the 1960s when he wrote the Eliza software program, an early attempt at using a computer to simulate a person. Eliza was designed to mimic the conversational style of a psychotherapist, and many people who used the program found the conversations so realistic that they were convinced that Eliza had a capacity for empathy.

The reaction to Eliza startled Weizenbaum, and after much soul-searching he became, as John Markoff wrote in his New York Times obituary, a "heretic" in the computer-science world, raising uncomfortable questions about man's growing dependence on computers. Computer Power and Human Reason, published in 1976, remains one of the best books ever written about computing and its human implications. It's dated in some its details, but its messages seem as relevant, and as troubling, as ever. Weizenbaum argued, essentially, that computers impose a mechanistic point of view on their users — on us — and that that perspective can all too easily crowd out other, possibly more human, perspectives.

The influence of computers is hard to resist and even harder to escape, wrote Weizenbaum:

The computer becomes an indispensable component of any structure once it is so thoroughly integrated with the structure, so enmeshed in various vital substructures, that it can no longer be factored out without fatally impairing the whole structure. That is virtually a tautology. The utility of this tautology is that it can reawaken us to the possibility that some human actions, e.g., the introduction of computers into some complex human activities, may constitute an irreversible commitment. . . . The computer was not a prerequisite to the survival of modern society in the post-war period and beyond; its enthusiastic, uncritical embrace by the most "progressive" elements of American government, business, and industry quickly made it a resource essential to society's survival in the form that the computer itself had been instrumental in shaping.

The machine's influence shapes not only society's structures but the more intimate structures of the self. Under the sway of the ubiquitous, "indispensable" computer, we begin to take on its characteristics, to see the world, and ourselves, in the computer's (and its programmers') terms. We become ever further removed from the "direct experience" of nature, from the signals sent by our senses, and ever more encased in the self-contained world delineated and mediated by technology. It is, cautioned Weizenbaum, a perilous transformation:

Science and technology are sustained by their translations into power and control. To the extent that computers and computation may be counted as part of science and technology, they feed at the same table. The extreme phenomenon of the compulsive programmer teaches us that computers have the power to sustain megalomaniac fantasies. But the power of the computer is merely an extreme version of a power that is inherent in all self-validating systems of thought. Perhaps we are beginning to understand that the abstract systems — the games computer people can generate in their infinite freedom from the constraints that delimit the dreams of workers in the real world — may fail catastrophically when their rules are applied in earnest. We must also learn that the same danger is inherent in other magical systems that are equally detached from authentic human experience, and particularly in those sciences that insist they can capture the whole man in their abstract skeletal frameworks.

His own invention, Eliza, revealed to Weizenbaum the ease with which we will embrace a fabricated world. He spent the rest of his life trying to warn us away from the seductions of Eliza and her many friends. The quest may have been quixotic, but there was something heroic about it too.

Reality Club Discussion

jaron_lanier's picture
Jaron Lanier
Computer Scientist; Musician; Author, Who Owns The Future?

 

We have lost a lion of Computer Science. Joseph Weizenbaum’s life is proof that someone can be an absolute alpha-geek and a compassionate, soulful person at the same time. He displayed innovative courage in recognizing the seductive dangers of computation.

History will remember Weizenbaum as the clearest thinker about the philosophy of computation. A metaphysical confrontation dominated his interactions with the non-human centered mainstream. There were endless arguments about whether people were special in ways that cybernetic artifacts could never be. The mainstream preferred to sprinkle the magic dust of specialness on the “instruments,” as Weizenbaum put it, instead of people.

But there was a less metaphysical side of Weizenbaum’s thinking that is urgently applicable to the most pressing problems we all face right now. He warned that if you believe in computers too much, you lose touch with reality. That’s the real danger of the magic dust so liberally sprinkled by the mainstream. We pass this fallacy from the lab out into the world. This is what apparently happened to Wall Street traders in fomenting a series of massive financial failures. Computers can be used rather too easily to improve the efficiency with which we lie to ourselves. This is the side of Weizenbaum that I wish was better known.

We wouldn’t let a student become a professional medical researcher without learning about double blind experiments, control groups, placebos, the replication of results, and so on. Why is computer science given a unique pass that allows us to be soft on ourselves? Every computer science student should be trained in Weizenbaumian skepticism, and should try to pass that precious discipline along to the users of our inventions.

Weizenbaum’s legacy includes an unofficial minority school in computer science that has remained human-centered. A few of the other members, in my opinion, are David Gelernter, Ted Nelson, Terry Winograd, Alan Kay, and Ben Schneiderman.

Everything about computers has become associated with youth. Turing’s abstractions have been woven into a theater in which we can enjoy fantasies of eternal youth. We are fascinated by wiz kids and the latest young billionaires in Silicon Valley. We fantasize that we will be uploaded when the singularity arrives in order to become immortal, and so on. But when we look away from the stage for a moment, we realize that we computer scientists are ultimately people. We die.

kevin_kelly's picture
Kevin Kelly
Senior Maverick, Wired; Author, What Technology Wants and The Inevitable

 

The Machine That Made Us

Computer scientist Joseph Weizenbaum recently passed away at the age of 85. Weizenbaum invented the famous Eliza chat bot forty years ago. Amazingly this pseudo-AI still has the power to both amusing and confuse us. But later in life Weizenbaum became a critic of artificial intelligence. He was primarily concerned about the pervasive conquest of our culture by the computational metaphor — the idea that everything interesting is computation — and worried that in trying to make thinking machines, we would become machines ourselves. Weizenbaum's death has prompted a review of his ideas set out in his book "Computer Power and Human Reason".

On the Edge Nick Carr says this book "remains one of the best books ever written about computing and its human implications. It's dated in some its details, but its messages seem as relevant, and as troubling, as ever. Weizenbaum argued, essentially, that computers impose a mechanistic point of view on their users — on us — and that that perspective can all too easily crowd out other, possibly more human, perspectives." He highlights one passage worth inspecting.

The computer becomes an indispensable component of any structure once it is so thoroughly integrated with the structure, so enmeshed in various vital substructures, that it can no longer be factored out without fatally impairing the whole structure. That is virtually a tautology. The utility of this tautology is that it can reawaken us to the possibility that some human actions, e.g., the introduction of computers into some complex human activities, may constitute an irreversible commitment. . . . The computer was not a prerequisite to the survival of modern society in the post-war period and beyond; its enthusiastic, uncritical embrace by the most "progressive" elements of American government, business, and industry quickly made it a resource essential to society's survival in the form that the computer itself had been instrumental in shaping.

That's an elegant summary of a common worry: we are letting the Machine take over, and taking us over in the process.

Reading this worry, I was reminded of a new BBC program called "The Machine That Made Us." This video series celebrates not the computer but the other machine that made us — the printing press. It's a four part investigation into the role that printing has played in our culture. And it suggested to me that everything that Weizenbaum said about AI might be said about printing.

So I did a search-and-replace in Weizenbaum's text. I replaced "computer" with this other, older technology, "printing."

Printing becomes an indispensable component of any structure once it is so thoroughly integrated with the structure, so enmeshed in various vital substructures, that it can no longer be factored out without fatally impairing the whole structure. That is virtually a tautology. The utility of this tautology is that it can reawaken us to the possibility that some human actions, e.g., the introduction of printing into some complex human activities, may constitute an irreversible commitment. . . . Printing was not a prerequisite to the survival of modern society; its enthusiastic, uncritical embrace by the most "progressive" elements of government, business, and industry quickly made it a resource essential to society's survival in the form that the printing itself had been instrumental in shaping.

Stated this way its clear that printing is pretty vital and foundational, and it is. I could have done the same replacement with the technologies of "writing" or "the alphabet" — both equally transformative and essential to our society.

Printing, writing, and the alphabet did in fact bend the culture to favor themselves. They also made themselves so indispensable that we cannot imagine culture and society without them. Who would deny that our culture is unrecognizable without writing? And, as Weizenbaum indicated, the new embedded technology tends to displace the former mindset. Orality is gone, and our bookish culture is often at odds with oral cultures.

Weizenbaum's chief worry seems to be that we would become dependent on this new technology, and because it has its own agenda and self-reinforcement, it will therefore change us away from ourselves (whatever that may be).

All these are true. But as this exercise makes clear, we've gone through these kind of self-augmentating transitions several times before, and I believe come out better for it. Literacy and printing has improved us, even though we left something behind.

Weizenbaum (and probably Carr) would have been one of those smart, well-meaning elder figures in ancient times preaching against the coming horrors of printing and books. They would highlight the loss or orality, and the way these new-fangled auxiliary technologies demean humanity. We have our own memories, people: use them! They would have been in good company, since even Plato lamented the same.

There may indeed be reasons to worry about AI, but the fact that AI and computers tend to be pervasive, indispensable, foundational, self-reinforcing, and irreversible are not reasons alone to worry. Rather, if the past history of printing and writing is any indication, they are reasons to celebrate. With the advent of ubiquitous computation we are about to undergo another overhaul of our identity.