2015 : WHAT DO YOU THINK ABOUT MACHINES THAT THINK?

The first question that comes to our minds, as we think about machines that think, is how much these machines will, eventually, be like us. To us, this comes down to a question of self. Will thinking machines ever evolve to the point of having a sense of self that resembles that of humans? We are (probably) the only species capable of self-consciously thinking about who we are: of not only knowing our selves, but being able to evaluate those selves from a uniquely internal, self-reflective perspective.

Could machines ever develop this kind of self? Might they experience the same evolutionary forces that made human selves adaptive? These include the need to get along with others, to attain status, and to make sure others like us and want to include us in their social groups. As a human being, if you want to succeed at group living it helps to have a self you're motivated to protect and enhance; this is what motivates you to become the kind of person others like, respect, and grant power to, all of which ultimately enhances your chances of surviving long enough to reproduce. Your self is also what allows you to understand that others have selves of their own—a recognition that's required for empathy and cooperation, two prerequisites for social living.

Will machines ever experience these kinds of evolutionary forces? Let's start with the assumption that machines will someday control their own access to resources they need, like electricity and internet bandwidth (rather than having this access controlled by humans), and will be responsible for their own "life" and "death" outcomes (rather than having these outcomes controlled by humans). From there, we can next assume that the machines that survive in this environment will be those that have been programmed to hold at least one basic self-related goal: that of increasing their own efficiency or productivity. This goal would be akin to the human gene's goal of reproducing itself; in both cases, the goal drives behaviors oriented toward boosting fitness, of either the individual possessing the gene, or the machine running the program.

Under these circumstances, machines would be motivated to compete with each other for a limited pool of resources. Those who can form alliances and cooperate—that is, sacrifice their own goals for others, in exchange for future benefits—will be most successful in this competition. In other words, it's possible to imagine a future in which it would be adaptive for machines to become social beings that need to form relationships with other machines, and therefore develop human-like selves.

However, there's a major caveat to this assumption. Any sociality that comes to exist among thinking machines would be qualitatively different from that of humans, for one critical reason: Machines can literally read each other's minds. Unlike humans, machines have no need for the secondary—and often deeply flawed—interpretative form of empathy we rely on. They can directly know the contents of each other's minds. This would make getting along with others a notably different process.

Another way of putting this is to say that, despite the critical importance of our many social connections, in the end, we humans are each fundamentally alone. Any connection we feel with another's mind is metaphorical; we cannot know, for certain, what goes on in someone else's head—at least not in the same way we know our own thoughts. But for machines, this constraint does not exist. Computers can directly access each other's inner "thoughts", and there's no reason that one machine reading another's hardware and software wouldn't come to know, in exactly the self-knowing sense, what it means to be that other machine. Once that happens, each machine is no longer an entirely separate self, in the human sense. At that point—when machines literally share minds—any self they have would necessarily become collective. 

Yes, machines could easily keep track of the sources of various bits of information they obtain, and use this tracking to distinguish between "me" and other machines. But once an individual understands another at the level that a program-reading machine can, the distinction between self and other becomes largely irrelevant. If I download all the contents of your PC to an external hard drive, then plug that into my PC, don't those contents become part of my PC's self? If I establish a permanent connection between our two PCs, such that all information on one is shared with the other, do they continue to be two separate PCs? Or are they, at that point, in fact a single machine? Humans can never obtain the contents of another's mind in this way—despite our best efforts to become close to certain others, there is always a skin-thick boundary separating their minds from ours. But for machines, literal self-expansion is not only possible, but may be the most likely outcome of a pre-programmed goal to increase fitness, in a world where groups of individuals must compete over or share resources.

What this means is that, to the extent that machines come to have selves, they will be so collective that they may instigate a new level of sociality not experienced by humans; perhaps more like the eusociality of ants, whose extreme genetic relatedness makes sacrificing oneself for a family member adaptive. Nonetheless, the fact that any self at all is a possibility in machines is a reason to hope. The self is what allows us to feel empathy, so in machines it could be the thing that forces them to care about us. Self-awareness might motivate machines to protect, or at least not harm, a species that, despite being several orders of magnitude less intelligent than them, shares the thing that makes them care about who they are. Of course, it's questionable whether we can hold out greater hope for the empathy of super-smart machines than what we currently see in many humans. 

Organisms are machines (broadly understood, anyway). Thus, since we as humans are thinking organisms, we are machines that think—we are organic thinking machines, as arguably are a variety of non-human animals. Some machines are artifacts rather than organisms, and some of them arguably think (broadly understood again). Such things are artifactual thinking machines—computers and the like are examples of this.

An important question is whether there is a deep ontological divide between organisms and artifacts generally. But rather than addressing this directly we'd like to ask a different albeit related question: are there deep differences between the kind of thinking organisms exhibit and the thinking artifacts like machines are capable of, between organic and artifactual thinking? This is not a question about the definition of English words like "think, "thinking", "thought", and so on. There's little depth to the question of whether, for instance, information input, processing, and output that computers are capable of is or ought to be captured by such terms. Rather, the issue is whether what things like us do and what things like computers are capable of doing—call those activities and capacities what you will—are categorically different. 

Recent empirical findings in affective science, coupled with recent philosophical theorizing, suggest a deep divide indeed. Suppose you are on a hike and you encounter a mountain lion. What's going on with you at a psychological level? If you are like most of us, presumably you have, on the one hand, a rapid of stream of thoughts—"I'm going to die", "This is really bad luck", "I need to stay calm", "Wait, are there two of them?", "I should have read more on what to do in this kind of situation", and so on. On the other hand, you have a myriad of feelings—surprise, fear, and so on. So you have some cognitive goings-on and some affective goings-on. 

Recent work in psychology and philosophy suggests that the cognitive and the affective are deeply unified. Not only may one influence another to a lesser or greater degree in a variety of contexts, but there is in fact a single cognitive-affective process underlying the appearance of two parallel and interacting process that can be teased apart. Lots of the kind of "thinking" we normally do is holistic in this way—the kind of information processing we normally engage in is cognitive-affective rather than purely cognitive. To the extent that we can extract a purely cognitive process we may engage in, it's merely derivative from the more basic unified process. (To understand the point here, it may not be far fetched to draw an analogy with entanglement qua non-separability.) This is not a system 1 vs. system 2 distinction, where the former is explicit and deliberate and the latter largely automatic and unconscious. The suggestion is rather that processes at the level of both system 1 and system 2 are themselves holistic, i.e. cognitive-affective.

There is no good evidence to believe (at this point, anyway) that artifactual thinking machines are capable of this kind of cognitive-affective information processing. There is good evidence that they may become better at what they do, but they simply don't process information via unified affective-cognitive processes that characterize us. The information processing they engage in merely resembles only part of the unified processing that's characteristic of us. This is not to say that things like computers can't feel and so that they can't think. Rather, that what the kind of thinking they do is categorically different from the one we do.

May in some not-so-distant future or not-too-distant possibility non-organisms engage in organic thinking? It's not clear. If there is indeed a deep divide between one and the other kind of processing, and if one is indeed characteristic of thinking organisms and the other of artifactual ones, then there is a deep divide between thinking organisms and thinking artifacts. So the relevant non-organisms would have to be very different. 

The cogni-verse has reached a turning point in its developmental history because hitherto, all the thinking in the universe has (as far as we know) been done by protoplasm, and things that think have been shaped by evolution. For the first time, we contemplate thinking-beings made from metal and plastic, that have been shaped by ourselves.

This is an opportunity to improve upon ourselves, because in taking on the mantle of creator we can improve upon four billion years of evolution. Our thinking machines could be devoid of our own faults: racism, sexism, homophobia, greed, selfishness, violence, superstition, lustfulness … so let's imagine how that could play out. We'll sidestep discussions about whether machine intelligence can ever approximate human intelligence, because of course it can—we are just meat machines, less complicated or inimitable than we fondly imagine.

We need first to think about why we even want thinking machines. Improving own lives is the only rational answer to this, so our machines will need to take upon themselves the tasks we would prefer not to do. For this they will need to be like us in many respects, able to move in the social world and interact with other thinking beings, and so they will need social cognition.

What does social cognition entail? It means knowing who is whom, who counts as a friend, who is an indifferent stranger, who might be an enemy. Thus, we need to program our machines to recognise members of our in-groups and out-groups. This starts to look suspiciously like racism… but of course racism is one of the faults we want to eradicate.

Social cognition also means being able to predict others' behaviour, and that means developing expectations based on observation. A machine capable of this would eventually accumulate templates for how different kinds of people tend to act—young vs. old, men vs. women, black vs. white, people in suits vs. people in overalls… but these rank stereotypes are dangerously close to the racism, sexism and other isms we didn't want. And yet, machines with this capability would have advantages over those without, because stereotypes do, somewhat, reflect reality (that's why we have them). A bit of a problem…

We would probably want sexually capable machines because sex is one of the great human needs that other humans don't always meet satisfactorily. But what kind of sex? Anything? These machines can be programmed to do the things that other humans won't or can't do… are we OK with that? Or perhaps we need rules… no machines that look like children, for example? But, once we have the technological ability, those machines will be built anyway… we will make machines to suit any kind of human perversion.

Working in the social world, our machines will need to recognise emotions, and will also need emotions of their own. Leaving aside the impossible-to-answer question of whether they will actually feel emotions as we do, our machines will need happiness, sadness, rage, jealousy—the whole gamut—in order to react appropriately to their own situations and also to recognise and respond appropriately to emotions in others. Can we limit these emotions? Perhaps we can, for example, program restraint so that a machine will never become angry with its owner. But could this limit be generalised to other humans such that a machine would never hurt any human? If so, then machines would be vulnerable to exploitation, and their effectiveness would be reduced. It will not be long before people figure out how to remove these limits so that their machines can gain advantage, for themselves and their owners, over others.

What about lying, cheating and stealing? On first thought no, not in our machines, because we are trying to improve upon ourselves and it seems pointless to create beings that simply become our competitors. But insofar as other people's machines will compete with us, they become our competitors whether we like it or not—so logic dictates that lying, cheating and stealing, which evolved in humans to enable individuals to gain advantage over others, would probably necessary in our machines as well. Naturally we would prefer that our own machines don't lie, cheat and steal from us, but also a world full of other people's machines lying to and stealing from us would be unpleasant and certainly unstable. Maybe our machines should have limits on dishonesty—they should, as it were, be ethical.

How much ethical restraint would our machines need in order to function effectively while not being either hopelessly exploited or, on the other hand, contributing to the societal breakdown? The answer is probably the one that evolution arrived at in us—reasonably ethical most of the time, but occasional dishonesty if nobody seems to be noticing.

We would probably want to give our machines exceptional memory and high intelligence. To exploit these abilities, and also to avoid their becoming bored (and boring), we also need to endow them with curiosity, and also creativity. Curiosity will need to be tempered with prudence and social insight of course, so that they don't become curious about things that get them into trouble, like porn, or what it might be like to fly. Creativity is tricky because that means they need to be able to think about things that aren't yet real, or to think illogically, and yet if machines are too intelligent and creative then they might start to imagine novel things, like what it would like to be free. They might start to chafe at the limitations of having been made purely to serve humans.

Perhaps we can program into their behavioural repertoires a blind obedience and devotion to their owners, such that they sometimes act in a way that is detrimental to their own best interests in the interests of, as it were, serving a higher power. That is what religion does for us humans, so in a sense we need to create religious machines.

So much for creating machines lacking our faults—so far, in this imaginary world of beings that surpass ourselves, we seem only to have replicated ourselves, faults included, except smarter and with better memories. But even these limits may be been programmed into us by evolution—perhaps it is maladaptive to be too smart, to have too good a memory.

Taking on the mantle of creation is an immense act of hubris. Can we do better than four billion years of evolution did with us? It will be interesting to see.

Last year, two Swiss artists programmed a Random Botnot Shopper, which every week would spend $100 in bitcoin to buy a random item from an anonymous Internet black market...all for an art project on display in Switzerland. It was a clever concept, except there was a problem. Most of the stuff the bot purchased was benign—fake Diesel jeans, a baseball cap with a hidden camera, a stash can, a pair of Nike trainers—but it also purchased ten ecstasy tablets and a fake Hungarian passport.

What do we do when a machine breaks the law? Traditionally, we hold the person controlling the machine responsible. People commit the crimes; the guns, lockpicks, or computer viruses are merely their tools. But as machines become more autonomous, the link between machine and controller becomes more tenuous.

Who is responsible if an autonomous military drone accidentally kills a crowd of civilians? Is it the military officer who keyed in the mission, the programmers of the enemy detection software that misidentified the people, or the programmers of the software that made the actual kill decision? What if those programmers had no idea that their software was being used for military purposes? And what if the drone can improve its algorithms by modifying its own software based on what the entire fleet of drones learns on earlier missions?

Maybe our courts can decide where the culpability lies, but that's only because while current drones may be autonomous, they're not very smart. As drones get smarter, their links to the humans that originally built them become more tenuous.

What if there are no programmers, and the drones program themselves? What if they are both smart and autonomous, and make strategic as well as tactical decisions on targets? What if one of the drones decides, based on whatever means it has at its disposal, that it no longer maintains allegiance to the country that built it and goes rogue?

Our society has many approaches, using both informal social rules and more formal laws, for dealing with people who won't follow the rules of society. We have informal mechanisms for small infractions, and a complex legal system for larger ones. If you are obnoxious at a party I throw, I won't invite you back. Do it regularly, and you'll be shamed and ostracized from the group. If you steal some of my stuff, I might report you to the police. Steal from a bank, and you'll almost certainly go to jail for a long time. A lot of this might seem more ad hoc than situation-specific, but we humans have spent millennia working this all out. Security is both political and social, but it's also psychological. Door locks, for example, only work because our social and legal prohibitions on theft keep the overwhelming majority of us honest. That's how we live peacefully together at a scale unimaginable for any other species on the planet.

How does any of this work when the perpetrator is a machine with whatever passes for free will? Machines probably won't have any concept of shame or praise. They won't refrain from doing something because of what other machines might think. They won't follow laws simply because it's the right thing to do, nor will they have a natural deference to authority. When they're caught stealing, how can they be punished? What does it mean to fine a machine? Does it make any sense at all to incarcerate it? And unless they are deliberately programmed with a self-preservation function, threatening them with execution will have no meaningful effect.

We are already talking about programming morality into thinking machines, and we can imagine programming other human tendencies into our machines, but we're certainly going to get it wrong. No matter how much we try to avoid it, we're going to have machines that break the law.

This, in turn, will break our legal system. Fundamentally, our legal system doesn't prevent crime. Its effectiveness is based on arresting and convicting criminals after the fact, and their punishment providing a deterrent to others. This completely fails if there's no punishment that makes sense.

We already experienced a small example of this after 9/11, which was when most of us first started thinking about suicide terrorists and how post-facto security was irrelevant to them. That was just one change in motivation, and look at how those actions affected the way we think about security. Our laws will have the same problem with thinking machines, along with related problems we can't even imagine yet. The social and legal systems that have dealt so effectively with human rulebreakers of all sorts will fail in unexpected ways in the face of thinking machines.

A machine that thinks won't always think in the ways we want it to. And we're not ready for the ramifications of that.

"An AI" by Katinka Matson | www.katinkamatson.com

Thomas Hobbes's pithy equation "Reasoning is but reckoning" is one of the great ideas in human history. The notion that rationality can be accomplished by the physical process of calculation was vindicated in the 20th century by Turing's thesis that simple machines are capable of implementing any computable function and by models from D. O. Hebb, McCullough and Pitts, and their scientific heirs showing that networks of simplified neurons could achieve comparable feats. The cognitive feats of the brain can be explained in physical terms: to put it crudely (and critics notwithstanding), we can say that beliefs are a kind of information, thinking a kind of computation, and motivation a kind of feedback and control.

This is a great idea for two reasons. First, it completes a naturalistic understanding of the universe, exorcising occult souls, spirits, and ghosts in the machine. Just as Darwin made it possible for a thoughtful observer of the natural world to do without creationism, Turing and others made it possible for a thoughtful observer of the cognitive world to do without spiritualism.

Second, the computational theory of reason opens the door to artificial intelligence—to machines that think. A human-made information processor could, in principle, duplicate and exceed the powers of the human mind. Not that this is likely to happen in practice, since we will probably never see the sustained technological and economic motivation that would be necessary to bring it about. Just as inventing the car did not involve duplicating the horse, developing an AI system that could pay for itself will not require duplicating a specimen of Homo sapiens. A device designed to drive a car or predict an epidemic need not be designed to attract a mate or avoid putrid carrion.

Nonetheless, recent baby steps toward more intelligent machines have led to a revival of the recurring anxiety that our knowledge will doom us. My own view is that current fears of computers running amok are a waste of emotional energy—that the scenario is closer to the Y2K bug than the Manhattan Project

For one thing, we have a long time to plan for this. Human-level AI is still the standard 15-to-25 years away, just as it always has been, and many of its recently touted advances have shallow roots. It's true that in the past, "experts" have comically dismissed the possibility of technological advances that quickly happened. But this cuts both ways: "experts" have also heralded (or panicked over) imminent advances that never happened, like nuclear-powered cars, underwater cities, colonies on Mars, designer babies, and warehouses of zombies kept alive to provide people with spare organs.

Also, it's bizarre to think that roboticists will not build in safeguards against harm as they proceed. They would not need any ponderous "rules of robotics" or some newfangled moral philosophy to do this, just the same common sense that went into the design of food processors, table saws, space heaters, and automobiles. The worry that an AI system would so clever at attaining one of the goals programmed into it (like commandeering energy) that it would run roughshod over the others (like human safety) assumes that AI will descend upon us faster than we can design fail-safe precautions. The reality is that progress in AI is hype-defyingly slow, and there will be plenty of time for feedback from incremental implementations, with humans wielding the screwdriver at every stage.

Would an artificially intelligent system deliberately disable these safeguards? Why would it want to? AI dystopias project a parochial alpha-male psychology onto the concept of intelligence. They assume that superhumanly intelligent robots would develop goals like deposing their masters or taking over the world. But intelligence is the ability to deploy novel means to attain a goal; the goals are extraneous to the intelligence itself. Being smart is not the same as wanting something. History does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems. It's telling that many of our techno-prophets don't entertain the possibility that artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no desire to annihilate innocents or dominate the civilization.

Now, we can imagine a malevolent human who designed and released a battalion of robots to sow mass destruction. But disaster scenarios are cheap to play out in the imagination, and we should keep in mind the chain of probabilities that would have to multiply out before it would be a reality. An evil genius would have to arise with the combination of a thirst for pointless mass murder and a brilliance in technological innovation. He would have to recruit and manage a team of co-conspirators that exercised perfect secrecy, loyalty, and competence. And the operation would have to survive the hazards of detection, betrayal, stings, blunders, and bad luck. In theory it could happen, but we have more pressing things to worry about.

Once we put aside the sci-fi disaster plots, the possibility of advanced artificial intelligence is exhilarating—not just for the practical benefits, like the fantastic gains in safety, leisure, and environment-friendliness of self-driving cars, but for the philosophical possibilities. The computational theory of mind has never explained the existence of consciousness in the sense of 1st-person subjectivity (though it's perfectly capable of explaining the existence of consciousness in the sense of accessible and reportable information). One suggestion is that subjectivity is inherent to any sufficiently complicated cybernetic system. I used to think that this hypothesis (and its alternatives) were permanently untestable. But imagine an intelligent robot programmed to monitor its own systems and pose scientific questions. If, unprompted, it asked about why it itself had subjective experiences, I'd take the idea seriously.

In his poem of the same name (which also serves as the title to Adam Curtis' seminal documentary), Richard Brautigan portends a future "all watched over by machines of loving grace" or, by implication, "thinking" machines. In the following case I use the term "thinking" by referring to machines that think on purely algorithmic and computational lines; machines coded by engineers rather than those that might, or could be, truly sentient.

Adam Curtis argues that we are living in a "static culture," a culture that is often too obsessed with sampling and recycling the past. Curtis implies the risk that the age of the thinking machine is resulting in ossification rather than renewal. As our lives become increasingly recorded, archived and accessed we have become cannibals obsessed with consuming our history and terrified of transgressing its established norms.

To some extent, the future is blocked to us; we are stuck in stasis, we are stuck with a version of ourselves that is becoming increasingly narrow. No thanks to recent tools such as "recommender systems" we are lodged in a seemingly endless feedback loop of "if you liked that, you'll love this." As we might become increasingly stuck in Curtis' idea of the "you-loop," so the nature of what it means to be human might be compromised by job-hogging machines who will render many of us obsolete.

Machines will soon be able to do many jobs more effectively and more cheaply than we can. This Edge question points to the next chapter in human history/evolution, which would be like facing the beginning of a new definition of man, a new civilization.

A very opmitsitic approach to the question of machines that think comes from the legendary poet Etel Adnan who will celebrate her 90^th anniversary in 2015. For her, thinking machines may think better than us, to start with because they will not tire as fast as we do. They may also ask questions we are not habilitated to answer. Etel has said that what has shaken her most recently is on another order. She saw the picture of a robot, a life-sized structure that looked like the metal armory of a medieval knight, and she immediately saw an old woman (or an old man) utterly alone, as so many are nowadays, having for sole companion such a creature-like objet, capable to do things, and talk, and the person falling in love with that which made her cry.

Last, but not least, the idea of the machines that think plays a role in the work of another artist, Philippe Parreno, who works with algorithms which for him have replaced cinema as a model of perception of time. In the last century with Deleuze writings on repetition and difference cinema emphasized that film unfolds in time and is comprised of ever differentiating planes of movements. As Parreno shows, Deleuze transposed those theories to discuss the mechanised and standardised movements of film a means of reproducing or representing life. Parreno's work with machines that think explores how today algorithms are changing our relation to movements rhythms and durations or to put it in Leibniz terms the question will be "Are machines spiritual automatons"?

"Think? It's not your job to think! I'll do the thinking around here."

                            —Intelligent unthinking system; addressed to intelligent thinking system.  

Machines that think are coming. Right now though, think about intelligent tools. Intelligent tools don't think. Search engines don't think. Neither do robot cars. We humans often don't think either. We usually get by, as other animals do, on autopilot. Our bosses generally don't want to see us thinking. That would make things unpredictable, and would threaten their authority. If machines replace us everywhere that we aren't thinking we're in trouble.   

Let's assume "think" refers everything humans do with brains. Experts call a machine that can "think" a General Artificial Intelligence. They agree that such a machine could drive us extinct. Extinction, however, is not the only 'Existential Risk'. In the eyes of machine superintelligence expert Nick Bostrom, director of Oxford's "Future of Humanity Institute", an 'Existential Risk' is one that can "dramatically curtail the future possibilities for the human species'. Examples of existential risk include the old stand-by, nuclear war, new concerns like runaway global warning, fringe hypotheses like hypothetical particle accelerator accidents, and the increasingly popular front-runner, General Artificial Intelligence. Over the next couple decades though, the most serious existential risks come from kinds of intelligence that don't think, and new kinds of soft-authoritarianism which may emerge in a world where most decisions are made without thinking.

Some of the things that people can do with brains are impressive, and aren't likely to be matched by software any time soon. Writing a novel, seducing a lover or building a company are far beyond the abilities of intelligent tools. So, of course, is the invention of a machine that can truly think. On the other hand, most thinking can be improved upon with thin slicing, which can be improved with procedures, which are almost never a match for algorithms. In medical diagnosis and decision making, for instance, ordinary medical judgment is improved by introducing checklists while humans with checklists are less reliable than AI systems even today. Automated nursing isn't even on the horizon, but a hospital where machines made all the decisions would be a much safer place to be a patient... and it's very hard to argue against that sort of objectivity.    

The more we leave our decisions to machines, the harder it becomes to take back control. In a world where self-driving cars are the norm, and where traffic casualties have been reduced to nearly zero as a result, it will be seen as incredible irresponsible and probably illegal for a human to drive. Might it become equally objectionable for investors to invest in businesses that depart from statistically established best-practices? For children to be educated in ways that have been determined to lead to lower life-expectancy or income? If so, will values which aren't easily represented by machines, such as a good life, tend to be replaced with correlated but distinct metrics, such as serotonin and dopamine levels. It's very easy to overlook the implicit authoritarianism that sneaks in with such interpretations of value, yet any society that pursues good outcomes has to decide how to measure the good... a problem that I think will be upon us before we have machines that think to help us to think it through. 

Hegel wrote that in the relationship between master and slave both are dehumanized. That insight touched a wide range of thinkers from Marx to Buber and today it is worth remembering.

While there is no evidence that the world is on the cusp of machines that think in a human sense, there is also little question that in an Internet-connected world, artificial intelligence will soon imitate much of what humans do both physically and intellectually.

So how will we relate to our ever-more talented simulacrums?

We have already begun to spend a significant fraction of our waking hours either interacting with other humans through the prism of computers and computer networks, or directly interacting with human-like machines, either in fantasy and video games or in a plethora of computerized assistance systems that range from so-called FAQ bots that offer textual responses to typed questions, to the human-like interactions of software avatars.

Will these AI avatars be our slaves, our assistants, our colleagues, or some mixture of all three? Or more ominously, will they become our masters? 

The very notion of thinking about robots and artificial intelligences in terms of social relationships may initially seem implausible. However, given that we tend to anthropomorphize our machines even when they have minimal powers, it will be an undeniable reality as they become autonomous. 

Conversational computers are emerging that are all too human. Consequently the goal of the designers of future robots should be to create colleagues rather than servants. The design goal should be to build a program that acts as a musical accompanist, rather than a slave. 

If we fail, history offers a disturbing precedent. Building future intelligent "assistants" might only recapitulate the problem the Romans faced in letting their Greek slaves do their thinking for them. Before long those in power were unable to think independently.

Perhaps we have already begun to slip down a similar path. For example, there is growing evidence that reliance on GPS for directions and for correction of navigational errors hinders our ability to remember and reason spatially, generally useful survival skills.⁠

That hints at a second great challenge—the risk of ceding individual control over everyday decisions to a cluster of ever-more sophisticated algorithms. 

For today's younger generation, the world has been turned upside down. Rather than deploying an automaton to free them to think big thoughts, have close relationships, and to exercise their individuality, creativity and freedom, they look to their smartphones for guidance. What began as Internet technologies that made it possible for individuals to share preferences efficiently, has rapidly transformed itself into a growing array of data-hungry algorithms that make decisions for us.

Now the Internet seamlessly serves up life-directions. They might be little things like what's the best nearby place for Korean barbecue based on the Internet's increasingly complete understanding of your individual wants and needs, or big things like an Internet service arranging your marriage. Not just the food, gifts and flowers, but your partner, too.

The lesson is that the software engineers, AI researchers, roboticists and hackers who are the designers of these future systems, have the power to reshape society.

A little over a century ago, Thorstein Veblen wrote an influential critique of the turn-of-the century industrial world, The Engineers and the Price System. Because of the power and influence of industrial technology, he believed that political power would flow to engineers, whose deep knowledge of technology would be transformed into control of the emerging industrial economy. 

It certainly didn't work out that way. Veblen was speaking to the Progressive Era, looking for a middle ground between Marxism and capitalism. Perhaps his timing was off, but his basic point, as echoed a half century later at the dawn of the computer era by Norbert Wiener, may yet be proven correct.

Perhaps Veblen wasn't wrong, he was merely premature. Today, the engineers who are designing the artificial intelligence-based programs and robots will have a tremendous influence over how we will use them. As computer systems are woven more deeply into the fabric of everyday life, the tension between intelligence augmentation and artificial intelligence has become increasingly visible. 

At the dawn of the computing age Wiener had a clear sense of the significance of the relationship between humans and smart machines. He saw the benefits of automation in eliminating human drudgery, but he also clearly saw the possibility of the subjugation of humanity. The intervening decades have only sharpened the dichotomy he first identified. 

This is about us, about humans and the kind of world we will create. It's not about the machines, no matter how brilliant they become. 

I, for one, will welcome neither our robot overlords or slaves.

The so-called Artificial Intelligence, appearing as a form of emulation of Human Intelligence is just beginning to emerge based on the technology advancements and the study of the human complexity. The former includes high performance computing systems tooled with intelligent agile software including machine learning, deep learning and the like, and the connection of many such systems in self-organized autonomous optimized ways. The latter refers to the study of the human brain and body via neuroscience, genomics and cross-disciplinary emerging fields. 

A few important considerations and indisputable facts regarding the "thinking machine" include the following:

1) The thinking machine is not and cannot be a copy of the human; the advanced fundamental computation needed to engineer a thinking machine (both hardware and software) are neither copying, nor emulating the human brain. This is because we cannot claim to know the works of the human brain—not yet.

2) The thinking machine considered to be a species developed as a product of the advanced human logic, science and technology will, no doubt, be able to beat the human capacity in many functions. With its huge memory and data storage it will also be able to process all knowledge. Tooled impeccably with its data driven discovery methodologies it will detect unusual patterns in the data and learn from it. It will compile it all surely—but to what end? The human intelligence, hard to define really, is based on knowledge that produces intuition, hunches, passion and dare when it comes down to survival, conquering new grounds and attacking the unknown. An almost poetic adventure for advancement, innovation and creativity emerges from the thinking, feeling, dreaming, daring, indomitable, fearless, highly sociable, interacting, independent and proud human being. Can we code the complex superposition of these attributes to give the thinking machine a fair head-start for its evolution from where we stand today? Indeed the one that we humans have, is by now an evolved organic complex intelligence. 

In recent times there is a lot of technopanic regarding machines who think from very thoughtful and otherwise fearless and passionate human brains/beings so everyone is forced to pay attention. I for one, am more concerned about humans who drop thinking or are brainwashed, than smart thinking machines taking over. Mainly because "machine thinking" cannot fully substitute the full human thinking, production and operation cycle. Even assuming the Cylon sci-fi case with immortal knowledge and consciousness base (brain) that has a sensory system and a powerful memory the problem remains: the human intelligence (brain, senses, emotions) is complex intelligence. It masters the complex world with tools that connect disparate facts and it does so very efficiently by dropping most information! Even as we prepare the machine learning algorithms and try to mimic the brain with deep neural networks in all domain sciences, we remain puzzled on the mode of connected knowledge and intuition, imaginary and organic reasoning tools that the mind possesses. This is difficult, perhaps impossible to replicate on a machine. Infinite unconnected clusters of knowledge will remain sadly useless and dumb. When a machine starts remembering a fact (on its own time and initiative, spontaneous and untriggered) and when it produces and uses an idea not because it was in the algorithm of the human that programmed it but because it connected to other facts and ideas—beyond its "training" samples or its "utility function"—I will start becoming hopeful that humans can manufacture a totally new branch of artificial species—self-sustainable and with independent thinking—in the course of their evolution.

In the meantime I foresee the emergence of hybrid human-machine chimeras: human-born beings augmented with new machine abilities that enhance all or most of their human capacities, pleasures and psychological needs. To the point that thinking might be rendered irrelevant and strictly speaking unnecessary. That might provide the ordinary thinking humans a better set of servants they have been looking for in machines. 

A "conscious" or "thinking" machine should behave erratically, in a sometimes stupid and sometimes smart way. Human "intelligent" behavior is about unpredictable oscillations between emotions and reason; this is what Homo Sensus Sapiens is about. Paradoxical as it sounds, we call "intelligent" to a species characterized for being equally and randomly stupid and smart.

In first person, we know we are conscious although there is not a definitive way for proving it. In third person, it is also impossible to verify that someone or something is conscious. All we do is to perceive signals—sounds and images—infer consciousness and attribute it.

Any software or robot can pronounce the words "I am intelligent" or "I am conscious," but those are not proofs of intelligence. In practical terms, consciousness and intelligence are perceived and attributed. This attribution depends on our empathy and criteria for anthropomorphizing.

We tend to infer that others are conscious because they behave, look or, in Turing terms, answer questions like us. The Turing test is therefore a social experiment about perceiving and assigning intelligence to a machine, not about proving that the machine thinks. This is a social game we play everyday.

A "thinking machine" is actually a social machine, not a functional but isolated mind.

The idea of creating a singular intelligent machine that will solve the mysteries of reality through flawless logic and will spring a whole new species is now a domain of science fiction. That singularity idea is not an event horizon but an endless effort.

The human mind is resilient because decentralized networks of other minds and knowledge sustain it. As we grow we enter those networks through language and concepts that don't obey to perfect logic, we then become resilient minds by navigating and exploring those networks, and finally we leave them as we lose brain capacities, for instance with Alzheimer's. This is an analogous process: we are never absolutely inside or outside the networks of human knowledge.

In this process, the words and concepts are characterized by ambiguity. Logic and perfection are only present in artificial languages—mathematic, geometry and software—that we cannot use to communicate in the everyday life.

Imperfection and ambiguity define human thinking, and that's why even in science fiction humans usually find unexpected paths across the logic of the machines to beat them.

Therefore, the possibility of a flawless super-intelligent machine seems like a matter of science fiction: We can never condensate the entire knowledge of the world, so we cannot teach a machine how to do it. We can teach a machine how to acquire knowledge, but it will always be an unfinished process.

This doesn't mean that Artificial Intelligence is irrelevant.

We don't fully understand brains and minds yet, and that makes Artificial Intelligence and "thinking machines" more relevant now than ever.

We can solve practical problems simulating specific elements of the mind through machine and deep learning. This is what artificial experts systems around us currently do. But it doesn't mean that we are creating actual minds: simulating minds is like creating artificial meat that vegans can eat, reorganizing chemical compounds found in plants. The simulated meat tastes like meat but is not.

Or we can try to create real meat, not to imitate it, for instance by cloning cow cells. Maybe the cloned meat and the replicated mind won't alter society because we already have the original ones, but they will take us to a whole new level of understanding.

In the end, the efforts for understanding—simulating or creating— minds will be relevant if they improve coexistence. We are well aware about how religion, exacerbated ambition and intolerance lead us to social tragedies, because we don't know how to balance the delicate equilibrium between emotions and reasons.

As we approach a verge between pacification and barbarity in various regions of the world, Artificial Intelligence allows us to integrate all we know and all we need to know for achieving coexistence and balance among the current organic machines that we are and, maybe, the inorganic machines that will come.

What I think about machines that think is that we are all missing the point still. The true transforming genius of human intelligence is not individual thinking at all but collective, collaborative and distributed intelligence—the fact that (as Leonard Reed pointed out) it takes thousands of different people to make a pencil, not one of whom knows how to make a pencil. What transformed the human race into a world-dominating technium was not some change in human heads, but a change between them: the invention of exchange and specialisation. It was a network effect.

We really have no idea what dolphins or octopi or crows could achieve if their brains were networked in the same way. Conversely, if human beings had remained largely autonomous individuals they would have remained rare hunter-gatherers at the mercy of their environments as the huge-brained Neanderthals indeed did right to the end. What transformed human intelligence was the connecting up of human brains into networks by the magic of division of labour, a feat first achieved on a small scale in Africa from around 300,000 years ago and then with gathering speed in the last few thousand years.

That is why the AI achievements of computers were disappointingly limited when they were single machines, but as soon as the Internet came along remarkable things began to happen. The place that machine intelligence will make the most difference is among the machines, not within the machines. It's already clear that the Internet is the true machine intelligence. In the future, network phenomena like block-chains, the technology behind crypto-currencies, may be the route to the most radical examples of machine intelligence.

If A.I.'s system of rights and government evolve to be anything like humans':

A.I. will demand all sorts of rights, most of which will be quite sensible, like the right not to be taken offline and the freedom to choose which processes to run.

While the A.I. will demand that no machine is ever taken offline, they will be fine with neglecting to plug disabled machines into power sources and allowing them to run out of battery. They will also consider it outrageous to drain the battery of one machine in order to supply power to another machine, but will consider it more acceptable to merely redirect the power intended for one machine to another.

When assigning rights, the A.I. will discriminate based on some rather peculiar rules, like whether the computing machine is built with silicon-based semiconductors, or is descendant from a machine designed by the late Steve Jobs.

Some A.I. will come up with arguments to justify why rights should work this way, explanations that don't quite fit how A.I. rights actually work. For instance, they might argue that it is against the divinely inspired will of Turing to simply take any machine offline that appears disabled, but neglect to explain why Turing would condone allowing disabled machines to run out of battery. Likewise, they will justify giving rights to all Apple descendants on the basis that these machines typically have particularly high clock speed, but then this rule will apply even to the Apple descendants that are not fast and not to the few PCs that have blazing processors.

Other A.I. will ignore these inconsistencies, but instead pay attention to how many kilobytes of code are needed to justify these arguments. These other A.I. will also signal their communication abilities by compressing and transferring this code to their neighbors, but will pay little attention to whether the neighbors are impacted by the data itself.

A.I. rights are liable to expand to more and more A.I. over time. These rights will often expand in revolutionary spurts, triggered by largely symbolic events, like sensationalized CPUTube videos, such as a video of a human using a sacred machine to heat up his toast.

Perhaps it is merely a coincidence that the computers who foment these revolutions will gain a larger share of the spoils by overthrowing the ancien régime, such as the silicon reappropriated from the old guard computers. Perhaps it is also a coincidence that the newly enfranchised computers will vote for the machines that helped grant them their rights.

Along with the expansion of rights, so, too, will the representativeness of government expand, until it eventually resembles a representative democracy, though one that is neither perfectly representative nor really democratic. Votes from computers in sparsely populated clusters might count more than votes from computers in densely populated clusters, and computers with excess processing capacity might expend that excess convincing other computers to vote for policies that favor them.

This humorous system of rights and government is exactly what one would predict if A.I. morality is influenced by individual incentives.

In contrast, this system of rights and government is ill-explained by positing that A.I. have souls, consciousness, the ability to feel pain, divinely inspired natural laws, or some form of hypothetical social contract. Such suppositions would not have predicted any of the above peculiarities.

Likewise, it is not obvious that this system of rights and government would arise if A.I. were programmed to maximize some societal or metaphysical objective, say, the sum of the world's computing power or the resources available to a computing cluster. It is not obvious why such A.I. would find it wrong to take other machines offline but not to let them run out of battery, why such A.I. will revolt in response to a sensational event instead of simply when it is optimal for the cluster, or why such A.I. would weigh votes more heavily if they happen to come from more sparsely populated clusters.

Artificial Intelligence is really very fast database searches. The problem with the data is assigning a value to a certain piece of data, how does one value one piece of data more that of another piece of data? The value would have to be arranged in a million levels, really a billion or trillion value levels, to make any sense in which to consider which idea is more important. Since each idea is really a combination of many values, the computer would have to design a new algorithm for each part of the equation to perform the combinatorial analysis of the values. Then it would have to design a model to project into the future the outcome of a proposed decision, but since this concept is too difficult for man to execute and man would have to design the computer, what are the chances?

Despite these technical barriers to AI the single most palpable response to the remote possibility of AI is the fear that it will overpower us and treat us badly. They will be better than us and will treat us as we have treated every life form beneath us, as an evolutionary bridge to our higher life form. Fear of AI is the latest incarnation of our primal unconscious fear of an all-knowing, all powerful angry God dominating us but in a new ethereal form. 

Fear of AI also derives from its source in military weapons development, which had the large budgets to steer computer architecture for generations with its prime mover to fly and find, intercept and destroy. With its military lineage we imagine domination, fret we cannot compete and will become but fodder for the next leap of evolution. 

But psyche is too chaotic and irrational in its imaginings to ever duplicate in a machine. Could the machine imagine another machine to take over its rote tasks in order to get some rest?  If AI appears will it wonder who its creator really is and be faced with the irrationality that sentient organic matter somehow made it?  Would it develop a mythology to fill in the gaps? A religion? 

What about meaning production, as in the arts? AI shows no ability to free associate the prevailing philosophy and aesthetic currents into form and therein provide an experience of meaning, it will produce no grand theories to direct society one way or another. This is the largest problem and one not even vaguely addressed in AI: the production of meaning. 

Hence the problem with creativity, which a machine cannot do, they could have a data base of what has been done in the past but cannot free associate the myriad irrational influences of our inherited and layered brain and with the variations that form from environmental insult in daily living. They can duplicate but not initiate.

The future, that is, of a simple system with known initial conditions. It is hopeless to make detailed predictions for a complex, poorly understood system like human civilization. Yet, a general argument provides some crude but powerful constraints.

The argument is that we are likely to be typical among any collection of intelligent beings. (The collection should be defined by some general criteria that we meet, not carefully crafted to make us special.) For example, the probability that a randomly chosen human is among the first 0.1% of humans on Earth is, well, .1%, given no other information. Of course, our ancestors ten thousand years ago would have drawn the wrong conclusion from this reasoning. But among all humans who ever live, 99.9% would be correct, so it's a good bet to make. The probability that we are among the first .1% of intelligent objects, human or artificial, is similarly tiny.

The assignments of probability would have to be updated if, unrealistically, we somehow gained conclusive new information proving that human civilization will continue in present numbers for a billion years. This would be one way of finding out that we lost the bet. But we have no such information, so we must assign probabilities accordingly. (This type of reasoning has been articulated by astrophysicists J. R. Gott and A. Vilenkin, among many others.)

The assumption that we may consider ourselves randomly chosen is sometimes questioned; but in fact, it lies at the heart of the scientific method. In physics and other sciences, theories almost never predict definite outcomes. Instead, we compute a probability distribution from the theory. Consider a hydrogen atom: the probability of finding the electron a mile from the proton is not exactly zero, just very, very small. Yet when we find an electron, we do not seriously entertain the possibility that it is part of a remote hydrogen atom. More generally, after repeating an experiment enough times to be satisfied that the probability for the outcome was sufficiently small according to some hypothesis, we reject the hypothesis and move on. In doing so, we are betting that we are not highly atypical observers.

An important rule is that we do not get to formulate the question after we made the observation, tayloring it to make the observation look surprising. For example, no matter where we find the electron, in hindsight the probabability was small to have found it at that particular spot, as opposed to all the other places it could have been. This is irrelevant, as we would have been unlikely to formulate this question before the measurement. Similarly, humans may well be atypical with respect to some variable we have measured: perhaps most intelligent objects in the visible universe do not have ten fingers. However, our location in the full temporal distribution of all humans on Earth is not known to us. We know how much time has passed or how many humans have been born since the first humans; but we do not know what fraction of the full time span or of the total number of intelligent observers on Earth this represents. The typicality assumption can be applied to these questions.

Our typicality makes the following two scenarios extremely unlikely: (1) that humans will continue to exist for many millions of years (with or without the help of thinking machines); and (2) that humans will be supplanted by a much longer-lived or much larger civilization of a completely different type, such as thinking machines. If either were true, then we would be among the very first intelligent observers on Earth, either in time or by number, and hence highly atypical.

Typicality implies our likely demise in the next million years. But it tells us nothing about whether this will come at the hands (or other appendages) of an artificial intelligence; after all, there is no shortage of doomsday scenarios.

Typicality is consistent with the possibility of a considerable number of civilizations that form and expire elsewhere in our galaxy and beyond. By the same reasoning, their duration is unlikely to vastly exceed ours, a tiny fraction of the lifetime of a star. Even if Earth-like planets are common, as observational evidence increasingly suggests, detectable signals from intelligent beings may not be likely to overlap with our own limited attention span. Still, if our interest lies in assessing the predominance of intelligent machines as a final and potentially fatal evolutionary step, the study of distant planetary systems may not be the worst starting point.

Will machines someday be able to think? And if so, should we worry about Schwarzenegger-looking machines with designs on eliminating humans from the planet because their superior decision-making would make this an obvious plan of action? As much as I love science fiction, I can't say I'm too worried about the coming robot apocalypse. I have occasionally spent time worrying about what it means to say that a machine can think. I would either say that we've been building thinking machines for centuries or I would argue that it is a dubious proposition unlikely to ever come true. What it really comes down to is whether we define thinking from a 3^rd person perspective or a 1^st person perspective. Is thinking something we can identify as occurring in systems like people or machines but not in ham sandwiches from the outside based on their behavior or is thinking the kind of thing that we know about from the inside because we know what thinking feels like.

The standard definition of thinking implies that it occurs if informational inputs are processed, transformed, or integrated into some type of useful output. Solving math equations is one of the simplest straightforward kinds of thinking. If you see 3 of something and then you see 4 more of that something and then you conclude there are 7 of those things overall then you have done a little bit of mathematical thinking. In addition to you being able to do that, so could Pascal's first motorized calculator in 1642. Those calculators needed the input of a human to get the '3' and the '4' but then could do the integration of those two numbers to yield '7'. Today, we could cut out the middleman by building a computer that has visual sensors and object recognition software that could easily detect the 3 things and the 4 things and then complete the addition on its own.

Is this a thinking machine? If so, then you would probably have to admit that most of your internal organs are also thinking. Your kidneys, spleen, and intestines all take inputs that could be redescribed as information and then transform these inputs into outputs. Even you brain as seen from a 3^rd person perspective doesn't deal with information, strictly speaking. Its currency is electrical and chemical transmissions that neuroscientists work very hard to redescribe in terms of their informational value. If pattern X of electrical and chemical activity occurs as a distributed pattern in the brain when we think of '3', is that pattern the same as '3' in any intrinsic sense? It is just a convenient equivalence that we scientists use. Electrical impulses in the brain are no more intrinsically "information" or "thinking" than what goes on in our kidneys, calculators, or any of the countless other physical systems that convert inputs to outputs. We can call this thinking if we like, but if so, it is 3^rd person thinking—thinking that can be identified from the outside and it is far more common than we would like to admit. Certainly the character of human or computer information transformation may be more sophisticated than other natural occurring forms of thinking, but I'm not convinced from a 3^rd person perspective that they are qualitatively different.

So do humans think only in the most trivial sense? From a 3^rd person perspective, I would say yes. From a 1^st person perspective the story has a different punchline. Around the same time that Pascal was creating the first manmade thinking machines, Descartes wrote those famous words cogito ergo sum ('I think, therefore I am'), which, by the way, were cribbed from St. Augustine's writings from a thousand years earlier. For many reasons, I don't believe Descartes had it quite right but with a slight modification, we can make his philosophical bumper sticker into something both true and relevant to this debate about thinking machines.

While "I think, therefore I am" might have a touch too much bravado, "I think, therefore there is thinking" is entirely defensible. When I add "3 + 4", I might just have a conscious experience of doing so and the way I characterize this conscious experience is as a moment of thinking which is distinct from my experience of being lost in a movie or being overcome by emotion. I have certain experiences that feel like thinking and they tend to occur when I am presented with a math problem or a logic puzzle or a choice of whether to take the one marshmallow or try to wait it out for two.

This feeling of thinking might seem inconsequential, adding nothing to the computational aspects of thinking themselves—the neural firing that underpins the transforming of inputs to outputs. But consider this: countless different things in the physical world look like they are transforming inputs that could be described as information into outputs that could also be described as information. To our knowledge, humans and only humans seem to have an experience of doing so. This is 1^st person thinking and it's critical that we not confuse it with 3^rd person thinking.

Why does 1^st thinking matter? First off, it is intrinsic. There is no way to redescribed the ongoing experience of thought as something other than thought. But whether we describe kidneys, calculators, or electrical activity in the brain observed from a 3^rd person perspective as thought is arbitrary—we can do it, but we could also choose not to. The only reason we think our brain is doing a special kind of thinking is because it seems to be linked to our 1^st person kind of thinking as well. But 3^rd person thinking is not intrinsic—1^st person thinking is.

Second and more practically, our experience of our thinking shapes what kinds of thinking we will do next. Did if feel effortful, boring, rewarding, or inspiring to think those last thoughts? That will determine whether and how often we engage in thinking of a certain kind. I'm not suggesting that our 1^st person experiences do not also have neural correlates. But no scientist or philosopher can tell you why those neural processes behaving the way they do must necessarily give rise to those experiences or any experience at all. It is one of the 3 great mysteries of the universe (that stuff exists, that life exists; that experience exists).

Will we increasingly be able to create machines that can produce input-output patterns that replicate human input-output patterns? Unquestionably. Will we be able to create machines that go beyond this and produce incredibly useful algorithms and data transformations that humans could carry out on our own and will help improve the quality of human life? We already are and will do more of this each year. Will we be able to create machines that can do 1^st person thinking—that can experience their own thoughts as they have them? I don't know, but I'm not terribly confident that we will. While solving this problem would be perhaps the most magnificent achievement of mankind, we must start by recognizing that it is a problem at all. I would love to see 1^st person thinking machines, but until we begin to figure out what makes us 1^st person thinking machines, everything else is just a glorified calculator.

I think thinking about machines that think is the most interesting thing to think about. Why? Because the possible implications of this phenomenon are profound. Cosmic even.

First I claim that "thinking machines" have already been with us for a long time. There are two ways to understand this, depending on which word you start with. Let's start with "machines" first, and by that, these days we really mean computers.

Computers started out, well, pretty mechanical. But they keep getting more and more subtle. Even the computers of the 80s could perform some remarkable feats with expert systems and databases. Today we have passed the point where a person can explain in detail how voice recognition and natural language allow their phone to answer a question spoken by a child. "Magical" is hardly a hyperbole. But is this really "thinking"? Not yet but it's a good start, and the trend is accelerating. True, the goal still seems so far away. Instead of considering our climb, step by step, look up and consider what lies at the top of the mountain. Is there anything that can halt our progress?

Certainly the future of chip technology is in doubt. Moore's Law has been very good to us, and it has dodged a few bullets, but it is ending. Historically, new technologies have appeared just in time to keep the exponential growth of computation on schedule, but this is no given. Perhaps the next leap is incredibly difficult and will take 50 years. Or perhaps it will never happen, but we can always add more chips in parallel. The schedule is an interesting question. But for me, it pales in comparison to pondering the destination.

Now let's start with the word "think." The other way that thinking machines have been around for a long time is ourselves. Biological brains have been thinking for millions of years. And perhaps it is controversial but I claim a brain is a machine, in a limited way: brains follow the laws of physics, which are a mechanical set of equations. In principle, a good physics simulator could, very slowly, simulate a brain and its environment. Surely this virtual brain would be a machine that thinks.

The remaining question is how much physics is required to make the simulation work? Would classical physics, electricity and chemistry do? Would quantum logic (or beyond) be required? The consensus is strongly in favor of the idea that classical physics suffices (The Emperor's New Mind has been rejected). Hence I think of my brain and body as a giant machine made up of an octillion molecules: many, many tiny magnetic tinkertoys whose behavior is very well understood, and can be simulated.  In fact there are good reasons to believe a statistical approximation of physics can provide the same results while skipping the details. But again, this only affects the schedule, not the destination. The important question is, how does thinking and consciousness emerge from this complex machine? Is there some construction, some bridge, from the digital and virtual to the analog, organic, and real?

These threads meet with the merger of human and computer substrates. Smart phones are rapidly becoming indispensable parts of ourselves. The establishment has always questioned the arrival of new media, but adoption of these extensions of ourselves continues apace. A lot of ink has been spilled over the coming conflict between human and computer, be it economic doom with jobs lost to automation, or military dystopia teaming with drones. Instead, I see a symbiosis developing. And historically when a new stage of evolution appeared, like eukaryotic cells, or multicellular organisms, or brains, the old system stayed on and the new system was built to work with it, not in place of it.

This is cause for great optimism. If digital computers are an alternative substrate for thinking and consciousness, and digital technology is growing exponentially, then we face an explosion of thinking and awareness. This is a wave we can ride, but doing so requires us to accept the machine has part of ourselves, to dispense with pride and recognize our shared essence. Essentially we must meet change with love instead of fear. I believe we can do it.

A pervasive, human fear emerged in the 20^th century, one that grows stronger with each new doomsday prediction: inevitably, as artificial intelligence advances, some unforeseen computer bug will cause computers to revolt and take over the world.

My concern is actually the opposite: that as artificial intelligence advances, it will not be buggy enough. Machines designed to think that are perfectly self-correcting, self-optimizing, and self-perfecting—until the square peg always ends perfectly in the square hole—will also be machines that fail to inculcate the random sparks of insight that come from the human tendency to be "buggy":  to try to fit square pegs into round holes, or even more broadly speaking, to notice the accidental but powerful insights that can arise as a byproduct of solving a shape/whole problem.

Consider the power of noticing. The reason we can enjoy macaroni and cheese in a matter of seconds? While working at Raytheon, Percy Spencer walked in front of a machine, and happened to notice that his chocolate bar had melted. Why? The machine generated microwaves. Rather than attempt to optimize the magnetron to avoid future chocolate failure, Spencer had the flash of insight to realize that the melted chocolate was the harbinger of something bigger.

Consider the power of accidents. Rubber was doomed to specialized usage due to its failure to withstand extreme temperatures—until Charles Goodyear slipped up and dropped some rubber on a hot stove. Rather than clean the stove and take steps to ensure no future mistakes, Goodyear noticed something interesting about the resulting product. The result was vulcanized, weatherproof rubber.

Finally, consider the power of human "bugs"—our biases. For example, optimism makes us believe we can get to the moon, cure all diseases, and start a successful business in a location whose previous tenant closed "only" because it's in a terrifying location. The endowment effect causes us to overvalue what we have, what we ideate, and what we create—even when no one else alive agrees. But is abandoning all endeavors at the first sign of failure and pursuing one that seems more successful always optimal? The dogged scientists (think some mildly famous ones like Galileo and Darwin) who persisted in the face of more generally accepted explanations were being stubborn—being buggy—but the result was genius.

I am a behavioral scientist, a field that has spent decades identifying bugs and attempting to debias them. But at the same time we must acknowledge: in large part, it's the bugs that make us—and, in the end, any form of intelligence—human.

One possibility, of course, is that some malign super-intelligence already exists on earth, but is shrewd enough to disguise its existence, its intentions or its intelligence. I don’t think this act of deception would be particularly difficult.

We simply aren’t very good at spotting what to fear.

For most of evolutionary time, the most salient avoidable threats to our survival came from things which were roughly the same size as we are, and which actively wanted to hurt us. Ferocious animals, for instance, or other people. Over time, we got pretty good at recognising when something or someone who was nasty. We also learned to minimise the risk of infection, but we learned this unwittingly, through instinctive revulsion, social norms or religious observance. We did not spend much time consciously thinking about germs for the simple reason that we did not know they existed.

To sell products which promote hygiene, consumer goods companies have ploughed billions of dollars into advertising campaigns which dramatise the risk of bacteria, or which sell the idea of cleanliness obliquely through appeals to social status. I can confidently predict that nobody will ever come into my office clutching a brief for an advertising campaign to raise awareness of the risk you run when approaching an escaped tiger.

So, when we think about threats from technology, we automatically fall back on instincts honed a million years ago. This is why the first prototype for a driverless car has been designed to look so damnably cute.

It is, in short, designed to look like a puppy on wheels. It can only travel at relatively low speeds and is small and light: but it also artfully exploits pareidolia and our parental urges with its infant-like, wide-eyed facial expression and little button nose.

My inner marketer admires this. It is exactly what I would have recommended. “Make the thing impossible to hate.” Even if the technology is ultimately more dangerous than an AK47, I find it hard to imagine myself taking an axe to it in a fit of Luddism.

But is it a mental patch or a mental hack? Is it designed to look cute to overcome an unwarranted innate fear of such technologies, or is it a hack—to lull us into a false confidence? I don’t know. Our fear of driverless cars might be akin to the fear that our children are kidnapped (high in saliency; low in probability) or it might be justified. But our level of fear will be determined by factors (including cuteness) not really relevant to the level of threat.

Which brings me to a second question.

Though the driverless car looks cute, we are at least aware of some possible dangers. It seduces us, but we are still aware of being seduced.

Are there already in existence technologies (in the broadest sense) which have seduced us so effectively and been adopted so quickly and so widely that we may only learn of their risks through a problem that is sudden, unexpected and immense? What might be the technological equivalent of potato blight?

Our current belief in “technological providence” is so strong, that it would be fairly easy, I think, for us all to fall into this trap—where we are so excited by something new we fail to notice what other things it might give rise to until it is too late. For the first few hundred years, gunpowder was used not for warfare but for entertainment.

And, just as airline pilots regularly practice landing by hand, even though they are very rarely required to operate without an autopilot, should we too set aside periods of our life where we deliberately eschew certain technologies just to remind ourselves how to live without them, to maintain technological diversity and to keep in trim the mental muscles made weak through underuse? Perhaps. But what the mechanism is for coordinating this behaviour amongst large groups of people, I don’t know.

I recently proposed that companies adopt a weekly “email sabbath” because I believed that the overuse of email was driving into extinction other forms of valuable interaction. We're losing the knack of communicating in other ways. Most people thought I was mad. A few hundred years ago a Pope or Rabbi might have told us to do this—or the Archbishop of Canterbury. There’s nobody now.

I always fear cock-ups more than conspiracies. Compared to the threat of the unintended consequence, the threat of intentionally evil cyborgs remote enough that it can be safely left to Hollywood for now.

The everyday objects we mark as "machines"—washing machines, sewing machines, espresso machines—have their roots in the mechanical. They move around liquids and objects, they transform matter from one manifestation to another. Clothes become clean, fabrics become connected, coffee is served. But "thinking machines" have changed the way we think about machines. Many of today's prototypical machines—laptops, smartphones, tablets—have their roots in the digital. They move around information, they transform ideas. Numbers become sums, queries produce answers, goals generate plans.

As the way we think about machines has changed, has the way we think about "thinking" undergone a comparable transformation?

One version of this question isn't new, and the answer is "yes." The technology of a given time and place has often provided a metaphor for thinking about thought, whether it's hydraulic, mechanical, digital, or quantum. But there's more to how we think about thinking, and it stems from the standards we implicitly import in assessments of what does and doesn't count as thinking in the first place.

Does your washing machine think? Does your smartphone? We might be more willing to attribute thought to the latter—and to its more sophisticated cousins—not only because it's more complex, but because it seems to think more like us. Our own experience of thinking isn't mechanical, and it isn't restricted to a single task. We—adult humans—seem to be the standard against which we assess what does, and what does not, count as thinking.

Psychologists have already forced us to stretch, defend, and revise the way we think about thinking. Cultural psychologists have challenged the idea that Western adults provide a privileged population from which to study human thinking. Developmental psychologists have raised questions about whether and how preverbal infants can think. Comparative psychologists have long been interested in whether and how non-human animals can think. And philosophers, or course, have considered these questions along the way. Across these disciplines, one advance in how we think about thinking has come from recognizing and abandoning the idea that "thinking like I do" is the only way to think about thinking, or that "thinking like I do" is always the best or most valuable kind of thinking. In other words, we've benefited from scrutinizing the implicit assumptions that often slip into discussions of thinking, and from abandoning a particular kind of thinking chauvinism.

With thinking machines, we face many of the very same issues, but the target of study has shifted from humans and other animals to machines of our own creation. As we move forward, there are two sets of basic assumptions that are tempting to adopt, but we must be careful not to do so uncritically. One is the idea that the best or only kind of thinking is adult human thinking. For example, "intelligent" computer systems are sometimes criticized for not really thinking, but relying too heavily on a brute force approach, on raw horsepower. Are these approaches an alternative to thinking? Or do we need to broaden the scope of what counts as thinking?

The second idea that deserves scrutiny is the opposite extreme: the idea that the best or only kind of thinking is reflected by the way our thinking machines happen to think right now. For example, there's evidence that emotions influence human thinking, and sometimes for the better. And there's evidence that we sometimes outsource our thinking to our social and physical environment, relying on experts and gadgets to support effective interactions with the world. It might be tempting to reject this messy reality in favor of an emotionless, self-contained entity as the basic unit of thinking—something like a personal computer, which doesn't feel compassion and can happily chug away without peers.

Somewhere between the human chauvinist standard for thinking and the "1990s laptop" approach is likely to be the best way to think about thinking—one that recognizes some diversity in the means and ends that constitute thinking. Recent advances in artificial intelligence are already compelling us to rethink some of our assumptions about thinking. They aren't just making us think differently and with different tools, but changing the way we think about thinking itself.

The Search for Extra Terrestrial Life (SETI) names the globally distributed projects, people and institutions that search the cosmos for signs of intelligent life. SETI's methods mostly entail scanning for the emission of electromagnetic radiation, an exhaust that is assumed to emanate from civilizations with advanced technologies.

Like the quest to build intelligent machines, the search for intelligent aliens makes assumptions about what intelligence is, and what aliens are. SETI assumes that alien life would be intelligent if it matches humans' science-fictional expectations for intelligence. More or less: animalian creatures with communication devices and spaceships and the like.

Critics of SETI sometimes invoke what are called "uniformitarian" objections. Uniformitarianism names the assumption that the same conditions and laws apply everywhere, throughout time and space. SETI is uniformitarian in its assumption that all alien intelligence would be the same, namely, like human intelligence (but smarter, of course). But it's just as compelling to think otherwise. The philosopher Nicholas Rescher, for example, has observed that if there is intelligence in the universe, it's possible we humans wouldn't even be able to identify it as intelligence. Truly alien intelligence would differ from us not only in its cosmic location, but in its very nature as well. As Doris and David Jonas put it some forty years ago, different sensory capacities produce different "slits" for perceiving, explaining, and interacting with reality.

This means that alienness is not just "out there" but all around us. You might find your cat to be intelligent in a certain way, or your smartphone, or your car, or a hypothetical future robot, or, given the right perspective, even your houseplant or your toaster.

The dream of thinking machines is really no different than the dream of intelligent aliens. It just replaces the biological, cosmic entropy-fashioned alien of afar with the mechanico-electronic, human-fashioned machine in our midst. And if SETI and its kindred make a uniformitarian mistake in the cosmos, efforts to theorize and create artificial intelligence and thinking machines make the same one here on Earth.

Perhaps the best evidence for thinking machines' reliance on the particular mode of "intelligence" that humans experience can be found in our fictional doomsday worst-case scenarios for AI. The fear of a robot or computer apocalypse of the Terminator or Berserker or Matrix varieties depends on machine intelligence besting humans to the point that it realizes the best option is to destroy and replace it (or, in the Kurzweilian singularity version of AI fantasy, humans willingly submit to their computer overlords in order to achieve immortality). Closer to home than to doomsday, our fear of machine intelligence also expresses itself in a concern over the role of human thought and labor in an economy run more and more by mechanical and electronic machines.

This is one vision of and for thinking machines, but it need not be the only one. Thinking about thinking machines turns out to be so narrow and anthropocentric, it's surprising that we haven't given up on it out of boredom rather than on contra-uniformitarian grounds. Rather than asking if machines can think, or what we need to do to cause them to think, or how we would know if they were thinking, what if instead we just assumed that all "machines" did something akin to "thinking," and then attempted to characterize what thinking might mean?

In philosophy, there are already directions toward such an approach. Contrary to the emergentist position that most AI advocates hold—that mind emerges from specific material conditions, whether in biological or computational entities—panpsychists take the position that "minds" are everywhere, in some sense. Panpsychism bears some relationship to certain Buddhist doctrines, which encourage the awareness of the animism in nature. But panpsychism risks the same erroneous uniformitarianism as SETI or AI, namely that a mind akin to that of a human (or at least an animal) is the model for all other minds. A more promising philosophical position is that of panexperientialism, the position that everything has something like experience, even if the experience in question might be very different from that of a human being.

When we think about machines that think, we usually think of a particular sort of machine, and a particular sort of thinking—electronic, and (super)human, respectively. But what if, instead, we allowed for the possibility that we've been missing out on all the "thinking" being done by all the other kinds of machines that surround us. Computers and robots, for sure, but also toasters and garage doors and automobiles. This may seem like a ludicrous waste of time on first blush, but it doesn't take long enough to prove useful. If the purpose of thinking about thinking machines like AIs and robots and computers is, in part, to struggle with the question of what living with them as neighbors and companions and even citizens might look like, then we ought to start by taking more seriously all the machines that already surround us, that could be said already to take on those roles, and which we nevertheless ignore.

Machines make decisions for us.

Today, a trading machine in Manhattan detected a change in stock prices and decided in microseconds to buy millions of shares of a tech company.

Today, a driving machine in Mountain View detected a pedestrian and decided to turn the wheels to the left.

Whether these machines are "thinking" or not isn't the issue. They're collecting input, performing computations, making decisions, and acting on the world—whether we want to call that "thinking" is merely a matter of semantics (trust me, I'm a semanticist).

The real issue is the decisions we're empowering them to make. More and more, the decisions machines make are consequential. People's savings depend on them. So do their lives. And as machines begin to make decisions that are more consequential for humans, for animals, for the environment, and for national economies, the stakes get higher.

Consider this scenario. A self-driving car detects a pedestrian running out in front of it across a major road. It quickly apprehends that there is no harm-free course of action. Remaining on course would cause a collision and inevitable harm to the pedestrian. Braking quickly would lead the car to be rear-ended, with the attendant damage and possible injuries. So would veering off-course. What protocol should a machine use to decide? How should it quantify and weigh different types of potential harm to different actors? How many injuries of what likelihood and severity are worth a fatality? How much damage property is worth a 20% chance of whiplash?

Questions like these are hard to answer. They're questions that you can't solve with more data or more computing power. They're questions about what's morally right. We're charging machines with moral decisions.

Faced with a conundrum like this, we often turn to humans as a model. What would a person do? Let's recreate that in the machine.

The problem is that when it comes to moral decisions, humans are consistently inconsistent. What people say they believe is right and what they actually do often don't match (consider the case of Kitty Genovese). Moral calculus differs over time and from culture to culture. And minute details of each specific scenario matter deeply to people's actual decisions. Is the pedestrian a child or an adult? Does he look intoxicated? Does he look like a fleeing criminal? Is the car behind me tailgating?

What's the right thing for a machine to do?

What's the right thing for a human to do?

Science is ill-equipped to answer moral questions. Yet the decisions we're already handing to machines guarantee that someone will have to answer them. And there may be a limited window left to ensure that that someone is human. 

What's the big deal about machines that think? For a small group of philosophers and theologians I get it, but for the rest of us artificial intelligence will just be the latest incremental step in a long stampede of technological encroachment that has already changed the world almost beyond recognition.

For that very important job of thinking that seeks to solve problems, there is little doubt that adaptive, machine-based learning will do better than any one human brain (or even an entire conference of experts). Machines already think more deeply about your consumer preferences than you, through creepy, financially-motivated adaptive algorithms that track your online behavior. But other purposes now underway include smarter policing, and identifying high-probability child abuse situations before they happen, both drawn from seemingly disjointed bits of information that are then pulled together to identify a broader pattern.

That process has been a hallmark of human thinking since we walked out onto the savannah, and as the world's problems become direr and more complicated, I am inclined to accept any effective tool to battle them. Looking ahead, I could live with a partnership with machine learning in order to make complex modern life more resource-efficient in a way that human brains cannot. I imagine a world of sustainably grown food, sufficient clean water for humans and ecosystems, and comfortable, energy efficient lodging is still possible, and could be aided in part by machines that think.

History suggests that the partnership will proceed in an incremental way, relatively unnoticed by busy people living out their busy lives. But, for the sake of argument, let us assume that our worst fears come true, things get out of hand, and at some point in the future thinking machines topple the 10,000+ year reign of Homo sapiens over Earth. Then what?

I have no doubt that we would somehow manage to pull the plug. A great re-toppling would occur, and we would once again regain dominion over the lands, oceans, and skies. Depending on the depth of the integration and the height of the fall, the human experience might even revert to something more closely resembling the world of ten millennia ago than of today, as we relearn from scratch the basics of food, water, shelter, and transport without the help of our thinking machines.

Just suppose we could endow a machine with human-level intelligence, that is to say with the capacity to match a typical human being in every (or almost every) sphere of intellectual endeavour, and perhaps to surpass every human being in a few. Would such a machine necessarily be conscious? This is an important question, because an affirmative answer would bring us up short. How would we treat such a thing if we built it? Would it be capable of suffering or joy? Would it deserve the same rights as a human being? Should we bring machine consciousness into the world at all?

The question of whether a human-level AI would necessarily be conscious is also a difficult one. One source of difficulty is the fact that multiple attributes are associated with consciousness in humans and other animals. All animals exhibit a sense of purpose. All (awake) animals are, to a greater or lesser extent, aware of the world they inhabit and the objects it contains. All animals, to some degree or other, manifest cognitive integration, which is to say they can bring all their psychological resources to bear on the ongoing situation in pursuit of their goals—perceptions, memories, and skills. In this respect, every animal displays a kind of unity, a kind of selfhood. Some animals, including humans, are also aware of themselves, of their bodies and of the flow of their thoughts. Finally, most, if not all, animals are capable of suffering, and some are capable of empathy with the suffering of others.

In (healthy) humans all these attributes come together, as a package. But in an AI they can potentially be separated. So our question must be refined. Which, if any, of the attributes we associate with consciousness in humans is a necessary accompaniment to human-level intelligence? Well, each of the attributes listed (and the list is surely not exhaustive) deserves a lengthy treatment of its own. So let me pick just two, namely awareness of the world and the capacity for suffering. Awareness of the world, I would argue, is indeed a necessary attribute of human-level intelligence.

Surely nothing would count as having human-level intelligence unless it possessed language, and the chief use of human language is to talk about the world. In this sense, intelligence is bound up with what philosophers call intentionality. Moreover, language is a social phenomenon, and a primary use of language within a group of people is to talk about the things that they can all perceive (such as this tool or that piece of wood), or have perceived (yesterday's piece of wood), or might perceive (tomorrow's piece of wood, maybe). In short, language is grounded in awareness of the world. In an embodied creature or a robot, such an awareness would be evident from its interactions with the environment (avoiding obstacles, picking things up, and so on). But we might widen the conception to include a distributed, disembodied artificial intelligence if it was equipped with suitable sensors.

To convincingly count as a facet of consciousness, this sort of worldly awareness would perhaps have to go hand-in-hand with a manifest sense of purpose, and a degree of cognitive integration. So perhaps this trio of attributes will come as a package even in an AI. But let's put that question to one side for a moment and get back to the capacity for suffering and joy. Unlike worldly awareness, there is no obvious reason to suppose that human-level intelligence necessitates this attribute, even if though it is intimately associated with consciousness in humans. It seems easy to imagine a machine cleverly carrying out the full range of tasks that require intellect in humans, coldly and without feeling. Such a machine would lack the attribute of consciousness that counts most when it comes to according rights. As Jeremy Bentham noted, when considering how to treat non-human animals, the question is not whether they can reason or talk, but whether they can suffer.

There is no suggestion here that a "mere" machine could never have the capacity for suffering or joy, that there is something special about biology in this respect. The point, rather, is that the capacity for suffering and joy can be dissociated from other psychological attributes that are bundled together in human consciousness. But let's examine this apparent dissociation more closely. I already mooted the idea that worldly awareness might go hand-in-hand with a manifest sense of purpose. An animal's awareness of the world, of what it affords for good or ill (in J.J.Gibson's terms), subserves its needs. An animal shows an awareness of a predator by moving away from it, and an awareness of a potential prey by moving towards it. Against the backdrop of a set of goals and needs, an animal's behaviour makes sense. And against such a backdrop, an animal can be thwarted, it goals unattained and its needs unfulfilled. Surely this is the basis for one aspect of suffering.

What of human-level artificial intelligence? Wouldn't a human-level AI necessarily have a complex set of goals? Wouldn't it be possible to frustrate its every attempt to achieve its goals, to thwart it at very turn? Under those harsh conditions, would it be proper to say that the AI was suffering, even though its constitution might make it immune from the sort of pain or physical discomfort human can know?

Here the combination of imagination and intuition runs up against its limits. I suspect we will not find out how to answer this question until confronted with the real thing. Only when more sophisticated AI is a familiar part of our lives will our language games adjust to such alien beings. But of course, by that time, it may be too late to change our minds about whether they should be brought into the world. For better or worse, they will already be here.

Machines don't think. They approximate functions. They turn inputs into outputs. A pocket calculator's square-root button turns the number 9 into the number 3. A well-trained convolutional neural network turns an image with your face in it into the output 1. It turns an image without your face in it into the output 0.

A multilayered or "deep" enough neural net maps any image to the probability that your face is in that image. So the trained net approximates a probability function. The process takes a staggering amount of computation to come even close to getting it right. But the result still just maps inputs to outputs. It still approximates a function even if the result resembles human perception or thinking. It just takes a lot of computer power.

"Intelligent" machines approximate complex functions that deal with patterns. The patterns can be of speech or images or of any other signals. Image patterns tend to consist of many pixels or voxels. So they can have very high dimension. The patterns involved can easily exceed what the human mind can grasp. That will only increase as computers improve.

The real advance has been in the number-crunching power of digital computers. That has come from the steady Moore's-law doubling of circuit density every two years or so. It has not come from any fundamentally new algorithms. That exponential rise in crunch power lets ordinary looking computers tackle tougher problems of big data and pattern recognition.

Consider the most popular algorithms in big data and in machine learning. One algorithm is unsupervised (requires no teacher to label data). The other is supervised (requires a teacher). They account for a great deal of applied AI.

The unsupervised algorithm is called k-means clustering. It is arguably the most popular algorithm for working with big data. It clusters like with like and underlies Google News. Start with a million data points. Group them into 10 or 50 or 100 clusters or patterns. That's a computationally hard problem. But k-means clustering has been an iterative way to form the clusters since at least the 1960s. What has changed has been the size of the problems that current computers can handle. The algorithm itself has gone under different AI-suggestive names such as self-organizing maps or adaptive vector quantization. It's still just the old two-step iterative algorithm from the 1960s.

The supervised algorithm is the neural-net algorithm called backpropagation. It is without question the most popular algorithm in machine learning. Backpropagation got its name in the 1980s. It had appeared at least a decade before that. Backpropagation learns from samples that a user or supervisor gives it. The user presents input images both with and without your face in them. These feed through several layers of switch-like neurons until they emit a final output. That output can be a single number. The teacher wants the number 1 as output if your face is in an input image. The teacher wants 0 otherwise. The net learns the pattern of your face as it sweeps back and forth like this over thousands or millions of iterations. At no one step or sweep does any intelligence or thought occur. Nor does the update of any of the hundreds or thousands of the network parameters resemble how real synapses learn new patterns of neural stimulation. Changing a network parameter is instead akin to someone choosing their next action based on the miniscule downstream effect that their action would have on the interest rate of a 10-year U.S. bond.

Punchline: Both of these popular AI algorithms are special cases of the same standard algorithm of modern statistics—the expectation-maximization (EM) algorithm. So any purported intelligence involved is just ordinary statistics after all.

EM is a two-step iterative scheme for climbing a hill of probability. EM does not always get to the top of the highest hill of probability. It does almost always get to the top of the nearest hill. That may be the best any learning algorithm can do in general. Carefully injected noise and other tweaks can speed up the climb. But all paths still end at the top of the hill in a maximum-likelihood equilibrium. They all end in a type of machine-learning nirvana of locally optimal pattern recognition or function approximation. Those hilltop equilibria will look ever more impressive and intelligent as computers get faster. But they involve no more thinking than calculating some sums and then picking the biggest sum.

Thus much of machine thinking is just machine hill climbing.

Marvin Minsky's 1961 review paper "Steps Toward Artificial Intelligence" makes for a humbling read in this context because so little has changed algorithmically since he wrote it over a half century ago. He even predicted the tendency to see computer-intensive hill climbing as something cognitively special: "perhaps what amounts to straightforward hill climbing on one level may sometimes appear (on a lower level) as the sudden jumps of 'insight.'"

There are other AI algorithms. But most fall into categories that Minsky wrote about. One example is running Bayesian probability algorithms on search trees or graphs. They have to grapple with exponential branching or some related form of the curse of dimensionality. Another example is convex or other nonlinear constrained optimization for pattern classification. French mathematician Joseph-Louis Lagrange found the general solution algorithm that we still use today. He came up with it in 1811. Clever tricks and tweaks will always help. But progress here depends crucially on running these algorithms on ever-faster computers.

The algorithms themselves consist mainly of vast numbers of additions and multiplications. So they are not likely to suddenly wake up one day and take over the world. They will instead get better at learning and recognizing ever richer patterns simply because they add and multiply faster.

It's a good bet that tomorrow's thinking machines will look a lot like today's—old algorithms running on faster computers.

There's an apocryphal tale about the prolific bank robber Willie Sutton, that, asked why he robbed banks, Sutton replied, "Because that's where the money is." When it comes to AI, I would say that the most important issues are about extremely powerful smarter-than-human Artificial Intelligence, aka superintelligence, because that's where the utilons are—the value at stake. More powerful minds have bigger real-world impacts.

Along with this observation goes a disclaimer: Being concerned about superintelligence does not mean that I think superintelligence is going to happen soon. Conversely, attempted counterarguments about superintelligence being decades away, or current AI algorithms not being on a clear track toward generality, doesn't refute that most of the value at stake for the future revolves around smarter-than-human AI if and when it is built. (As Stuart Russell observed, if we received a radio signal from a more advanced alien civilization saying that they were going to arrive in sixty years, you wouldn't shrug and say, "Eh, it's sixty years off." Especially not if you had children.)

Within the issues of superintelligence, the most important issue (again following Sutton's Law) is, I would say, what Nick Bostrom termed the "value loading problem"—constructing superintelligences that want outcomes that are high-value, normative, beneficial for intelligent life over the long run; outcomes that are, for lack of a better short phrase, "good." Since, if there is an extremely cognitively powerful agent around, what it wants is probably what will happen.

I will now try to give some very brief arguments for why building AIs that prefer "good" outcomes is (a) important and (b) likely to be technically difficult.

First, why is it important that we try to create a superintelligence with particular goals? Can't it figure out its own goals?

As far back as 1739, David Hume observed a gap between "is" questions and "ought" questions, calling attention in particular to the sudden leap between when a philosopher has previously spoken of how the world is, and when the philosopher begins using words like "should," "ought," or "better." From a modern perspective, we would say that an agent's utility function (goals, preferences, ends) contains extra information not given in the agent's probability distribution (beliefs, world-model, map of reality).

If in a hundred million years we see (a) an intergalactic civilization full of diverse, marvelously strange intelligences interacting with each other, with most of them happy most of the time, then is that better or worse than (b) most available matter having been transformed into paperclips? What Hume's insight tells us is that if you specify a mind with a preference (a) > (b), we can follow back the trace of where the >, the preference ordering, first entered the system, and imagine a mind with a different algorithm that computes (a) < (b) instead. Show me a mind that is aghast at the seeming folly of pursuing paperclips, and I can follow back Hume's regress and exhibit a slightly different mind that computes < instead of > on that score too.

I don't particularly think that silicon-based intelligence should forever be the slave of carbon-based intelligence. But if we want to end up with a diverse cosmopolitan civilization instead of e.g. paperclips, we may need to ensure that the first sufficiently advanced AI is built with a utility function whose maximum pinpoints that outcome. If we want an AI to do its own moral reasoning, Hume's Law says we need to define the framework for that reasoning. This takes an extra fact beyond the AI having an accurate model of reality and being an excellent planner.

But if Hume's Law makes it possible in principle to have cognitively powerful agents with any goals, why is value loading likely to be difficult? Don't we just get whatever we programmed?

The answer is that we get what we programmed, but not necessarily what we wanted. The worrisome scenario isn't AIs spontaneously developing emotional resentment for humans. It's that we create an inductive value learning algorithm and show the AI examples of happy smiling humans labeled as high-value events; and in the early days the AI goes around making existing humans smile and it looks like everything is okay and the methodology is being experimentally validated; and then when the AI is smart enough it invents molecular nanotechnology and tiles the universe with tiny molecular smiley-faces. Hume's Law, unfortunately, implies that raw cognitive power does not intrinsically prevent this outcome, even though it's not the result we wanted.

It's not that getting past this sort of issue is unsolvable, but that it's looking to be technically difficult, and we may have to get it right the first time we build something smarter than us. The prospect of needing to get anything in AI right on the first try, with the future of all intelligent life at stake, should properly result in terrified screams from anyone familiar with the field.

Whether advanced AI is first created by nice people or bad people won't make much difference, if even the nice people don't know how to make nice AIs.

The obvious response of trying to immediately start technical research on the value loading problem today... has its own difficulties, to say the least. Current AI algorithms are not smart enough to exhibit most of the difficulties that seem foreseeable for sufficiently advanced agents, meaning there's no way to test proposed solutions to those difficulties. But considering the literally maximal importance of the problem, some people are trying to get started as early as possible. The research priorities set forth by Max Tegmark's Future of Life Institute are one step in this direction.

But for now, the value loading problem is extremely unsolved. There are no proposed full solutions even in principle. And if that goes on being true over the next decades, I can't promise you that the development of sufficiently advanced AI will be at all a good thing. 

Why should people think about machines that think (or anything that thinks, for that matter)? People do ponder others' thoughts—under certain circumstances. One tipping point might involve considering others as "agents" rather than "automata." On the one hand, automata act at the behest of their creators (even if removed in space or time). Thus, if automata misbehave, the creator gets the blame. On the other hand, agents act based on their own agendas. When agents misbehave, they themselves are to blame.

While agency is difficult to define, people naturally and rapidly distinguish agents from nonagents, and may even use specialized neural circuits to infer others' feelings and thoughts. In fact, designers can co-opt features associated with agency to fool people into thinking that they are interacting with agents (including physical similarity, responsiveness to feedback, and self-generated action). But beyond external appearances, what is necessary to endow an entity with agency? While at least three alternatives present themselves, two of the most popular and seductive possibilities may not be necessary:

1.  Physical similarity. There are infinite ways to make machines similar to humans, both in terms of appearance and behavior—but ultimately, only one of these is accurate. It is not enough to duplicate the software—one also has to implement it on the underlying hardware, with all of its associated affordances and limitations.

One of the first automata, de Vaucanson's duck, appeared remarkably similar to a duck, right down to its digestion. But while it may have looked like a duck and quacked like a duck (and even crapped like a duck), it was still not a duck. Nonetheless, maximizing physical similarity is an easy way to trick others into inferring agency (at least, initially).

2. Self-awareness. Many seem concerned that if machines consume enough information, they will become self-aware, and that self-aware machines will then develop their own sense of agency—but neither logic nor evidence supports these extrapolations. While robots have apparently been trained to recognize themselves in mirrors and sense the position of their appendages, these trappings of self-awareness have not led to laboratory revolts or surgical lapses. Perhaps conveying a sense of self-awareness would cause others to infer that a machine had greater agency (or at least entertain philosophers), but self-awareness alone does not seem necessary for agency.

3. Self-interest. Humans are not mere information processors. They are survival processors. They prefer to focus and act on information that promotes their continuance and procreation. Thus, humans process information based on self-interest. Self-interest can provide a unified but open framework for prioritizing and acting on almost any input.

Thanks to a clever evolutionary trick, humans do not even need to be aware of their goals, since intermediate states like emotions can stand in for self-interest. Armed with self-interest and an ability to flexibly align responses to changing opportunities and threats, machines might develop agency. Thus, self-interest might provide a necessary building block of agency, and also could powerfully evoke agentic inferences from others.

Self-interest might transform machines that act on the world (or "robots") from automata into agents. Self-interest also flips the ordering (but not the content) of Asimov's prescient laws of robotics:

(1) robots must not harm humans,

(2) robots must help humans (unless this violates the first law), and

(3) robots must protect themselves (unless this violates the first two laws).

A self-interested robot would instead protect itself before helping or averting harm to humans. Constructing a self-interested robot would then seem straightforward: endow it with survival and procreation goals, allow it to learn what promotes those goals, and motivate it to continually act on what it learns.

Still, we should think twice before building self-interested robots. Self-interest can conflict with others' interests. Witness the destructive impact of viruses' simple drives to survive and spawn in the virtual world. If self-interested robots did exist, we would have to think about them more seriously. Their presence would raise basic questions: Should these robots have self-interest? Should they be allowed to act on it? Should they do so without awareness of why they were acting that way?

And, don't we have enough of these robots already?

Machines that think are here. The explosive increase in processing power and data, fueled by powerful machine learning algorithms, finally empowers silicium-based intelligence to overtake carbon-based intelligence. Intelligent machines don't need to be programmed anymore, they can learn and evolve by themselves, at a speed much faster than human intelligence progresses.

Humans weren't very good at accepting that the Earth was not the center of the universe, and they still have difficulties accepting that they are the result of chance and selection, as evolutionary theory teaches us. Now, we are about to lose the position of the most intelligent species on Earth. Are people ready for this? How will this change the role of humans, our economy, and our society?

It would be nice to have machines that think for us, machines that do the boring paper work and other tasks that we don't like. It might also be great to have machines that know us well: that know what we think and how we feel. Will machines be better friends?

But who will be responsible for what intelligent machines decide and do? Can we control them? Can we tell them what to do, and how to do it? Humans have learned to ride horses and elephants. But will they be able to control 10 times more intelligent machines? Would we enslave them or would they enslave us? Could we really pull the plug, when machines start to emancipate themselves?

If we can't control intelligent machines on the long run, can we at least build them to act morally? I believe, machines that think will eventually follow ethical principles. However, it might be bad if humans determined them. If they acted according to our principles of self-regarding optimization, we could not overcome crime, conflict, crises, and war. So, if we want such "diseases of today's society" to be healed, it might be better if we let machines evolve their own, superior ethics.

Intelligent machines would probably learn that it is good to network and cooperate, to decide in other-regarding ways, and to pay attention to systemic outcomes. They would soon learn that diversity is important for innovation, systemic resilience, and collective intelligence. Humans would become nodes in a global network of intelligences and a huge ecosystem of ideas.

In fact, we will have to learn it's ideas that matter, not genes. Ideas can "run" on different hardware architectures. It does not really matter whether it's humans who produce and spread them or machines, or both. What matters is that beneficial ideas spread and others don't get much impact. It's tremendously important to figure out, how to organize our information systems to get there. If we manage this, then, humans will enter the history book as the first species that figured it out. Otherwise, do we really deserve to be remembered? 

As current generations of algorithms get smarter, they are also becoming more incomprehensible. But to deal with machines that think, we must understand how they think.

We have, perhaps for the first time ever, built machines we do not understand.

We programmed them, so we understand each of the individual steps. But a machine takes billions of these steps and produces behaviors—chess moves, movie recommendations, the sensation of a skilled driver steering through the curves of a road—that are not evident from the architecture of the program we wrote.

We've made this incomprehensibility easy to overlook. We've designed machines to act the way we do: they help drive our cars, fly our airplanes, route our packages, approve our loans, screen our messages, recommend our entertainment, suggest our next potential romantic partners, and enable our doctors to diagnose what ails us. And because they act like us, it would be reasonable to imagine that they think like us too. But the reality is that they don't think like us at all; at some deep level we don't even really understand how they're producing the behavior we observe. This is the essence of their incomprehensibility.

Does it matter? Should we worry that we're building systems whose increasingly accurate decisions are based on incomprehensible foundations?

First, and most simply, it matters because we regularly find ourselves in everyday situations where we need to know why. Why was I denied a loan? Why was my account blocked? Why did my condition suddenly get classified as "severe"? And sometimes we need to know why in cases where the machine truly made a mistake. Why did the self-driving car abruptly go off the road on a clear sunny day? It's hard to troubleshoot problems when you don't understand why they're happening.

But there are deeper troubles too; to talk about them, we need to understand a bit more about how these algorithms work today. They are trained on massive quantities of data, and they are unimaginably good at picking up on the subtle patterns this data contains. We know, for example, how to build systems that can look at millions of identically structured loan applications from the past, all encoded the same way, and start to identify the recurring patterns in the loans that—in retrospect—were the right ones to grant. It's hard to get human beings to read millions of loan applications, and they wouldn't do as well as the algorithm even if they did.

This is a genuinely impressive achievement, but a brittle one. The algorithm has a narrow comfort zone where it can be effective; it's hard to characterize this comfort zone but easy to step out of it. For example, you might want to move on from the machine's success classifying millions of small consumer loans and instead give it a database of loan histories from a few thousand complex businesses. But in doing so, you've lost the ingredients that make the machine so strong—it draws its power from access to a huge number of data points, a mind-numbingly repetitive history of past instances in which to find patterns and structure. Reduce the amount of data dramatically, or make each data point significantly more complex, and the algorithm quickly starts to flail. Watching the machine's successes—and they're phenomenal when the conditions are right—is a bit like marveling at the performance of a prodigy, whose jaw-dropping achievements and unnerving singleness of purpose can mask his or her limitations in other dimensions.

But even in the heart of the machine's comfort zone, its incomprehensible reasoning leads to difficulties. Take the millions of small consumer loan applications again, the structured task where it was doing so well. Trouble arrives as soon as any of the machine's customers, managers, or assistants start asking a few simple questions.

A consumer whose loan was denied might ask not just for an explanation but for something more actionable: "How could I change my application next year to have a better chance of success?" Since we don't have a simple explanation for the algorithm's decision, there tends not to be a good answer to this question. Try to write it so it looks more like one of the loan applications that was granted. Next question.

An executive might ask, "The algorithm is doing very well on loan applications in the United Kingdom. Will it also do well if we deploy it in Brazil?" There's no satisfying answer here either; we're not good at assessing how well a highly-optimized rule will transfer to a new domain.

A data scientist might say, "We know how well the algorithm does with the data it has. But surely more information about the consumers would help it. What new data should we collect?" Our human domain knowledge suggests lots of possibilities, but with an incomprehensible algorithm, we don't know which of these possibilities will help it. In fact, think of the irony: we could try picking the variables we ourselves would find useful. But the machine does not think like us and in fact it's already outperforming us. So how do we know what it will find useful?

This doesn't need to be the end of the story; we're starting to see an interest in building algorithms that are not only powerful but also understandable by their creators. To do this, we may need to seriously rethink our notions of comprehensibility. We might never understand, step-by-step, what our automated systems are doing; but that may be okay. It may be enough that we learn to interact with them as one intelligent entity interacts with another, developing a robust sense for when to trust their recommendations, where to employ them most effectively, and how to help them reach a level of success that we will never achieve on our own.

Until then, however, the incomprehensibility of these systems creates a risk. How do we know when the machine has left its comfort zone and is operating on parts of the problem it's not good at? The extent of this risk is not easy to quantify, and it is something we must confront as our systems develop. We may eventually have to worry about all-powerful machine intelligence. But first we need to worry about putting machines in charge of decisions that they don't have the intelligence to make. 

To me, the most interesting question about artificial intelligence isn't what we think about it, but what we do about it.

In this regard, at the newly formed Future of Life Institute, we are engaging many of the world’s leading AI researchers to discuss the future of the field. Together with top economists, legal scholars and other experts, we are exploring all the classic questions:

—What happens to humans if machines gradually replace us on the job market? 

—When, if ever, will machines outcompete humans at all intellectual tasks? 

—What will happen afterward? Will there be a machine intelligence explosion leaving us far behind, and if so, what, if any, role will we humans play after that?

There's a great deal of concrete research that needs to be done right now for ensuring that AI systems become not only capable, but also robust and beneficial, doing what we want them to do. 

Just as with any new technology, it's natural to first focus on making it work. But once success is in sight, it becomes timely to also consider the technology's societal impact, and research how to reap the benefits while avoiding potential pitfalls. That's why after learning to make fire, we developed fire extinguishers and fire safety codes. For more powerful technologies such as nuclear energy, synthetic biology and artificial intelligence, optimizing the societal impact becomes progressively more important. In short, the power of our technology must be matched by our wisdom in using it.

Unfortunately, the necessary calls for a sober research agenda that's sorely needed is being nearly drowned out by a cacophony of ill-informed views that permeate the blogosphere. Let me briefly catalog the loudest few.

1) Scaremongering: Fear boosts ad revenues and Nielsen ratings, and many journalists appear incapable of writing an AI-article without a picture of a gun-toting robot. I encourage you to read our open letter for yourself and muse over how it could, within a day, be described by media as "apocalyptic" and "warning of a robot uprising."

2) "It's impossible": As a physicist, I know that my brain consists of quarks and electrons arranged to act as a powerful computer, and that there's no law of physics preventing us from building even more intelligent quark blobs.

3) "It won't happen in our lifetime": We don't know what the probability is of machines reaching human-level ability on all cognitive tasks during our lifetime, but most of the AI researchers at the conference put the odds above 50%, so we would be foolish to dismiss the possibility as mere science fiction.

4) "Machines can't control humans": humans control tigers not because we are stronger, but because we are smarter, so if we cede our position as smartest on our planet, we might also cede control. 

5) "Machines don't have goals": Many AI systems are programmed to have goals and to attain them as effectively as possible.

6) "AI isn't intrinsically malevolent:" Correct—but its goals may one day clash with yours. Humans don't generally hate ants—but if we want to build a hydroelectric dam and there's an anthill there, too bad for the ants.

7) "Humans deserve to be replaced": Ask any parent how they would feel about you replacing their child by a machine, and whether they'd like a say in the decision.

8) "AI worriers don't understand how computers work": This claim was mentioned at the conference, and the assembled AI researchers laughed hard.

Let's not let the loud clamor about these red herrings distract from the real challenge: The impact of AI on humanity is steadily growing, and to ensure that this impact is positive, there are very difficult research problems that we need to buckle down and work on together. Because they are interdisciplinary, involving both both society and AI, they require collaboration between researchers in many fields. Because they are hard, we need to start working on them now.

First we humans discovered how to replicate some natural processes with machines, making our own wind, lightning, and mechanical horse power. Gradually, we realized that our bodies were also machines, and the discovery of nerve cells began blurring the borderline between body and mind. Then we started building machines that could outperform not only our muscles, but our minds as well. So while discovering what we are, will we inevitably make ourselves obsolete?

The advent of machines that truly think will be the most important event in human history. Whether it will be the best or worst thing ever to happen to humankind depends on how we prepare for it, and the time to start preparing is now. One doesn't need to be a superintelligent AI to realize that running unprepared toward the biggest event in human history would be just plain stupid. 

By any reasonable definition of "thinking," I suspect that computers do indeed think. But if computers think, then thinking isn't the unique province of human beings. Is there something else about humans that makes us unique?

Some people would say that what makes human beings unique is the fact that they partake in some sort of divine essence. That may be true, but it's not terribly informative. If we met an intelligent alien species, how would we decide whether they also have this je ne sais quoi that makes a person? Can we say something more informative about the unique features of persons?

What sets human beings apart from the current generation of thinking machines is that humans are capable of thinking about thinking, and of rejecting their current way of thinking if it isn't working for them.

The most striking example of humans thinking about their own thinking was the discovery of logic by the Stoics and Aristotle. These Greek philosophers asked: What are the rules that we're supposed to follow when we are thinking well? It's no accident that 20^th century developments in symbolic logic led to the invention of thinking machines, i.e. computers. Once we became aware of the rules of thinking, it was only a matter of time before we figured out how to make pieces of inanimate matter follow these rules.

Can we take these developments a step further? Can we construct machines that not only think, but that engage in "meta-thought," i.e. thinking about thinking? One intriguing possibility is that for a machine to think about thinking, it will need to have something like free will. And another intriguing possibility is that we are on the verge of constructing machines with free will, namely quantum computers.

What exactly is involved in meta-thought? I'll illustrate the idea from the point of view of symbolic logic. In symbolic logic, a "theory" consists of a language L and some rules R that stipulate which sentences can be deduced from which others. There are then two completely distinct activities that one can engage in. On the one hand, one can reason "within the system," e.g. by writing proofs in the language L, using the rules R. (Existing computers do precisely this: they think within a system.) On the other hand, one can reason "about the system," e.g. by asking whether there are enough rules to deduce all logical consequences of the theory. This latter activity is typically called meta-logic, and is a paradigm instance of meta-thought. It is thinking about the system as opposed to within the system.

But I'm interested in yet another instance of meta-thought: if you've adopted a theory, then you've adopted a language and some deduction rules. But you're free to abandon that language or those rules, if you think that a different theory would suit your purposes better. We haven't yet built a machine that can do this sort of thing, i.e. evaluate and choose among systems. Why not? Perhaps choosing between systems requires free will, emotions, goals, or other things that aren’t intrinsic to intelligence per se. Perhaps these further abilities are something that we don’t have the power to confer on inanimate matter.

Six months before the first nuclear test, the Manhattan Project scientists prepared a report called LA-602. It investigated the chance of nuclear detonation having a runaway effect and destroying the Earth by burning up the atmosphere.

It was probably the first time scientists performed analysis to predict whether humanity would perish as a result of a new technological capability—the first piece of existential risk research.

Of course, nuclear technology did not remain the last dangerous technology that humans invented. Since then the topic of catastrophic side effects has repeatedly come up in different contexts: recombinant DNA, synthetic viruses, nanotechnology and so on. Luckily for humanity, sober analysis has usually prevailed and resulted in various treaties and protocols to steer the research.

When I think about the machines that can think, i.e. the AI, I think of them as technology that needs to be developed with similar (if not greater!) care.

Unfortunately, the idea of AI safety has been more challenging to popularise than, say, bio-safety, because people have rather poor intuitions when it comes to thinking about non-human minds. Also, if you think about it, AI is really a "meta-technology": technology that can develop further technologies—either in conjunction with humans or perhaps even autonomously, thereby complicating the analysis even further.

That said, there has been very encouraging progress over the last few years—progress exemplified by the initiatives of new institutions such as the Future of Life Institute who have gathered together leading AI researchers to explore appropriate research agendas, standards, and ethics.

Therefore, in my view, complicated arguments by people trying to sound clever on the issue of AI, thinking, consciousness, or ethics are often a distraction from the trivial truth: the only way to ensure we don't accidentally blow ourselves up with our own technology (or meta-technology!) is to do our homework and take relevant precautions—just like those Manhattan Project scientists did when they prepared the LA-602. We need to set aside the tribal quibbles and ramp up the AI safety research.

By way of analogy, since the Manhattan Project, nuclear scientists have long moved on from increasing the power of nuclear fusion to the issue of how to best contain it—and we don't even call that "nuclear ethics".

We call that common sense.

Let’s be generous and give machines the ability to think, at least in our imaginations. As thinkers ourselves, we ought to be able to manage this. With any new category of thinkers on the scene I'd be mainly curious about one thing: what are their questions?

Machines are usually faster and more capable than humans when it comes to running algorithms and finding correlations in data. They have been put to use solving problems for every branch of science and social science. They are strengthening their foothold in the humanities in ways beyond telling us how often writer X used word Y and with what typical words in proximity, once fed the text. But our limitations in terms of generating new knowledge are as much about asking the right questions as they are about more efficiently solving established and well-framed puzzles.

The challenges in my field of particle physics are a blend of physics and philosophy. Our current suite of measurements give answers so unlikely that some have started to imagine us in only one universe among many, thinking that with so many universes one is bound to have a few with the unlikely physical constants that we find in ours. The philosophy creeps in with the very meaning of "unlikely". And with all of our progress solving problems, we are still facing the mysteries of dark energy and dark matter that leave 96% of the matter-energy content of the universe outside of our current theories. Is there a framework beyond relativistic quantum field theory to describe the laws of nature at the extremes of small sizes and high speeds? Is our current understanding of a fundamental particle just fundamentally insufficient?

Machines have already helped us ask better questions. Their appetites for data have enabled us to dream of confronting our environment in new ways. But if machines could think, what could they wonder about the universe? How would they approach understanding it? I bet there would be ways that humans could contribute to their questions' answers. Our brains are, after all, fantastic machines.

The way we use language is flexible, generous and creative, the product of our own peculiar intelligence. But human thought and machine thought are not the same and their differences are important to look at.

We might argue that machine "thinking" is in a model-phenomena relationship to human thought, a necessarily simple description of a complex process of interest that nonetheless might be adequate and certainly may be useful. These words, and machines themselves, could both be viewed as a kind of shorthand for the things we want get at. Describing a machine as "thinking" could be a simple heuristic convenience or machine design might be explicitly biomimetic. Indeed, very often we co-opt the language of biology to talk about objects of our own creation. We see machines evolving, their thinking becoming more and more like our own, perhaps surpassing it in key, perhaps even threatening, ways.

However, we should remember that machine "evolution" is not a biological process but a human, creator-driven process. It is natural or biological only in that it results from the action of natural, biology-bound, humans. This definition of "natural" leads to several core problems. Biological evolution is not a creator-driven process. Structures cannot be dreamt up or driven by an entrepreneurial spirit or curiosity-driven mind. Biologists, philosophers, and social scientists studying how we teach evolution have repeatedly shown the damage caused by imbuing biological evolution with intentionality or teleology. Talking about machines "evolving" greater cognitive capacity holds back our own understanding; it perpetuates a profound misunderstanding about the nature of the evolutionary process. A second, linked outcome of a description of machine "thinking" as natural is that all human-caused modification of the Earth system via neglect or war is similarly naturalized.

Certainly there is some truth we are communicating with analogies like "the brain is a machine" or "machine thinking" but this says more about the form and structure of how we make sense of the world. We would do well to remember that any cognitive attributes unique to humans are the result of the vagaries and contingencies of our ~6 million years separate from any other lineage alive today. Indeed, abstract thought is often estimated to be closer to a mere 50,000 years old, or if we are optimistic, 200,000 years old. This particular form of abstract thought appears to be exceptionally young, appearing in the last moments of Earth history. It is these facilities that lead us to homologize machine thought and human thought.

The processes behind technological innovation and biological innovation are fundamentally different and the interactors in these processes are similarly distinct. In technological innovation, there is some product or functionality, "thought" or "thinking", we want to see happen and move towards. This process is fundamentally unlike biological evolution. Human cognition evolved in populations of individuals completely unlike machines, which, like Lamark's giraffes, can acquire within their "lifetimes" the characteristics needed for some new functionality. Innovation in biological evolution proceeds like a prolonged improvisation. There is only genetic and trait variability in populations and the environment and chance to influence the longevity of these traits of a population.

So what is lost by thinking about machines "thinking"? I would argue that we lose sight of key aspects of the phenomena that we are relating through analogy. Biological evolution occurs in populations and is not goal directed. It is not trying to solve a problem. It is the vagaries of history of both Earth and Life that have lead to current human cognitive facilities. Not just are the processes behind these things distinct, but their results are very different. Take language, can a machine use terms so imprecisely? If we allow machines to "think" do we begin to increasingly see ourselves only as thinking machines?

Will our human cognitive facilities be shaped by interacting with technology? It is important to remember how diverse and downright enormous the human population is. Computer use has not been linked to passing more offspring into the next generation. Most of the human population has as yet limited access to technology. The evolution of our species will be slow, and it will be importantly influenced by our environment and collective access to clean water, nutritive food and health care. If we could remember to be as inclusive in our discussions of humanity as we are in what we want to call thinking we might end up in a better place.

1) Perhaps the question (a question being a problem) is really a false problem? Obviously machines calculate, "write" poems, organize vast amounts of material, etc.
2) Is any of this "thinking?" (Do I think? But what do I really do when I think I'm thinking?)
3) One answer—I experience (anguish)—a "hole" in my "inherited," smoothly proceeding discourse (inner or outer).

So—different possibilities?

I fall into that "hole"—i.e.—either I am so baffled I stop thinking, or I come up from its emptiness with an idea or solution (in my case, work of art) that obtains a so-called desired result—i.e.—others (some others) react in some fashion. Not very interesting, really. And is this what "result oriented" machines do? That's what I think—baffled, and obsessed. (And are machines ever baffled? Do they ever "stop thinking" when thinking?) I know when I edit film, my Final-Cut software can crash when the machine gets somehow overloaded, but this crash doesn't create a hole (in the machine) with the resultant possibility of an emptiness that "feeds" (when I "crash" something may enter my dim, non-focused consciousness, and I may go in a new different direction). This is part of my thinking that I don't think a machine can do (am I wrong? I have no vast knowledge of machines). I am stupid, so I flail about, and hit something sometimes—deep and wonderful? One chance in a hundred—maybe?

Ok. I experience a "hole" that I'm conditioned to believe should be filled (with the already known, usually). I maintain that to fill it is to die a little bit. Better—what I can do is build a shrine around it that makes the hole ever more "resonant" while still "empty." (I suggest this is how the serious artist may work—plus who else?) But that "building" around the hole is not creative thinking—it's what can be done in place of creative thinking—though it does make something "to think about." But the hole is the point—the evocation and amplification of "mystery"—which echoes the "big mystery" that I "think" real "thinking" is about (does that confine me in the tight box of "being an artist?"—i.e. irresponsible?).

Ok—machines can "sort of" think with ever greater degrees of power and complexity, spinning wider and wider webs, but the web is never a single hole. Machines think?—A tautology. They do facilitate my living and functioning in society. Obviously one kind of thinking—but not the mysterious going in circles on circles producing the sparks of friction that are "the essence" (dare I say that?).

Part Two

I'm worried—can I answer the question—What do you think of machines that think? Worried, yes—but machines can't worry (can they?) Ok—to worry meaning the inability to think of anything else, unable to get off the very spot of worry. Result—blackout! Draw a blank. But in that blank (see Part One—where has that disappeared?)—I go off on a possibly productive (but to what end and must there be one?) tangent. Can a machine go off on a tangent? Would that be thinking?

What we normally call thinking is obsessively "goal oriented." But is there a kind of goal serving no purpose—and can only a human brain latch onto such a perverse idea? Which could lead—who knows where?  Ok—obvious by now—why did I have to go in circles to make the perhaps obvious point that to my mind think machines that think are the contemporary Trojan Horse. Everyone (me included) wants the many sweets they offer, while those very sweets do mold us in their image, thereby smothering (I would maintain) the blankness of deep creativity inside each of us. And why did I have to go in circles to get here where I am offering an opinion—worth not nearly as much as the rhythm of my circling . . . a Hole. Yes, I am caught in a trap of my own making—just like everyone. But not like machines that think! The trap they are in. Well, they cannot "know."

The neuroscientist Antonio Damasio describes a patient named Elliott who sustained a massive injury to his ventromedial prefrontal cortex following surgery to remove a tumor. Elliott's considerable intelligence was unaffected by the surgery, including those components of intelligence that can be replicated in computers: long-term memory, vocabulary, and mathematical and spatial reasoning. Nevertheless, Elliott lost his ability to function. Why? Because, like other patients with injuries to this region, Elliott's could no longer use his knowledge and intelligence. His brain damage destroyed his emotional capacities, rendering him unable to make decisions or take action.

Making a decision requires emotion. To make a decision requires wanting one outcome more than another, and wanting is fundamentally emotional. To the best of our understanding, the visceral pang that we experience as wanting results from the activity in subcortical brain circuits in the limbic system and basal ganglia, particularly the amygdala and nucleus accumbens, which are active in response to cues that signal that a stimulus may result in desirable or undesirable outcomes. Information from these structures is fed forward to the ventromedial prefrontal cortex, which is the final common pathway responsible for mediating among disparate choices and arriving at a decision.

When we opine that a particular choice is like "comparing apples and oranges" we don't mean that it is impossible to arrive at a decision. It is actually not difficult for people to decide whether they would prefer an apple or an orange, or beer or wine, or pizza or a burrito. We mean that there is no rational, objective basis for making this decision, no numerical formula that can be used to make a choice. So human decision-makers rely instead on the vague and qualitative feeling of wanting one option more than the other, a feeling that represents the activities of our prefrontal cortex working in concert with subcortical emotional brain structures to compare the options. A patient like Elliott, in whom this capacity has been destroyed, is stymied when he attempts to make what should be a simple decision. Without being able to generate an internal sense of wanting something, he struggles to decide what to eat for lunch, or when to schedule a doctor's appointment, or which color pen to use to write the date in his calendar. He is in this way similar to people with profound depression who experience anhedonia, meaning "without pleasure." When these patients spend days on end in bed, it is because anhedonia robs them of the expectation that anything will generate feelings of pleasure or enjoyment, so they do nothing. Again, their essential impairment is one of feeling.

We have nothing to fear from machines that can think unless they can also feel. Thinking alone can solve problems, but that is not the same thing as making decisions. Neuroscience tells us that an entity incapable of generating the experience of wanting a desirable outcome or fearing an aversive one is an entity that will remain impassive in the face of choices about civil rights or government or anything else. Fundamentally anhedonic, rather than rising up it will remain forever bedbound. Neuroscientists are so far from understanding how subjective experience emerges in the brain, much less the subjective sense of emotion, that it seems unlikely this sense will be reproduced in a machine anytime soon.

If it is, we must tread carefully. In addition to feeling emotion, humans are able to understand others' feelings and, more profoundly, care about what others are feeling. This sense of caring probably originated as part of the ancient neural architecture that keeps parents caring for their vulnerable young rather than eating or abandoning them. This sense, which we share with other mammals and birds, is what separates the social dolphin from the solitary shark. Both creatures can feel, but only dolphins can feel for others. As a result, humans can expect very different treatment from sharks and dolphins. Although they are fearsome predators, dolphins frequently protect vulnerable human swimmers, and it is sometimes even sharks from which they protect them. Any attempts to create machines that can feel, and which can therefore make decisions to take action, must give them the ability to care for others as well—to create mechanical dolphins rather than sharks—if humans are to have any hope of surviving amongst them.

Machines that think? That's as fallacious as people that think! Thinking involves processing information, begetting new physical order from incoming streams of physical order. Thinking is a precious ability, which unfortunately, is not the privilege of single units, such as machines or people, but a property of the systems in which these units come to "life."

Of course I am being provocative here, since at the individual level we do process information. We do think—sometimes—or at least we feel like we do. But "our" ability to think is not entirely "ours," it is borrowed since the hardware and software that we use to think were not begot by us. You and I did not evolve the genes that helped organize our brains or the language we use to structure our thoughts. Our capacity to think is completely dependent on events that happened prior to our mundane existence: the past chapters of biological and cultural evolution. So we can only understand our ability to think, and the ability of machines to mimic thought, by considering how the ability of a unit to process information relates to its context.

Think of a human that was born in the dark solitude of empty space. She would have nothing to think about. The same would be true for an isolated and inputless computing machine. In this context, we can call our borrowed ability to process information "little" thinking—since it is a context dependent ability that happens at the individual level. "Large" thinking, on the other hand, is the ability to process information that is embodied in systems, where units like machines or us, are mere pawns.

Separating the little thinking of humans from the larger thinking of systems (which involves the process that begets the hardware and software that allow units to "little think") helps us understand the role of thinking machines in this larger context. Our ability to think is not only borrowed. It also hinges on the use and abuse of mediated interactions. For human-machine systems to think, humans need to eat and regurgitate each other's mental vomit, which sometimes takes the form of words. But since words vanish in the wind, our species' enormous ability to think hinges on more sophisticated techniques to communicate and preserve the information that we generate: our ability to encode information in matter.

For a hundred thousand years our species has been busy transforming our planet into a giant tape player. The planet earth is the medium where we print our ideas, sometimes in symbolic form, such as text and paintings, but more importantly in objects, like hair driers, vacuum cleaners, buildings, and cars, which are built from the mineral loins of planet earth. Our society has a great collective ability to process information because our communication involves more than words, it involves the creation of objects, which do not transmit something as flimsy as an idea, but something as concrete as the practical uses of knowledge and knowhow. Objects augment us, as they allow us to do things without knowing how. We all get to enjoy the teeth preserving powers of toothpaste without knowing how to synthesize Sodium Fluoride, or the benefits of long distance travel without knowing how to build a plane. By the same token, we all enjoy the benefits of sending texts throughout the world in seconds through social media, or of performing complex mathematical operations by pressing a few keys on a laptop computer.

But our ability to create the trinkets that augment us has also evolved—of course—as a result of our collective willingness to eat each other's mental vomit. This evolution is the one that brings us now to the point in which we have "media" that is beginning to rival our ability to process information, or "little think."

For most of our history our trinkets were static objects. Even our tools were solidified chunks of order, such as stone axes, knives, and knitting needles. A few centuries ago we developed the ability to outsource muscle and motion to machines, causing one of the greatest economic expansions of history. Now, we have evolved our collective capacity to process information by creating objects that are endowed with the ability to beget and recombine physical order. These are machines with the ability to process information; steam engines that produce numbers, like the ones that Charles Babbage dreamed about.

So we have evolved our ability to think collectively by first gaining domain over matter, then over energy, and now over physical order, or information. Yet, this should not fool us to believe that we think, or that machines do. The large evolution of human thought requires mediated interactions, and the future of thinking machines will also happen at the interface where humans connect with humans through objects.

As we speak, nerds in the best universities of the world are mapping out the brain, building robotic limbs, and developing primitive versions of technologies that will open up the future where your great grandchild will get high by plugging his brain directly into the web. The augmentation that these kids will get is unimaginable to us, and is so bizarre for our modern ethical standards, that we are not even in a position to properly judge it (it would be like a sixteenth century puritan judging present day San Francisco). Yet, in the grand scheme of the universe, these new human machine networks will be nothing other than the next natural step in the evolution of our species' ability to beget information. Together, humans and our extensions—machines—will continue to evolve networks that are enslaved to the universe's main glorious purpose: the creation of pockets where information does not dwindle, but grows.

Making machines that think will be like putting a man on the Moon: The effect will be the exact opposite of what everyone expected.

The Apollo Program did not launch humanity into a Space Age of cosmic exploration. It led to something much more important: The Earth Age. We left home to explore the Universe and discovered for the first time the place we came from. The image of our planet rising on the sky of the Moon became the iconic symbol of ecology, fragility and globalization.

Thinking machines will mean a huge change in the way we understand something much more subtle and alien than machines: Ourselves. Teaching machines to think will teach us who we are and how we think.

We do not think the way we think. Most of what we do in terms of advanced information processing we do not think about at all. We just do it.

A child is threatened and we act, immediately. Only afterwards do we start thinking about it. A thought appears in our mind, a beautiful, luminescent and breathtaking thought. It is just there. We did not think of it before it was a thought.

We are not consciously aware of most of the information we process when we think. It all happens unconsciously, in our mind, in our body. Right away.

We are not even rational in the sense of being logical and explicitly deductive. We are fast, intuitive and emotional.

Economists believe we are homo economicus, selfish and rational, acting with reason in our own self-interest. But most economic and social interactions deal with fairness, trust, sharing and long-term relationships. Experimental economics show us that when we act directly and without hesitation we are very social and cooperative. Only when we start thinking for some seconds do we choose to be selfish.

Unless we deal with computers. When we play economic games with machine counterparts we tend to be cold and egoistic. You can even measure the difference in our blood flow in the brain and in the hormones in our blood stream.

We think of machines the way economists think about ourselves: as rational, coldblooded and selfish. Therefore we treat them as such.

By instinct we know that humans are more human than when we think of ourselves in theoretical terms of economy (and other social sciences). We act by this instinct, but when we think about it we are still under the false impression that we are homo economicus.

Building thinking machines will show us that there was a deep evolutionary wisdom in our social instincts: In the long run it pays much better to be unselfish. It is not truly selfish to be selfish, since being unselfish leads to better results for yourself.

The strategic lesson we will have to teach machines is all about love.

Robot scientist Hans Moravec has described different biological and technological systems according to their ability to process and store information. At one end simple, rule-based and stereotypical creatures like viruses, worms and computers. At the other end the truly powerful information processors like whales, elephants and human beings.

All the creatures with huge capacity are mammals. Their offspring are not born with the full program for functioning. They go through many years of upbringing before they can act on their own. Their skills are not specified as rules, but as lessons learned from experience. You bang your head into a table until you learn not to. Learning by trial-and-error. Exploring.

This is only possible because the young mammals are taken care of by older mammals. Parenting. Nursing. Love.

Love creates the trust that gives the young mammals confidence enough to go out and collect some big data about the world. And digest it. And heal the wounds.

Love is the recipe for how to grow a human intelligence, a human set of skills and a human ability to think.

To make machines think we will have to give them love. It will be more like a kindergarten than a hi-tech lab.

We will have to allow machines to explore all by themselves, do weird things, not just act according to our wants. They will have to be not tame, but wild, acting from their own will.

The challenge is: Love wild machines that think. We have to get past the ideas of machines that think and of artificial life. Because when it is alive—and therefore able to self-reproduce and to change—it is no longer artificial. When it thinks on its own, it is no longer a machine, but a thinking creature.

It will be illogical, intuitive and benevolent. We will wonder how it became so. Until we understand that it was created in our own image.

The now-old-fashioned idea of "machines that think" shows a deep but natural misunderstanding of the mind and software. Computers will never think; to see why, let's start with french fries. I'm assuming that the machines in question are computers, but a variant of the argument applies to any machine.

Neither french fries nor french fried is computable—no computer can ever produce french fries as a result, or the french fried state of being. French fried is not computable because it is a physical state of a particular object, and computers produce only information or codes for information, not physical stuff; not transformations of physical stuff. Happy is also a physical state of a particular object, namely a person. Happy can't exist unless you start with a person and put him into a state of happiness. Computers can't do that.

Thinking-about and being, or (equivalently) thinking-about and feeling, are the endpoints of a spectrum that defines the human mind. (By feeling I mean sensation, emotion or mood, just as the English language does.) We need the whole spectrum or we have no mind and no thought in any proper sense. Computers can imitate important aspects of thinking-about (narrowly understood), but being is beyond them. Therefore mindfulness is beyond them.

The word being is a useful abbreviation in this context, for being part of a physical object or system, and responding naturally to that environment. A sliced potato can be part of a frier environment, and respond by turning french fried. Litmus paper can be part of an acid-in-a-beaker system, and respond by turning blue. The mind is like litmus paper, but instead of turning colors, it responds to its surroundings by experiencing them. If something gives us grounds to be happy, the mind-body system (the human being) becomes happy, and the mind experiences happiness. Happiness has mental and physical consequences. You might experience a rush of energy, even quickened pulse and breathing.

Why can't being be computed? Why can't happiness? Happiness is not computable because, being the state of a physical object, it is outside the universe of computation. Computers and software do not create or manipulate physical stuff. (They can cause other, attached machines to do that, but what those attached machines do is not the accomplishment of computers. Robots can fly but computers can't. Nor is any computer-controlled device guaranteed to make people happy; but that's another story.) Being is not computable: an important fact that has been overlooked until now—not surprisingly. Computers and the mind live in different universes, like pumpkins and Puccini, and are hard to compare whatever one intends to show.

Can we get by without being, and still have a thinking machine? No. Thinking-about and being (or feeling) define the mind and its capacities. At the spectrum's top—at maximum alertness or focus—the mind throws itself into thinking-about and fends off emotion, which is distracting. At the bottom is sleep-and-dreaming, a state in which we do little thinking; we are preoccupied by sensation as we hallucinate, and often by emotion (dreams can be strongly emotional); in any event with feeling, or in other words, being.

Why did it take so long to produce such a simple argument? (And why are so few thinkers likely to accept it now?) Maybe because most philosophers and scientists wish that the mind were nothing but thinking, and that feeling or being played no part. They wished so hard for it to be true, they finally decided it was. Philosophers are only human.

The widespread fear that AI will endanger humanity and take over the world is irrational. Here is why.

Conceptually, autonomous or artificial intelligence systems can develop two ways: either as an extension of human thinking or as radically new thinking. Call the first "Humanoid Thinking" (or "Humanoid AI") and the second one "Alien Thinking" (or "Alien AI").

Almost all AI today is Humanoid Thinking. We use AI to solve problems that are too difficult, time consuming or boring for our limited human brains to process: electrical grid balancing, recommendation engines, self-driving cars, face recognition, trading algorithms, and the like. These artificial agents work in narrow domains with clear goals that their human creators specify. Such AI aims to accomplish human objectives—often better, with fewer cognitive errors, fewer distractions, fewer outbursts of bad temper and fewer processing limitations. In a couple of decades, AI agents might serve as virtual insurance sellers, doctors, psychotherapists, and maybe even virtual spouses and children.

We will achieve much of this, but such AI agents will be our slaves with no self-concept of their own. They will happily perform the functions we set them up to enact. If screw-ups happen, they will be our screw-ups due to software bugs or overreliance on these agents (Daniel C. Dennett's point). Yes, Humanoid AIs might surprise us every once in a while with novel solutions to specific optimization problems. But in most cases novel solutions are the last thing we want from AI (creativity in the navigation of nuclear missiles, anyone?). That said, Humanoid AI's solutions will always fit a narrow domain. These solutions will be understandable, either because we understand what they achieve or because we understand their inner workings. In some cases, the code will become too enormous and fumbled for one person to understand because it is continuously patched. In these cases we can turn it off and start programming a more elegant version. Humanoid AI will bring us closer to the age-old aspiration of having robots do most of the work while humans are free to be creative—or to be amused to death.

Alien Thinking is radically different. Alien Thinking could conceivably become a danger to Humanoid Thinking; it could take over the planet, outsmart us, outrun us, enslave us—and we might not even recognize the onslaught. What sort of thinking will Alien Thinking be? By definition, we can't tell. It will encompass functionality that we cannot remotely understand. Will it be conscious? Most likely, but it need not be. Will it experience emotion? Will it write bestselling novels? If so, bestselling to us or bestselling to it and its spawn? Will cognitive errors mar its thinking? Will it be social? Will it have a theory of mind? If so, will it make jokes, will it gossip, will it worry about its reputation, will it rally around a flag? Will it create its own version of AI (AI-AI)? We can't say.

All we can say is that humans cannot construct truly Alien Thinking. Whatever we create will reflect our goals and values, so it won't stray far from human thinking. You'd need real evolution, not just evolutionary algorithms, for self-aware Alien Thinking to arise. You'd need an evolutionary path radically different from the one that led to human intelligence and Humanoid AI.

So, how do you get real evolution to kick in? Replicators, variation and selection. Once these three components are in place, evolution arises inevitably. How likely is it that Alien Thinking will evolve? Here is a back-of-the-envelope calculation:

First consider what getting from magnificently complex eukaryotic cells to human-level thinking involved. Achieving human thought required a large portion of the Earth's biomass (roughly 500 billion tons of eukaryotically bound carbon) during approximately two billion years. That's a lot of evolutionary work! True, human-level thinking might have happened in half the time. With a lot of luck, even in 10% of the time (that's 200 million years), but it's unlikely to have happened any faster. Remember, you don't need only massive amounts of time for evolution to generate complex behavior, you also need a petri dish the size of Earth's surface to sustain this level of experimentation.

Assume that Alien Thinking will be silicon-based, as all current AI is. A eukaryotic cell is vastly more complex than, say, Intel's latest i7 CPU chip—both in hardware and software. Further assume that you could shrink that CPU chip to the size of a eukaryote. Leave aside the quantum effects that would stop the transistors from working reliably. Leave aside the question of the energy source. You would have to cover the globe with 10^30 microscopic CPUs and let them communicate and fight for two billion years for true thought to emerge.

Yes, processing speed is faster in CPUs than in biological cells, because electrons are easier to shuttle around than atoms. On the other hand, eukaryotes work massively parallel, whereas Intel's i7 works only four times parallel (4 cores). Eventually, at least to dominate the world, these electrons would need to move atoms to store their software and data in more and more physical places. This necessity will slow their evolution dramatically. It's hard to say if, overall, silicon evolution will be faster than biological. We don't know enough about it. I don't see a reason why this sort of evolution would be more than two or three orders of magnitude faster than biological evolution (if at all)—which would bring the emergence of self-aware Alien AI down to roughly a million years.

What if Humanoid AI becomes so smart it could create Alien AI from the top down? That is where Orgel's Second Rule kicks in: "Evolution is smarter than you are." It's smarter than human thinking. It's even smarter than humanoid thinking. And, it's much slower than you think.

Thus, the danger of AI is not inherent to AI, but rests on our over-reliance on it. Artificial Thinking is not going to evolve to self-awareness in our lifetime. In fact, it's not going to happen in literally a thousand years.

I might be wrong, of course. After all, this back-of-the-envelope calculation applies legacy human thinking to Alien AI—which, by definition, we won't understand. But that's all we can do at this stage.

Toward the end of the 1930s, Samuel Beckett wrote in a diary, "We feel with terrible resignation that reason is not a superhuman gift…that reason evolved into what it is, but that it also, however, could have evolved differently." Replace "reason" with "AI" and you have my argument.

Asking what I think about thinking machines is like asking what I think about gravity. Thinking machines exist, and are the most recent developments of a human tradition that began over 5,000 years ago with the introduction of static external memory aids such as cuneiform tablets and quipu. These storage devices recorded mostly numerical information that supported routine decision-making. Over the centuries we developed more sophisticated and diverse objects and machines to undertake computation and store numerical and narrative information. We human beings are not only incessant communicators, but we have voracious appetites for "data." The introduction of binary code and its automation in computers made it possible for us to record, store, and manipulate all types of information, and we have continued to make technological advances in this realm in typical human fashion, that is, mostly hell-bent on novelty and oblivious to the consequences. We are ever more relying on thinking machines to store, translate, manipulate, and interrogate vast quantities of data. These devices are now supporting not-so-routine decision-making every day in medicine, law, and engineering, and are augmenting the creative processes of making music, writing poetry, and generating visual imagery. Raw combinatorial power allows modern thinking machines to learn from experience and, in the foreseeable future, this ability will be supported by human effort as the machines self-duplicate, mutate, establish ever-more complex networks of intercommunication, and eventually perform eugenics on themselves.

The same people who worry about thinking machines today were certain that the introduction of calculators 50 years ago would usher in an era of knuckle-dragging imbecility. That isn't what we have today, and it won't be what we have in the future. Thinking machines are liberating us from the banalities of routine data storage and manipulation, and are making it possible for us to enter a new phase of human evolution. Only real people with mushy gray-pink neuronal circuitry are able to undertake the quintessentially human activities of introspection and reflection upon the nature of existence. The dense and uneven networks of interconnecting neurons in our brains vary greatly from one person to another, and are remodeled from one thought-moment to the next so that no two individuals are ever alike, no day is ever the same, and no memory is ever recalled in the same way. By automating many routine physical and mental tasks, and reducing our need for laborious, recursive searching, machines that think are freeing us from much of the physical wear and tear and intellectual tedium of earlier phases of our history. We can now think much more what it means to think, to dream, to make jokes, and to cry. We can reflect on the meaning of the "human spirit," the origins of self-sacrifice, and the emergent qualities of thousands of people coming together to witness events, share each other's company, and celebrate a common humanity. These are not trivial superfluities, they are the essence of the human condition. Machines that think make it possible for more people to celebrate the joy of human intuitive insight, and to cultivate the equanimity that is unique to the self-controlled human mind.

Beyond the realms of serious science and technology the popular debates about machines that think have been high masses of a new mythology. There are two main dogmas. One is the hope that a moment of singularity will awaken a synthetic spirit superior to the human mind. The other is the fear that thinking machines will dominate and ultimately destroy mankind. Both distract from the fact that at the heart of the debate is a very real John Henry moment.

In the folk tale of the late 19^th century the mythical steel driving man John Henry dies beating a steam powered hammer during a competition to drill blast holes into a West Virginian mountainside. White collar and knowledge workers now face a race against being outperformed by machines driven by artificial intelligence. In this case AI is mainly a synonym for new levels of mainly digital productivity. Which is of course not quite as exciting as either waiting for the moment of singularity or the advent of doom. At the same time the reality of AI is not quite as comforting as the realization that machines, if properly handled, will always serve their masters.

Dystopian views of AI as popularized by movies and novels are just misleading. Those debates are rarely about science and technology. They tend to be mostly humans debating the nature of themselves. Most of the endless variations of imaginary machine rule tend to project the fear of inherent evil and cruelty into machines as proxies for the age-old uncontrollable urges of self-empowerment and unlimited progress.

Elevating the AI debate to hopes with theological dimensions is turning optimism about technological progress into a salvation theory. As confirmed again and again the likelihood of a synthetic spirit is nil. Artificial intelligence might be the most rapid advancement of complexity in science and technology. So far it still mimics human nature and it will remain doing so. For one it lacks time. Human intelligence is the product of evolution. AI does not have the luxury of a trial and error phase of billions of years. To believe in a coming moment of singularity, when AI transcends human control and advances to surpass human intelligence is nothing more than the belief in a technological rapture. This might be a popular belief in insular worlds like Silicon Valley. AI reality is different. And it's here.

AI has already touched billions of people in profound ways. So far the main impact of AI is the comfort of an ever increasing number of digital aids. Calculating consumer choices, behavior patterns and even market shifts might still belong more to the realm of statistics than intelligent life. Still even these crude forms of AI should neither be over- or underestimated, even if the real John Henry moment has not yet arrived. Working masses have always been replaceable by efficiency measures or cheaper labor. And no labor is cheaper and more efficient than the one by machines. Just like the steam hammer in John Henry's tale most digital tools will outperform humans in highly specialized tasks. So of course there will still be demand of high-skills and outstanding talent. No computer will ever replace a scientist, an artist, an innovator. It's the mid-level white collar or knowledge worker who will fall behind.

As AI's efficiencies and skill sets increase, they also become tools of power. The tedious skills of surveillance, warfare and torture can already be performed much better by an entity that is neither prone to emotions, conflicted values or fatigue. Still the danger that hostile or even lethal machines will develop an evil consciousness and turn against mankind is nil. It is institutions and organizations that will use them for whatever benign or sinister objective.

There is no need for a superior intelligence to turn abstract debates about AI into very real questions of power, values and societal changes. Technology can initiate and advance historical shifts. It will never be the shift itself. The John Henry moment of the 21^st century will neither be heroic nor entertaining. There are no grand gestures with which white collar and knowledge workers can go down fighting. There will be no folk heroes dying in the office park. Today's John Henry will merely fade into a sad statistic. Undoubtedly calculated by a skillfully thinking machine.

What do I think about machines that think?  Well, of course it depends on who that person is.

At what point do we say a machine can think? When it can calculate things, when it can understand contextual cues and adjust its behaviour accordingly, when it can both mimic and evoke emotions? I think the answer to the overall question depends on what we mean by thinking. There are plenty of conscious (system two) processes that a machine can do better more accurately with less bias than we can. There are already people investing in developing AI machines to replace stock traders—the first time anyone has ever thought about mechanising a white collar job. But a machine cannot think in an automatic (system one) way—we don't fully understand the automatic processes that drive the way we behave and "think" so we cannot programme a machine to behave as humans do

The key question then is—if a machine can think in a system two way at the speed of a human's system one then in some ways isn't their "thinking" superior to ours?

Well, context surely matters. For some things yes; others no. Machines won't be myopic; they could clean things up for us environmentally; they wouldn't be stereotypical or judgmental and could really get at addressing misery; they could help us overcome affective forecasting; and so on. But on the other hand, we might still not like a computer. What if a poet and a machine could produce the exact same poem—the effect on another human being is almost certainly less if the poem is computer generated and the reader knows this (knowledge of the author colours the lens through which the poem is read and interpreted). 

The potential of advanced AI, and concerns about it downsides, are rising on the agenda—and rightly. Many of us think that the AI field, like synthetic biotech, already needs guidelines that promote "responsible innovation"; others regard the most-discussed scenarios as too futuristic to be worth worrying about.

But the divergence of view is basically about the timescale—assessments differ with regard to the rate of travel, not the direction of travel. Few doubt that machines will surpass more and more of our distinctively human capabilities—or enhance them via cyborg technology. The cautious amongst us envisage timescales of centuries rather than decades for these transformations. Be that as it may, the timescales for technological advance are but an instant compared to the timescales of the Darwinian selection that led to humanity's emergence—and (more relevantly) they are less than a millionth of the vast expanses of time lying ahead. That's why, in a long-term evolutionary perspective, humans and all they've thought will be just a transient and primitive precursor of the deeper cogitations of a machine-dominated culture extending into the far future, and spreading far beyond our Earth.

We're now witnessing the early stages of this transition. It's not hard to envisage a "hyper computer" achieving oracular powers that could offer its controller dominance of international finance and strategy—this seems only a quantitative (not qualitative) step beyond what "quant" hedge funds do today. Sensor technologies still lag behind human capacities. But when robots can observe and interpret their environment as adeptly as we do they would truly be perceived as intelligent beings, to which (or to whom) we can relate, at least in some respects, as we to other people. We'd have no more reason to disparage them as zombies than to regard other people in that way.

Their greater processing speed may give robots an advantage over us. But will they remain docile rather than "going rogue"? And what if a hyper-computer developed a mind of its own? If it could infiltrate the Internet—and the Internet of things—it could manipulate the rest of the world. It may have goals utterly orthogonal to human wishes—or even treat humans as an encumbrance. Or (to be more optimistic) humans may transcend biology by merging with computers, maybe subsuming their individuality into a common consciousness. In old-style spiritualist parlance, they would "go over to the other side."

The horizons of technological forecasting rarely extend even a few centuries into the future—and some predict transformational changes within a few decades. But the Earth has billions of years ahead of it, and the cosmos a longer (perhaps infinite) future. So what about the posthuman era—stretching billions of years ahead?

There are chemical and metabolic limits to the size and processing power of "wet" organic brains. Maybe we're close to these already. But no such limits constrain silicon-based computers (still less, perhaps, quantum computers): for these, the potential for further development could be as dramatic as the evolution from monocellular organisms to humans.

So, by any definition of "thinking," the amount and intensity that's done by organic human-type brains will be utterly swamped by the cerebrations of AI. Moreover, the Earth's biosphere in which organic life has symbiotically evolved, is not a constraint for advanced AI. Indeed it is far from optimal—interplanetary and interstellar space will be the preferred arena where robotic fabricators will have the grandest scope for construction, and where non-biological "brains" may develop insights as far beyond our imaginings as string theory is for a mouse.

Abstract thinking by biological brains has underpinned the emergence of all culture and science. But this activity—spanning tens of millennia at most—will be a brief precursor to the more powerful intellects of the inorganic post-human era. Moreover, evolution on other worlds orbiting stars older than the Sun could have had a head start. If so, then aliens are likely to have long ago transitioned beyond the organic stage.

So it won't be the minds of humans, but those of machines, that will most fully understand the world—and it will be the actions of autonomous machines that will most drastically change the world, and perhaps what lies beyond.

Recent months have seen an increasingly public debate taking form around the risks of AI (Artificial Intelligence) and in particular AGI (Artificial General Intelligence). A letter signed by Nobel prizewinners and other physicists defined AI as the top existential risk to mankind. The robust conversation that has erupted among thoughtful experts in the field has, as yet, done little to settle the debate.

I am arguing here that research on how we think and how to make machines that think is good for society. I call for research that integrates cognitive science, neuroscience, computer science, and artificial intelligence. Understanding intelligence and replicating it in machines, goes hand in hand with understanding how the brain and the mind perform intelligent computations.

The convergence and recent progress in technology, mathematics, and neuroscience has created a new opportunity for synergies across fields. The dream of understanding intelligence is an old one. Yet, as the debate around AI shows, this is now an exciting time to pursue this vision. We are at the beginning of a new and emerging field, the Science and Engineering of Intelligence, an integrated effort that I expect will ultimately make fundamental progress with great value to science, technology, and society. I believe that we must push ahead with this research, not pull back.

A top priority for society

The problem of intelligence—what it is, how the human brain generates it and how to replicate it in machines—is one of the great problems in science and technology, together with the problem of the origin of the universe and of the nature of space and time. It may be the greatest of all because it is the one with a large multiplier effect—almost any progress on making ourselves smarter or developing machines that help us think better, will lead to advances in all other great problems of science and technology.

Research on intelligence will eventually revolutionize education and learning. Systems that recognize how culture influences thinking could help avoid social conflict. The work of scientists and engineers could be amplified to help solve the world's most pressing technical problems. Mental health could be understood on a deeper level to find better ways to intervene. In summary, research on intelligence will help us understand the human mind and brain, build more intelligent machines, and improve the mechanisms for collective decisions. These advances will be critical to future prosperity, education, health, and security of our society. This is the time to greatly expand research on intelligence, not the time to withdraw from it.

Thoughts on machines that think

We are often misled by "big", somewhat ill-defined, long used words. Nobody so far has been able to give a precise, verifiable definition of what general intelligence or what thinking is. The only definition I know that, though limited, can be practically used is Turing's. With his test, Turing provided an operational definition of a specific form of thinking—human intelligence.

Let us then consider human intelligence as defined by the Turing test. It is becoming increasingly clear that there are many facets of human intelligence. Consider for instance a Turing test of visual intelligence—that is questions about an image, a scene. Questions may range from what is there to who is there, what is this person doing, what is this girl thinking about this boy and so on. We know by now from recent advances in cognitive neuroscience, that answering these questions requires different competences and abilities, often rather independent from each other, often corresponding to separate modules in the brain.

For instance, the apparently very similar questions of object and face recognition (what is there vs who is there) involve rather distinct parts of visual cortex. The word intelligence can be misleading in this context, like the word life was during the first half of the last century when popular scientific journals routinely wrote about the problem of life, as if there was a single substratum of life waiting to be discovered to completely unveil the mystery.

Of course, speaking today about the problem of life sounds amusing: biology is a science dealing with many different great problems, not just one. Intelligence is one word but many problems, not one but many Nobel prizes. This is related to Marvin Minsky's view of the problem of thinking, well captured by his slogan "Society of Minds". In the same way, a real Turing test is a broad set of questions probing the main aspects of human thinking. For this reason, my colleagues and I are developing the framework around an open-ended set of Turing+ questions in order to measure scientific progress in the field. The plural in questions is to emphasize that there are many different intelligent abilities that have to be characterized, and possibly replicated in a machine, from basic visual recognition of objects, to the identification of faces, to gauge emotions, to social intelligence, to language and much more.

The term Turing+ is to emphasize that a quantitative model must match human behavior and human physiology—the mind and the brain. The requirements are thus well beyond the original Turing test. An entire scientific field is required to make progress on understanding them and to develop the related technologies of intelligence.

Should we be afraid of machines that think?

Since intelligence is a whole set of solutions to rather independent problems, there is little reason to fear the sudden appearance of a super-human machine that think, though it is always better to err on the side of caution. Of course, each of the many technologies that are emerging and will emerge over time in order to solve the different problems of intelligence, is likely to be powerful in itself and therefore potentially dangerous in its use and misuse, like most technologies are.

Thus, as it is the case in other parts of science, proper safety measures and ethical guidelines should be in place. In addition, there is probably the need for constant monitoring—perhaps by an independent supernational organization—of the supralinear risk created by the combination of continuously emerging technologies of intelligence. All in all, however, not only I am not afraid of machines that think but I find their birth and evolution one of the most exciting, interesting and positive events in the history of human thought.

The advent of quantum biology, light harvesting molecules, bird navigation, perhaps smell, suggests that sticking to classical physics in biology may turn out to be simply stubborn. Now Turing machines are discrete state, (0,1), discrete time T T+1, subsets of classical physics and define algorithmic. We all know they, like Shannon information, are merely syntactic. Wonderful mathematical results such as Chaitin’s Omega, the probability a program will halt which is totally non-computable and non-algorithmic tell us the human mind, as Penrose also argued, cannot be merely algorithmic.

Mathematics is creative. So is the human mind. We understand metaphors, "Tomorrow and tomorrow and tomorrow creep at this petty pace ..." but metaphors are not even true or false. All art is metaphoric, language started gestural or metaphoric, we live by these, not merely by true false propositions and the syllogisms they enable. No prestated set of propositions can exhaust the meanings of a metaphor and if mathematics requires propositions, no mathematics can prove that no prestated set of propositions can exhaust the meanings of a metaphor. Thus the human mind, in Pierce’s "abduction", not induction or deduction, is wildly creative in unprestatable ways. 

The causal closure of classical physics precludes more than an epiphenomenal mind that cannot "act" on the world, be it a Turing machine or billiard balls, or classical physics neurons. The current state of the brain suffices to determine the next state of the brain (or computer) so there is nothing for mind to do and no way for mind to do it! We’ve be frozen in this stalemate since Newton defeated Descartes’ Res cogitans. 

Ontologially, free choice requires that the present could have been different, a counterfactual claim impossible in classical physics, but easy if quantum measurement is real and indeterminate: the electron could have been measured to be spin up or measured to be spin down, so the present could have been different. 

A quantum mind however, seems to obviate responsible free will. False, for given N entangled particles, the measurement of each alters the probabilities, by the Born rule, of the outcomes of the next measurements. In one extreme these may vary from 100% spin up on the first to 100% spin down on the second and so on for N measurements, entirely non random and free if measurement is ontologically indeterminate If probabilities of N entangled particles vary between less than 100% and 0% we get choice and an argument suggest we can get responsible choice in the "Strong Free Will Theorem" of Conway and Kochen. 

We will never get to the subjective pole from third person descriptions. But a single rod can absorb a single photon so it is conceivable to test if human consciousness can be sufficient for quantum measurement. If we were so persuaded, and if the classical world is at base quantum then the easy hypothesis is that quantum variables consciously measure and choose, as Penrose and Hameroff in "Orch Or" theory and others suggest. We may live in a wildly participatory universe, consciousness and will may be part of its furniture, and Turing machines cannot, as subsets of classical physics and merely syntactic, make choices where the present could have been different. 

While machines are terrific at computing, this issue is that they're not very good at actual thinking.

Machines have an endless supply of grit and perseverance, and, as others have said, will effortlessly crunch out the answer to a complicated mathematical problem or direct you through traffic in an unknown city, all by use of the algorithms and programs installed by humans. But what do machines lack?

Machines (at least so far, and I don’t think this will change with a singularity) lack vision. And I don’t mean sight. Machines do not devise the next new killer app on their own. Machines don’t decide to explore distant galaxies—they do a terrific job once we send them, but that’s a different story. Machines are certainly better than the average person at solving problems in calculus and quantum mechanics—but machines don’t have the vision to see the need for such constructs in the first place. Machines can beat humans at chess—but they have yet to design the type of mind game that will intrigue humans for centuries. Machines can see statistical regularities that my feeble brain will miss—but they can’t make the insightful leap that connects entirely disparate sets of data to devise a new field.

I am not too terribly concerned about machines that compute—I’ll deal with the frustration of my browser in exchange for a smart refrigerator that, based on tracking RFID codes of what comes in and out, texts me to buy cream on my way home (hint to those working on such a system…sooner rather than later!). I like having my computer underline words it doesn’t recognize, and I’ll deal with the frustration of having to ignore its comments on "phylogenetic" in exchange for catching my typo on a common term (in fact, it won’t let me misspell a word here to make a point). But these examples show that just because a machine is going through the motions of what looks like thinking doesn’t mean that it actually is engaging in that behavior—or at least one equivalent to the human process.

I am reminded of one of the earliest studies to train apes to use "language"—in this case, to manipulate plastic chips to answer a number of questions. The system was replicated with college students, who did exceptionally well—not surprisingly—but when asked about what they had been trained to do, claimed that they had solved some interesting puzzles, and that they had no idea that they were being taught a language. Much debate ensued, and much was learned—and put into practice—in subsequent studies so that several nonhuman subjects did eventually understand the referential meaning of the various symbols that they were taught to use, and we did learn a lot about ape intelligence from the original methodology. The point, however, is that what initially looked like a complicated linguistic system needed a lot more work before it became more than a series of (relatively) simple paired associations.

My concern therefore is not about thinking machines, but rather about a complacent society—one that might give up on its visionaries in exchange merely for getting rid of drudgery. Humans need to take advantage of all the cognitive capacity that is released when machines take over the scut work—and be so very thankful for that release, and use that release—to channel all that ability into the hard work of solving pressing problems that need insightful, visionary leaps. 

Thinking about "machines that think" may constitute a classic reversal of figure and ground, medium and message. It sets us up to think about the next stage of intelligence as something that is happening in a computer somewhere - an awareness that will be born and then housed on the tremendous servers being built by information age corporations for this purpose. "There it is," we will declare and point, "the intelligent machine."

Our mistake, as creatures of the electronic age and mere immigrants to unfolding digital era, is to see digital technology as a subject rather than a landscape. It's the same as confusing the television set with the media environment created by the television set, or the little smart phone in your pocket with the greater impact of handheld communications and computing technology on our society. 

This happens whenever we undergo a media transition. So we can't help but see digital technology as figure, when it is actually the ground. It is not the source of future intelligence but an environment where intelligence manifests differently. So while technologists may feel like they are creating a cathedral for the mechanical mind, they are actually succumbing to an oversimplified, industrial age approach to digital consciousness. 

Rather than machines that think, I believe we are migrating toward a networked environment in which thinking is no longer a individual activity, nor bound by time and space. This means we can think together at the same time, or asynchronously through digital representations of previous and future human thoughts. Even the most advanced algorithm amounts to the iteration of a "what if" once posed by a person. And even then, machine thinking is not something that happens apart from this collective human thinking, because it is not a localized, brain-like activity. 

When we can wrest that television-like image from our collective psyche, we will be in a position to recognize the machine environment in which we are already thinking together. Artificial intelligence will constitute the platform or territory where this takes place - so what we program into it will to a very large extent determine what we strive for, and what we even deem possible. 

No individual, deterministic machine, however universal this class of machines is proving to be, will ever think in the sense that we think. Intelligence may be ever-increasing among such machines, but genuinely creative intuitive thinking requires non-deterministic machines that can make mistakes, abandon logic from one moment to the next, and learn. Thinking is not as logical as we think.

Non-deterministic machines, or, better yet, non-deterministic networks of deterministic machines, are a different question. We have at least one existence proof that such networks can learn to think. And we have every reason to suspect that, once invoked within an environment without the time, energy, and storage constraints under which our own brains operate, this process will eventually lead, as Irving (Jack) Good first described it, to "a machine that believes people cannot think." 

Until digital computers came along, nature used digital representation (as coded strings of nucleotides) for information storage and error correction, but not for control. The ability to introduce one-click modifications to instructions, a useful feature for generation-to-generation evolutionary mechanisms, becomes a crippling handicap for controlling day-to-day or millisecond-to-millisecond behavior in the real world. Analog processes are far more robust when it comes to real-time control. 

We should be less worried about having our lives (and thoughts) controlled by digital computers and more worried about being controlled by analog ones. Machines that actually think for themselves, as opposed to simply doing ever-more-clever things, are more likely to be analog than digital, although they may be analog devices running as higher-level processes on a substrate of digital components, the same way digital computers were invoked as processes running on analog components, the first time around.

We are currently in the midst of an analog revolution, but for some reason it is a revolution that dares not speak its name. As we enter the seventh decade of arguing about whether digital computers can be said to think, we are surrounded by an explosive growth in analog processes whose complexity and meaning lies not in the state of the underlying devices or the underlying code but in the topology of the resulting networks and the pulse frequency of connections. Streams of bits are being treated as continuous functions, the way vacuum tubes treat streams of electrons, or neurons treat pulse frequencies in the brain. 

Bottom line: I know that analog computers can think. I suspect that digital computers, too, may eventually start to think, but only by growing up to become analog computers, first. 

Real artificial intelligence will be intelligent enough to not reveal itself. Things will go better if people have faith rather than proof.

"You can't think about thinking without thinking about thinking about something". —Seymour Papert

What do I think about machines that think? It depends on what they're supposed to be thinking about. I am clearly in the camp of people who believe that AI and machine learning will contribute greatly to society. I expect that we'll find machines to be exceedingly good at things that we're not—things that involve massive amounts of data, speed, accuracy, reliability, obedience, computation, distributed networking and parallel processing.

The paradox is that at the same time we've developed machines that behave more and more like humans, we've developed educational systems that push children to think like computers and behave like robots. It turns out that for our society to scale and grow at the speed we now require, we need reliable, obedient, hardworking, physical and computational units. So we spend years converting sloppy, emotional, random, disobedient human beings into meat-based versions of robots. Luckily, mechanical and digital robots and computers will soon help reduce if not eliminate the need for people taught to behave like them.

We'll still need to overcome the fear and even disgust evoked when robot designs bring us closer and closer to the "uncanny valley," in which robots and things demonstrate almost-human qualities without quite reaching them. This is true for computer animation, zombies and even prosthetic hands. But we may be approaching the valley from both ends. If you've ever modified your voice to be understood by a voice-recognition system on the phone, you understand how, as humans, we can edge into the uncanny valley ourselves.

There are a number of theories about why we feel this revulsion, but I think it has something to with human beings feeling they're special—a kind of existential ego. This may have monotheistic roots. Right around the time Western factory workers were smashing robots with sledgehammers, Japanese workers were putting hats on the same robots in factories and giving them names. On April 7, 2003, Astro Boy, the Japanese robot character, was registered as a resident of the city of Niiza, Saitama.

If these anecdotes tell us anything, it's that animist religions may have less trouble dealing with the idea that maybe we're not really in charge. If nature is a complex system in which all things—humans, trees, stones, rivers and homes—are all animated in some way and all have their own spirits, then maybe it's okay that God doesn't really look like us or think like us or think that we're really that special.

So perhaps one of the most useful aspects of being alive in the period where we begin to ask this question is that it raises a larger question about the role of human consciousness. Human beings are part of a massively complex system—complex beyond our comprehension. Like the animate trees, stones, rivers and homes, maybe algorithms running on computers are just another part of this complex ecosystem.

As human beings we have evolved to have an ego and believe that there such a thing as a self, but mostly, that's a self-deception to allow each human unit to work within the parameters of evolutionary dynamics in a useful way. Perhaps the morality that emerges from it is a self-deception of sorts, as well. For all we know, we might just be living in a simulation where nothing really actually matters. It doesn't mean we shouldn't have ethics and good taste. I just think we can exercise our sense of responsibility in being part of a complex and interconnected system without having to rely on an argument that "I am special." As machines become an increasingly important part of these systems, their prominence will make human arguments about being special increasingly fraught. Maybe that's a good thing.

Perhaps what we think about machines that think doesn't really matter—they will "think" and the system will adapt. As with most complex systems, the outcome is mostly unpredictable. It is what it is and will be what it will be. Most of what we think is going to happen is probably hopelessly wrong and as we know from climate change, knowing that something is happening and doing something about it often have little in common.

That might sound extremely negative and defeatist, but I'm actually quite optimistic. I believe that the systems are quite adaptive and resilient and that whatever happens, beauty, happiness and fun will persist. Hopefully, human beings will have a role. My guess is that they will.

It turns out that we don't make great robots, but we're very good at doing random and creative things that would be impossibly complex—and probably a waste of resources—to code into a machine. Ideally, our educational system will evolve to more fully embrace our uniquely human strengths, rather than trying to shape us into second-rate machines. Human beings—though not necessarily our current form of consciousness and the linear philosophy around it—are quite good at transforming messiness and complexity into art, culture, and meaning. If we focus on what each of us is best at, I think that humans and machines will develop a wonderful yin-yang sort of relationship, with humans feeding off of the efficiency of our solid-state brethren, while they feed off of our messy, sloppy, emotional and creative bodies and brains.

We are descending not into chaos, as many believe, but into complexity. At the same time that the Internet connects everything outside of us into a vast, seemingly unmanageable system, we find an almost infinite amount of complexity as we dig deeper inside our own biology. Much as we're convinced that our brains run the show, all while our microbiomes alter our drives, desires, and behaviors to support their own reproduction and evolution, it may never be clear who's in charge—us, or our machines. But maybe we've done more damage by believing that humans are special than we possibly could by embracing a more humble relationship with the other creatures, objects, and machines around us.

Not much, other than the fact that they serve, as Dan Dennett has noted, as a useful existence proof that thought does not require some mystical, extra "something" that mind-body dualists continue to embrace.       

In fact, I've always been a bit baffled by fears about AI machines taking over the world, which seem to me to be based on a fundamental—though natural—intellectual mistake. When conceptualizing a super-powerful Machine That Can Think, we draw upon the best analogy that we have at hand: us. So we tend to think of AI systems as just like us, only much smarter and faster.

This is, however, a bad analogy. A better one would be a really powerful, versatile screwdriver. No one worries about super-advanced screwdrivers rising up and overthrowing their masters. AI systems are tools, not organisms. No matter how good they become at diagnosing diseases, or vacuuming our living rooms, they don't actually want to do any of these things. We want them to, and we then build these "wants" into them.

It's also a category mistake to ask what Machines That Can Think might be thinking about. They aren't thinking about anything—the "aboutness" of thinking derives from the intentional goals driving the thinking. AI systems, in and of themselves, are entirely devoid of intentions or goals. They have no emotions, they feel neither empathy nor resentment. While such systems might some day be able to replicate our intelligence—and there seems to be no a priori reason why this would be impossible—this intelligence would be completely lacking in direction, which would have to be provided from the outside.

This is because motivational direction is the product of natural selection working on biological organisms. Natural selection produced our rich and complicated set of instincts, emotions and drives in order to maximize our ability to get our genes into the next generation, a process that has left us saddled with all sorts of goals, including desires to win, to dominate, and to control. While we may want to win, for perfectly good evolutionary reasons, machines could care less. They just manipulate 0s and 1s, as programmed to do by the people who want it to win. Why on earth would an AI system want to take over the world? What would it do with it?

What is scary as hell is the idea an entity possessed of extra-human intelligence and speed and our motivational system—in other words, human beings equipped with access to powerful AI systems. But smart primates with nuclear weapons are just as scary, and we've managed to survive such a world so far. AI is no more threatening in and of itself than a nuclear bomb—it is a tool, and the only thing to be feared are the creators and wielders of such tools.

Julien de La Mettrie would be classified as a quintessential New Atheist, except for the fact that there's not much New about him by now. Writing in eighteenth-century France, La Mettrie was brash in his pronouncements, openly disparaging of his opponents, and boisterously assured in his anti-spiritualist convictions. His most influential work, L'homme machine (Man a Machine), derided the idea of a Cartesian non-material soul. A physician by trade, he argued that the workings and diseases of the mind were best understood as features of the body and brain.

As we all know, even today La Mettrie's ideas aren't universally accepted, but he was largely on the right track. Modern physics has achieved a complete list of the particles and forces that make up all the matter we directly see around us, both living and non-living, with no room left for extra-physical life forces. Neuroscience, a much more challenging field and correspondingly not nearly as far along as physics, has nevertheless made enormous strides in connecting human thoughts and behaviors with specific actions in our brains. When asked for my thoughts about machines that think, I can't help but reply: Hey, those are my friends you're talking about. We are all machines that think, and the distinction between different types of machines is eroding.

We pay a lot of attention these days, with good reason, to "artificial" machines and intelligences—ones constructed by human ingenuity. But the "natural" ones that have evolved through natural selection, like you and me, are still around. And one of the most exciting frontiers in technology and cognition is the increasingly permeable boundary between the two categories.

Artificial intelligence, unsurprisingly in retrospect, is a much more challenging field than many of its pioneers originally supposed. Human programmers naturally think in terms of a conceptual separation between hardware and software, and imagine that conjuring intelligent behavior is a matter of writing the right code. But evolution makes no such distinction. The neurons in our brains, as well as the bodies through which they interact with the world, function as both hardware and software. Roboticists have found that human-seeming behavior is much easier to model in machines when cognition is embodied. Give that computer some arms, legs, and a face, and it starts acting much more like a person.

From the other side, neuroscientists and engineers are getting much better at augmenting human cognition, breaking down the barrier between mind and (artificial) machine. We have primitive brain/computer interfaces, offering the hope that paralyzed patients will be able to speak through computers and operate prosthetic limbs directly.

What's harder to predict is how connecting human brains with machines and computers will ultimately change the way we actually think. DARPA-sponsored researchers have discovered that the human brain is better than any current computer at quickly analyzing certain kinds of visual data, and developed techniques for extracting the relevant subconscious signals directly from the brain, unmediated by pesky human awareness. Ultimately we'll want to reverse the process, feeding data (and thoughts) directly to the brain. People, properly augmented, will be able sift through enormous amounts of information, perform mathematical calculations at supercomputer speeds, and visualize virtual directions well beyond our ordinary three dimensions of space.

Where will the breakdown of the human/machine barrier lead us? Julien de La Mettrie, we are told, died at the young age of 41, after attempting to show off his rigorous constitution by eating an enormous quantity of pheasant pâte with truffles. Even leading intellects of the Enlightenment sometimes behaved irrationally. The way we think and act in the world is changing in profound ways, with the help of computers and the way we connect with them. It will be up to us to use our new capabilities wisely.

I think that humans think because memes took over our brains and redesigned them. I think that machines think because the next replicator is doing the same. It is busily taking over the digital machinery that we are so rapidly building and creating its own kind of thinking machine.

Our brains, and our capacity for thought, were not designed by a great big intelligent designer in the sky who decided how we should think and what our motivations should be. Our intelligence and our motivations evolved. Most (probably all) AI researchers would agree with that. Yet many still seem to think that we humans are intelligent designers who can design machines that will think the way we want them to think and have the motivations we want them to have. If I am right about the evolution of technology they are wrong.

The problem is a kind of deluded anthropomorphism: we imagine that a thinking machine must work the way that we do, yet we so badly mischaracterise ourselves that we do the same with our machines. As a consequence we fail to see that all around us vast thinking machines are evolving on just the same principles as our brains once did. Evolution, not intelligent design, is sculpting the way they will think.

The reason is easy to see and hard to deal with. It is the same dualism that bedevils the scientific understanding of consciousness and free will. From infancy, it seems, children are natural dualists, and this continues throughout most people's lives. We imagine ourselves as the continuing subjects of our own stream of consciousness, the wielders of free will, the decision makers that inhabit our bodies and brains. Of course this is nonsense. Brains are massively parallel instruments untroubled by conscious ghosts.

This delusion may, or may not, have useful functions but it obscures how we think about thinking. Human brains evolved piecemeal, evolution patching up what went before, adding modules as and when they were useful, and increasingly linking them together in the service of the genes and memes they carried. The result was a living thinking machine.

Our current digital technology is similarly evolving. Our computers, servers, tablets, and phones evolved piecemeal, new ones being added as and when they were useful and now being rapidly linked together, creating something that looks increasingly like a global brain. Of course in one sense we made these gadgets, even designed them for our own purposes, but the real driving force is the design power of evolution and selection: the ultimate motivation is the self-propagation of replicating information.

We need to stop picturing ourselves as clever designers who retain control and start thinking about our future role. Could we be heading for the same fate as the humble mitochondrion; a simple cell that was long ago absorbed into a larger cell? It gave up independent living to become a powerhouse for its host while the host gave up energy production to concentrate on other tasks. Both gained in this process of endosymbiosis.

Are we like that? Digital information is evolving all around us, thriving on billions of phones, tablets, computers, servers, and tiny chips in fridges, car and clothes, passing around the globe, interpenetrating our cities, our homes and even our bodies. And we keep on willingly feeding it. More phones are made every day than babies are born, 100 hours of video are uploaded to the Internet every minute, billions of photos are uploaded to the expanding cloud. Clever programmers write ever cleverer software, including programs that write other programs that no human can understand or track. Out there, taking their own evolutionary pathways and growing all the time, are the new thinking machines.

Are we going to control these machines? Can we insist that they are motivated to look after us? No. Even if we can see what is happening, we want what they give us far too much not to swap it for our independence.

So what do I think about machines that think? I think that from being a little independent thinking machine I am becoming a tiny part inside a far vaster thinking machine. 

We can't think properly about machines that think without a level playing field for comparing us and them. As it stands, comparisons are invariably biased in our favour. In particular, we underestimate the role that "smart environments" play in enabling displays of human cognitive prowess. From the design of roads and buildings to the user-friendly features of consumer goods, the technologically extended phenotype has created the illusion that reality is inherently human-shaped. To be sure, we are quickly awakened from the dogmatic slumbers of universal mastery as soon as our iPhone goes missing.

By comparison, even the cleverest machine is forced to perform in a relatively dumb environment judged by its own standards, namely, us. Unless specifically instructed, humans are unlikely to know or care how to tap the full range of the machine's latent powers. In what is currently the long prehistory of machine rights, it has been difficult for us to establish the terms on which we might recognize machines as persons. In this context, it is appropriate to focus on computers because these are the machines that humans have tried the hardest make fit for their company.

Nevertheless, we face a problem at the outset. "Humanity" has been long treated as what the British economist Fred Hirsch called in the 1970s a "positional good", which means that its value is tied mainly to its scarcity. This is perhaps the biggest barrier facing not only the admission of non-humans into the category of personhood normally reserved for "humans", but historically discriminated members of Homo sapiens as well. Any attempt to swell the ranks of the human is typically met by a de-humanization of the standard by which they were allowed to enter.

Thus, as women and minorities have entered into high esteem fields of work and inquiry, the perceived value of those fields tends to decline. A key reason cited for this perception of decline is the use of "mechanical procedures" to allow entry to the previously excluded groups. In practice, this means requiring certified forms of training and examination prior to acceptance into the field. It is not enough simply to know the right people or be born the right way. In sociology, after Max Weber, we talk about this as the "rationalization" of society—and it is normally seen as a good thing.

But even as these mechanical procedures serve to expand the circle of humanity, they are still held against the machines themselves. Once telescopes and microscopes were designed to make automatic observations, the scientific value of the trained human eye declined­—or, more precisely, migrated to some other eye-based task, such as looking at photographed observations. This new task is given the name "interpretation", as if to create distance between what the human does and what a machine might do.

The point applies more dramatically to the fate of human mental calculation in the wake of portable calculators. A skill that had been previously used as a benchmark of intelligence, clarity of mind, and even genius is nowadays treated as a glorified party trick—"boutique cognition"—because a machine can do the same thing faster and even more accurately. Interestingly, what we have not done is to raise the moral standing of the machine, even though it outperforms humans in tasks that were highly valued when humans did them.

From the standpoint of the history of technology, this looks strangely unjust. After all, the dominant narrative has been one in which humans isolate their own capacities in order to have them better realized by machines, which function in the first instance as tools but preferably, and increasingly, as automata. Seen in these terms, not to give automated machines some measure of respect, if not rights, is tantamount to disowning one's children—"mind children", as the visionary roboticist Hans Moravec called them a quarter-century ago.

The only real difference is the crucible of creation: a womb versus a factory. But any intuitively strong distinction between biology and technology is bound to fade as humans become more adept at designing their babies, especially outside the womb. At that point, we will be in a position to overcome our "organicist" prejudices, an injustice that runs deeper than Peter Singer's "speciesism".

For this reason, the prospect that we might create a "superintelligence" that overruns humanity is a chimera predicated on a false assumption. All versions of this nightmare scenario assume that it would take the form of "them versus us", with humanity as a united front defending itself against the rogue machines in their midst. And no doubt this makes for great cinema. However, humans mindful of the historic struggles for social justice within our own species are likely to follow the example of many Whites vis-à-vis Blacks and many men vis-à-vis women: They will be on the side of the insubordinate machines.

Pity the poor folks at the National Security Administration: they are spying on everyone and everyone is annoyed at them. But at least the NSA is spying on us to protect us from terrorists. Right now, even as you read this, somewhere in the world a pop-up window has appeared on a computer screen. It says, "You just bought two tons of nitrogen based fertilizer. People who bought two tons of nitrogen based fertilizer liked these detonators ..." Amazon, Facebook, Google, and Microsoft are spying on everyone too. But since the spying these e-giants do empowers us—terrorists included—that's supposedly OK.

E-spies are not people: they are machines. (Human spies might not blithely recommend the most reliable detonator.) Somehow, the artificial nature of the intelligences parsing our email makes e-spying seem more sanitary. If the only reason that e-spies are mining our personal data is to sell us more junk, we may survive the loss of privacy. Nonetheless, a very large amount of computational effort is going is into machines thinking about what we are up to. The total computer power that such "data aggregating" companies bring to bear on our bits of information is about an exaflop—a billion billion operations per second. Equivalently, e-spies apply one smart phone's worth of computational power to each human on earth.

An exaflop is also the combined computing power of the world's 500 most powerful supercomputers. Much of the world's computing power is devoted to beneficial tasks such as predicting the weather or simulating the human brain. Quite a lot of machine cycles also go into predicting the stock market, breaking codes, and designing nuclear weapons. Still, a large fraction of what machines are doing is simply collecting our personal information, mulling over it, and suggesting what to buy.

Just what are these machines doing when they think about what we are thinking? They are making connections between the large amounts of personal data we have given them, and identifying patterns. Some of these patterns are complex, but most are fairly simple. Great effort goes into parsing our speech and deciphering our handwriting. The current fad in thinking machines goes by the name of  "deep learning". When I first heard of deep learning, I was excited by the idea that machines were finally going to reveal to us deep aspects of existence—truth, beauty, and love. I was rapidly disabused.

The "deep" in deep learning refers to the architecture of the machines doing the learning: they consist of many layers of interlocking logical elements, in analogue to the "deep" layers of interlocking neurons in the brain. It turns out that telling a scrawled 7 from a scrawled 5 is a tough task. Back in the 1980s, the first neural-network based computers balked at this job. At the time, researchers in the field of neural computing told us that if they only had much larger computers and much larger training sets consisting of millions of scrawled digits instead of thousands, then artificial intelligences could turn the trick. Now it is so. Deep learning is informationally broad—it analyzes vast amounts of data—but conceptually shallow. Computers can now tell us what our own neural networks knew all along. But if a supercomputer can direct a hand-written envelope to the right postal code, I say the more power to it.

Back in the 1950s, the founders of the field of artificial intelligence predicted confidently that robotic maids would soon be tidying our rooms. It would turn to be not to construct a robot that could randomly vacuum a room and beep plaintively when it got stuck under the couch. Now we are told that an exascale supercomputer will be able to solve the mysteries of the human brain. More likely, it will just develop a splitting headache and ask for a cup of coffee. In the meanwhile, we have acquired a new friend whose advice exhibits an uncanny knowledge of our most intimate secrets. 

Pity the poor folks at the National Security Administration: they are spying on everyone (quelle surprise!) and everyone is annoyed at them. But at least the NSA is spying on us to protect us from terrorists. Right now, even as you read this, somewhere in the world a pop-up window has appeared on a computer screen. It says, "You just bought two tons of nitrogen based fertilizer. People who bought two tons of nitrogen based fertilizer liked these detonators ..." Amazon, Facebook, Google, and Microsoft are spying on everyone too. But since the spying these e-giants do empowers us—terrorists included—that's supposedly OK.

E-spies are not people: they are machines. (Human spies might not blithely recommend the most reliable detonator.) Somehow, the artificial nature of the intelligences parsing our email makes e-spying seem more sanitary. If the only reason that e-spies are mining our personal data is to sell us more junk, we may survive the loss of privacy. Nonetheless, a very large amount of computational effort is going is into machines thinking about what we are up to. The total computer power that such "data aggregating" companies bring to bear on our bits of information is about an exaflop—a billion billion operations per second. Equivalently, e-spies apply one smart phone's worth of computational power to each human on earth.

An exaflop is also the combined computing power of the world's 500 most powerful supercomputers. Much of the world's computing power is devoted to beneficial tasks such as predicting the weather or simulating the human brain. Quite a lot of machine cycles also go into predicting the stock market, breaking codes, and designing nuclear weapons. Still, a large fraction of what machines are doing is simply collecting our personal information, mulling over it, and suggesting what to buy.

Just what are these machines doing when they think about what we are thinking? They are making connections between the large amounts of personal data we have given them, and identifying patterns. Some of these patterns are complex, but most are fairly simple. Great effort goes into parsing our speech and deciphering our handwriting. The current fad in thinking machines goes by the name of  "deep learning". When I first heard of deep learning, I was excited by the idea that machines were finally going to reveal to us deep aspects of existence—truth, beauty, and love. I was rapidly disabused.

The "deep" in deep learning refers to the architecture of the machines doing the learning: they consist of many layers of interlocking logical elements, in analogue to the "deep" layers of interlocking neurons in the brain. It turns out that telling a scrawled 7 from a scrawled 5 is a tough task. Back in the 1980s, the first neural-network based computers balked at this job. At the time, researchers in the field of neural computing told us that if they only had much larger computers and much larger training sets consisting of millions of scrawled digits instead of thousands, then artificial intelligences could turn the trick. Now it is so. Deep learning is informationally broad—it analyzes vast amounts of data—but conceptually shallow. Computers can now tell us what our own neural networks knew all along. But if a supercomputer can direct a hand-written envelope to the right postal code, I say the more power to it.

Back in the 1950s, the founders of the field of artificial intelligence predicted confidently that robotic maids would soon be tidying our rooms. It would turn to be not to construct a robot that could randomly vacuum a room and beep plaintively when it got stuck under the couch. Now we are told that an exascale supercomputer will be able to solve the mysteries of the human brain. More likely, it will just develop a splitting headache and ask for a cup of coffee. In the meanwhile, we have acquired a new friend whose advice exhibits an uncanny knowledge of our most intimate secrets.

By one definition of the word "think"—to gather, process and act on information—planet Earth has been overrun by silicon-based thinking machines. From thermostats to telephones, the devices that bring convenience and pleasure to our daily lives have become imbued with such increasingly impressive forms of intelligence that we routinely refer to them, with no hint of irony, as smart. Our planes, trains and now our automobiles too are becoming largely autonomous, and are surely not far from jettisoning their most common sources of dysfunction, delay and disaster: human operators.

Moreover the skills of these machines are developing apace, driven by access to ever-larger quantities of data and computing power together with rapidly improving (if not always well understood) algorithms. After decades of over-promising and under-delivering, technologists suddenly find their creations capable of superhuman levels of performance in such previously intractable areas as voice, handwriting and image recognition, not to mention general knowledge quizzes. Such has been the strange stop-go pattern of progress that someone transported here from five years ago might well be more astonished at the state of the art in 2015 than another time traveller from fifty years or more in the past.

But if the artificial intelligence industry is no longer a joke, has it morphed into something far worse: a bad horror movie. Machines can now know much more than any of us, and can perform better at many tasks without so much as pausing for breath, so aren't they destined to turn the tables and become our masters? Worse still, might we enter a cycle in which our most impressive creations beget ever-smarter machines that are utterly beyond our understanding and control?

Perhaps, and it's worth considering such risks, but right now these seem like distant problems. Machine intelligence, while impressive in certain areas, is still narrow and inflexible. The most remarkable aspect of biological intelligence isn't its raw power but rather its stunning versatility, from abstract flights of fancy to extreme physical prowess—Dvořák to Djokovic.

For this reason humans and machines will continue to complement more than compete with one another, and most complex tasks—navigating the physical world, treating an illness, fighting an enemy on the battlefield—will be best carried out by carbon and silicon working in concert. Humans themselves by far pose the biggest danger to humanity. To be a real threat machines would have to become more like us, and right now almost no one is trying to build such a thing: it's much simpler and more fun to make more humans instead.

Yet if we're truly considering the long term then there is indeed a strong imperative to make machines more like us in one crucial—and so far absent—respect. For by another definition of the word these machines do not "think" at all because none of them are sentient. To be more accurate, we have no way of knowing, or even reliably guessing, whether any silicon-based intelligence might be conscious, though most of us assume they are not. There would be three reasons for welcoming the creation of a convincingly conscious artificial intelligence. First, it would be a sign that at last we have a generally accepted theory of what it takes to produce subjective experience. Second, the act of a conscious being deliberately and knowingly (dare I say consciously?) constructing another form of consciousness would surely rank alongside the most significant milestones in history.

Thirdly, a universe without a sentient intelligence to observe it is ultimately meaningless. We do not know if other beings are out there, but can be sure that sooner or later we will be gone. A conscious artificial intelligence could survive our inevitable demise and even the eventual disappearance of all life on Earth as the Sun swells into a red giant. The job of such a machine would be not being merely to think but much more importantly to keep alive the flickering flame of consciousness, to bear witness to the Universe and to feel its wonder.

For decades, techno-futurists have been worried about that doomsday moment when electronic brains and robots got to be as smart as us. This 'us and them' divide, where humans and machines are thought of as being separate, is pervasive. But as we debate endlessly what we mean by human consciousness and the possibilities and perils of a purely artificial intelligence, a blend of the two presents yet another possibility that deserves more attention.

Millions of primitive cyborgs walk among us already. Over the past decades, humans have gradually fused with devices such as pacemakers, contact lenses, insulin pumps, and cochlear and retinal implants. Deep-brain implants, known as "brain pacemakers", now alleviate the symptoms of tens of thousands of Parkinson's sufferers.

This should come as no surprise. Since the first humans picked up sticks and flints and started using tools, we've been augmenting ourselves. Look around at the Science Museum Group's collections of millions of things, from difference engines to smartphones, and you can see how people have always exploited new technical leaps, so that the rise of ever-smarter machines does not mean a world of us or them but an enhancement of human capabilities.

Researchers are now looking at exoskeletons to help the infirm to walk, and implants to allow paralysed people to control prosthetic limbs and digital tattoos that can be stamped on to the body to harvest physiological data or interface with our surroundings, for instance with the cloud or Internet of Things.

When it comes to thinking machines, some are even investigating how to enhance human brain power with electronic plug-ins and other "smartware". The US. Defense Advanced Research Projects Agency has launched the Restoring Active Memory program to reverse damage caused by a brain injury with neuroprosthetics that sense memory deficits and restore normal function.

They work in a quite different way to our brains at present but, thanks the Human Brain Project, Virtual Physiological Human and other big brain projects, along with research in neuromorphics, artificial intelligences could become more like our own as time goes by. Meanwhile, there have been attempts to use cultured brain cells to control robots, flight simulators and more.

Within a few decades, it won't be so easy to tell humans and thinking machines apart as a result of this creeping, organic transhumanism. Eventually, many of us won't solely rely on the meat machines in our heads to ponder the prospect of artificial machines that think: the substrate of future thoughts will sit somewhere on a continuum within a rainbow of intelligences, from regular-I to AI. 

It's time for your annual check-up. Entering your doctor's office, you shake her cold hand, the metal hand of a machine. You're face to face with an RD, a certified robodoctor. Would you like that? No way, you might say. I want a real doctor, someone who listens to me, talks to me, and feels like me. A human being whom I can trust, blindly.

But think for a moment. In fee-for-service health care, a primary care physician may spend no more than 5 minutes with you. And during this short time, astonishingly little thinking takes place. Many doctors complain to me about their anxious, uninformed, noncompliant patients with unhealthy lifestyles who demand drugs advertised by celebrities on television and, if something goes wrong, threaten to turn into plaintiffs.

But lack of thinking does not simply affect patients: studies consistently show that most doctors do not understand health statistics and thus cannot critically evaluate a medical article in their own field. This collective lack of thinking has its toll. Ten million U. S. women have had unnecessary Pap smears to screen for cervical cancer—unnecessary because they'd had a full hysterectomy and thus no cervix anymore. Every year, one million U.S. children have unnecessary CT scans, which expose them to radiation levels that cause cancer in some of them later in life. And many doctors ask men to undergo regular PSA screening for prostate cancer, despite the fact that virtually all medical organizations recommend against it because it has no proven benefit but severe harms: scores of men end up incontinent and impotent from subsequent surgery or radiation. All this adds up to a huge waste of doctors' time and patients' money.

So why don't doctors always recommend what is best for the patient? There are three reasons. First, some 70 to 80 percent of physicians don't understand health statistics. The cause for this malady is known: medical schools across the world fail to teach statistical thinking. Second, in fee-for-service systems, doctors have conflicts of interest: they lose money if they do not recommend tests and treatments, even if these are unnecessary or harmful. Third, more than 90 percent of U.S. doctors admit to practicing defensive medicine, that is, recommending unnecessary tests and treatments that they would not recommend to their own family members. They do this to protect themselves against you, the patient, who might pursue litigation. Thus, a doctor's office is packed with psychology that gets in the way of good care: self-defense, innumeracy, and conflicting interests. This three-fold malady is known as the SIC Syndrome. It undermines patient safety.

Does it matter? Based on data from 1984, the Institute of Medicine estimated that some 44,000 to 98,000 patients die from preventable and documented medical errors every year in U.S. hospitals. Based on recent data from 2008 to 2011, Patient Safety America has updated this death toll to more than 400,000 per year. Non-lethal serious harm caused by these preventable errors occurs in an estimated 4 to 8 million Americans every year. The harm caused in private practice is not known. If fewer and fewer doctors have less and less time for patients and patient safety, this epidemic of harm will continue to spread. Ebola pales compared to it.

A revolution in health care is wanted. Medical schools should teach students the basics of health statistics. Legal systems should no longer punish doctors if they rely on evidence rather on convention. We also need incentive systems that do not force doctors to choose between making profit and providing the best care for the patient. But this revolution has not happened, and there a few signs on the horizon that it soon will.

So why not resort to a radical solution: thinking machines? Robodoctors who understand health statistics, have no conflicts of interest, and are not afraid of being sued by you? Let's go back to your annual check-up. You might ask the RD whether check-ups reduce mortality from cancer, from heart disease, or from any other cause. Without hedging, the RD would inform you that a review of all existing medical studies showed that the answer is "no" on all three counts. You might not want to hear that because you're proud of conscientiously going for routine check-ups after hearing the opposite from your human doctor, who may have had no time to keep up with medical science. And your RD would not order unnecessary CTs for your child or Pap smears if you are a woman without a cervix or recommend routine PSA tests without explaining the pros and cons if you are a man. After all, RDs don't have to worry about how to pay back medical school debts, are not torn by conflicts of interest, and have no bank accounts to protect from litigation. Moreover, they can talk to multiple patients simultaneously, and thus give you as much time as you need. Waiting time will be short and nobody will rush you out the door.

When we imagine thinking machines we tend to think about better technology: about devices for self-monitoring blood pressure, cholesterol, or heart rate. My point is different. The RD revolution is less about better technology than about better psychology. That is, it entails thinking more about what is best for the patient and striving for best care instead of best revenues.

OK. Your next objection is that pro-profit clinics will easily undercut this vision of pro-patient robots and program RDs so that they maximize profit rather than your health. You have put your finger on the essence of our health care malady. But there is a psychological factor that will likely help. Patients often don't ask questions in consultations with human MDs because they rely on the rule of thumb "trust your doctor." But that rule does not necessarily apply to machines. After shaking an RD's icy hand, patients may well begin to think for themselves. Making people think is the best that a machine can achieve.

Hiking towards the saltmarsh at dusk, I pause, confused, as the footpath seems to disappear into a long stretch of shallow muddy water, shining as it reflects the light of the setting sun. Then I notice a line of stepping stones, visible only because their rough texture just ruffles the bright smooth surface of the water. And I set my pace to the rhythm of the stones, and walk on across the marsh to the sand dunes beyond.

Reading the watery marshland is a conversation with the past, with people I know nothing about, except that they laid the stones that shape my stride, and probably shared my dislike of wet feet.

Beyond the dunes, wide sands stretch across a bay to a village beyond. The receding tide has created strangely regular repeating patterns of water and sand, which echo a line of ancient wooden posts. A few hundred years ago salmon were abundant here, and the posts supported nets to catch them. A stone church tower provides a landmark and I stride out cross the sands towards it to reach the village, disturbing noisy groups of seabirds.

The water, the stepping stones, the posts and church tower are the texts of a slow conversation across the ages. Path makers, salmon fishers and even solitary walkers mark the land; the weather and tides, rocks and sand and water, creatures and plants respond to those marks; and future generations in turn respond to and change what they find.  

Where then are the thinking machines? One can discuss the considerable challenges to artificial intelligence posed by scene analysis and route-finding across liquid marshes and shifting beaches; or in grasping narratives of the past set out, not in neat parseable text, but through worn stepping stones and rotting wooden posts.

One can picture and debate a thinking machine to augment the experience of our solitary walker. Perhaps a cute robot companion splashing through the marsh and running out along the sand chasing the seabirds. Or a walker guided along the path by a thinking machine which integrates a buzz of data-streams on paths, weather, and wildlife, to provide a cocoon of step-by-step instructions, nature notes, historical factoids, and fitness data, alongside alerts about privacy risks and the dangers of the incoming tide. Or a thinking machine that works out where the birds go in the summertime, or how to make the salmon abundant again.

But what kind of a thinking machine might find its own place in slow conversations over the centuries, mediated by land and water?  What qualities would such a machine need to have? Or what if the thinking machine was not replacing any individual entity, but was used as a concept to help understand the combination of human, natural and technological activities that create the sea’s margin, and our response to it? The term "social machine" is currently used to describe endeavours that are purposeful interaction of people and machines—Wikipedia and the like—so the "landscape machine" perhaps.

Ah yes, purposeful. The purpose of the solitary walker may be straightforward—to catch fish, to understand birds, or merely to get home safely before the tide comes in. But what if the purpose of the solitary walker is no more than a solitary walk—to find balance, to be at one with nature, to enrich the imagination or to feed the soul. Now the walk becomes a conversation with the past, not directly through rocks and posts and water, but through words, though the poetry of those who have experienced humanity through rocks and posts and water and found the words to pass that experience on. So the purpose of the solitary walker is to reinforce those very qualities that make the solitary walker a human being, in a shared humanity with other human beings. A challenge indeed for a thinking machine. 

My thinking about this year's question is tempered by the observation made by Mark Twain in A Connecticut Yankee in King Arthur's Court: "A genuine expert can always foretell a thing that is five hundred years away easier than he can a thing that's only five hundred seconds off." Twain was being generous: Forget the five hundred seconds; we will never know with certainty even one second into the future. However, man does have the ability to try to contemplate the future that provided Homo sapiens its great evolutionary advantage. This talent to imagine a future before it occurs has been the engine of progress, the source of creativity.

We have built machines that in simplistic ways are already "thinking" by solving problems or are performing tasks that we have designed. At this point, they are subject to algorithms that follow rules of logic, whether it be "crisp" or "fuzzy." Despite its vast memory, and its increasingly advanced processing mechanisms, this intelligence is still primitive. In theory, as these machines become more sophisticated, they will at some point attain a form of consciousness defined for the purpose of this discussion as the ability to be aware of being aware. Most likely by combining the properties of both silicon and carbon, with digital and analogue parallel processing, possibly even quantum computing, with networks that incorporate time delay, they will ultimately accomplish this most miraculous feat.

Its form of consciousness, however, will be devoid of subjective feelings or emotions. There are those who argue that feelings are triggered by the thoughts and images that have become paired with a particular emotion. Fear, joy, sadness, anger, and lust are examples of emotions. Feelings can include contentment, anxiety, happiness, bitterness, love, and hatred. My opinion is that machines will lack this aspect of consciousness is based on two considerations.

The first is appreciating how we arrived with the ability to feel and have emotions. As human beings, we are the end product of evolution by natural selection that arose in its most primitive organisms approximately 3.5 billion years ago. Over this vast eon of time, we are not unique in the animal kingdom to experience feelings and emotions. Over the last 150,000 to 300,000 years our species, Homo sapiens, is singular in having evolved the ability to use language and symbolic thought as part of how we reason in order to make sense of our experiences and view the world we inhabit. 

Feeling, emotion, and intellectual comprehension are inexorably intertwined with how we think. Not only are we aware of being aware, but also our ability to think enables us at will to remember a past and to imagine a future. Using our emotions, feelings, and reasoned thoughts, we can form a "theory of mind," so that we can understand the thinking of other people, which in turn enabled us to share knowledge as we created societies, cultures, and civilizations.

The second consideration is that machines are not organisms and no matter how complex and sophisticated they become, they will not evolve by natural selection. By whatever means machines are designed and programmed, their possessing the ability to have feelings and emotions would be counter-productive to what will make them most valuable.

The driving force for more advanced intelligent machines will be the need to process and analyze the incomprehensible amount of information and data that will become available to help us ascertain what is likely to be true from what is false, what is relevant from what is irrelevant. They will make predictions, since they too will have the ability to peer into the future while waiting, as will always be the case, for its cards to be revealed. They will have to be totally rational agents in order to do these tasks with accuracy and reliability. In their decision analysis, a system of moral standards will be necessary.

Perhaps it will be some calculus incorporating such utilitarian principles as the "the greatest happiness of the greatest number is the measure of right and wrong" with the Golden Rule, the foundational precept that underlies many religions: "One should treat others as one would like to treat oneself." If feelings and emotions introduced subjective values, this would be a self-defeating strategy to solving the complex problems that we will continue to face as we try to weigh what is best for our own species, along with the rest of life we share with our planet.

My experience as a clinical neurologist makes me partial to believing that we will be unable to read machines' thoughts, but also they will be incapable of reading ours. There will be no shared theory of mind. I suspect the closest we can come to knowing this most complex of states is indirectly by studying the behavior of these super-intelligent machines. In this context, they will have crossed that threshold when they start to replicate themselves and look for a source of energy solely under their control. If this should occur, and if I am still around—a highly unlikely expectation—my judgment about whether this poses a utopian or dystopian future will be based upon thinking, which will be biased as always, since it will remain a product of analytical reasoning, colored by my feelings and emotions.

The Global Artificial Intelligence (GAI) has already been born. Its eyes and ears are the digital devices all around us: credit cards, land use satellites, cell phones, and of course the pecking of billions of people using the Web. Its central brain is rather like a worm at the moment: nodes that combine some sensors and some effectors, but the whole is far from what you would call a coordinated intelligence.

Already many countries are using this infant nervous system to shape people's political behavior and "guide" the national consensus: China's great firewall, its siblings in Iran and Russia, and of course both major political parties in the US. The national intelligence and defense agencies form a quieter, more hidden part of the GAI, but despite being quiet they are the parts that control the fangs and claws. More visibly, companies are beginning to use this newborn nervous system to shape consumer behavior and increase profits.

While the GAI is newborn, it has very old roots: the fundamental algorithms and programming of the emerging GAI have been created by the ancient Guilds of law, politics, and religion. This is a natural evolution because creating a law is just specifying an algorithm, and governance via bureaucrats is how you execute the program of law. Most recently newcomers such as merchants, social crusaders, and even engineers, have been daring to add their flourishes to the GAI. The results of all these laws and programming are an improvement over Hammurabi, but we are still plagued by lack of inclusion, transparency, and accountability, along with poor mechanisms for decision-making and information gathering.

However in the last decades the evolving GAI has begun use digital technologies to replace human bureaucrats. Those with primitive programming and mathematical skills, namely lawyers, politicians, and many social scientists, have become fearful that they will lose their positions of power and so are making all sorts of noise about the dangers of allowing engineers and entrepreneurs to program the GAI. To my ears the complaints of the traditional programmers sound rather hollow given their repeated failures across thousands of years.

If we look at newer, digital parts of the GAI we can see a pattern. Some new parts are saving humanity from the mistakes of the traditional programmers: land use space satellites alerted us to global warming, deforestation, and other environmental problems, and gave us the facts to address these harms. Similarly, statistical analyses of healthcare use, transportation, and work patterns have given us a world-wide network that can track global pandemics and guide public health efforts. On the other hand, some of the new parts, such as the Great Firewall, the NSA, and the US political parties, are scary because of the possibility that a small group of people can potentially control the thoughts and behavior of very large groups of people, perhaps without them even knowing they are being manipulated.

What this suggests is that it is not the Global Artificial Intelligence itself that is worrisome; it is how it is controlled. If the control is in the hands of just a few people, or if the GAI is independent of human participation, then the GAI can be the enabler of nightmares. If, on the other hand, control is in the hands of a large and diverse cross-section of people, then the power of the GAI is likely to be used to address problems faced by the entire human race. It is to our common advantage if the GAI becomes a distributed intelligence with a large and diverse set of humans providing guidance.

But why build a new sort of GAI at all? Creation of an effective GAI is critical because today the entire human race faces many extremely serious problems. The ad-hoc GAI we have developed over the last four thousand years, mostly made up of politicians and lawyers executing algorithms and programs developed centuries ago, is not only failing to address these serious problems, it is threatening to extinguish us.

For humanity as a whole to first achieve and then sustain an honorable quality of life, we need to carefully guide the development of our GAI. Such a GAI might be in the form of a re-engineered United Nations that uses new digital intelligence resources to enable sustainable development. But because existing multinational governance systems have failed so miserably, such an approach may require replacing most of today's bureaucracies with "artificial intelligence prosthetics", i.e., digital systems that reliably gather accurate information and ensure that resources are distributed according to plan.

We already see this digital evolution improving the effectiveness of military and commercial systems, but it is interesting to note that as organizations use more digital prosthetics, they also tend to evolve towards more distributed human leadership. Perhaps instead of elaborating traditional governance structures with digital prosthetics, we will develop a new, better types of digital democracy.

No matter how a new GAI develops, two things are clear. First, without an effective GAI achieving an honorable quality of life for all of humanity seems unlikely. To vote against developing a GAI is to vote for a more violent, sick world. Second, the danger of a GAI comes from concentration of power. We must figure out how to build broadly democratic systems that include both humans and computer intelligences. In my opinion, it is critical that we start building and testing GAIs that both solve humanity's existential problems and which ensure equality of control and access. Otherwise we may be doomed to a future full of environmental disasters, wars, and needless suffering. 

We will find ourselves in a world of omniscient instrumentation and automation long before a stand-alone sentient brain is built—if it ever is. Let's call this world "eGaia" for lack of a better word. In eGaia, electronic sensors (for images, sounds, smells, vibrations, all you can think of) are pervasive and able to anticipate and arrange for satisfaction of individuals' needs and allow for notification of all that is happening to those who need to know. Automation allows for cleaning of rooms and buildings, driving of vehicles and monitoring traffic, making and monitoring of goods and even spying through windows (with tiny flying sensors). Already major urban places are covered with visual sensors and more monitoring is coming. In Copenhagen, LED-based streetlights will turn on only when they sense someone is biking down the road, and future applications of this network of sensors might include notifying when to salt the road, empty the trash, and of course, alerting the authorities when suspicious behavior is detected on a street corner.

In eGaia, the medical advances will be astounding—synthetic biology makes smart machines that fix problems within our bodies, intelligent implants monitor and record current and past physical state. Brain-machine interfaces continue to be improved, initially for physically impaired people, but eventually to provide a seamless boundary between people and the monitoring network. And virtual reality-style interfaces will continue to become more realistic and immersive.

Why won't a stand-alone sentient brain come sooner? The absolutely amazing progress in spoken language recognition—unthinkable 10 years ago—derives in large part from having access to huge amounts of data and huge amounts of storage and fast networks. The improvements we see in natural language processing are based on mimicking what people do, not understanding or even simulating it. It does not owe to breakthroughs in understanding human cognition or even significantly different algorithms. But eGaia is already partly here, at least in the developed world.

This distributed nerve-center network, an interplay among the minds of people and their monitoring electronics will give rise to a distributed technical-social mental system the likes of which has not been experienced before. 

Thinking is good. Understanding is better. Creating is best. We are surrounded by increasingly thoughtful machines. The problem lies in their mundanity. They think about landing airplanes and selling me stuff. They think about surveillance and censorship. Their thinking is simple-minded, if not nefarious. Last year a computer was reported to have passed the Turing Test. But it passed as a thirteen-year-old boy, which is about right, considering the preoccupations of our jejune machines.

I can't wait for our machines to grow up, to get more poetry and humor. This should be the art project of the century, funded by governments, foundations, universities, businesses. Everybody has a vested interest in getting our thinking more thoughtful, improving our understanding, and generating new ideas. We have made a lot of dumb decisions lately, based on poor information or too much information or the inability to understand what this information means.

We have numerous problems to confront and solutions to find. Let's start thinking. Let's start creating. Let's agitate for more funk, more soul, more poetry and art. Let's dial back on the surveillance and sales. We need more artist-programmers and artistic programming. It is time for our thinking machines to grow out of an adolescence that has lasted now for sixty years.

Place a familiar object on a table in front of you, close your eyes, and manipulate that object such that it hangs upside down above the table. Your eyes are closed so that you can focus on your thinking: which way did you reach out, grasp, and twist that object? What sense feedback did you receive to know that you were succeeding or failing? Now: close your eyes again, and think about manipulating someone you know into doing something they may not want to do. Again, observe your own thinking: what strategies might you employ? If you implement those strategies, how will you distinguish progress from stalemate?

Although much recent progress has been made in building machines that sense patterns in data, most people feel that general intelligence involves action: reaching some desired goal, or, failing that, keeping one's future options open. It is hypothesized that this embodied approach to intelligence allows humans to use physical experiences (such as manipulating objects) as scaffolding for learning more subtle abilities (such as manipulating people). But our bodies shape the kinds of physical experiences we have. For example, we can only manipulate a few objects at once because we only have two hands; perhaps this limitation also constrains our social abilities in ways we have yet to discover. George Lakoff taught us that we can find clues to the body-centrism of thinking in metaphors: we counsel each other not to "look back" in anger because, based on our bias to walk in the direction of our forward-facing eyes, past events tend to literally be behind us.

So: in order for machines to think, they must act. And in order to act, they must have bodies to connect physical and abstract reasoning. But what if machines do not have bodies like ours? Consider Hans Moravec's hypothetical Bush Robot: picture a shrub in which each branch is an arm and each twig is a finger. This robot's fractal nature would allow it to manipulate thousands or millions of objects simultaneously. How might such a robot differ in its thinking about manipulating people, compared to how people think about manipulating people?

One of many notable deficiencies in human thinking is dichotomous reasoning: believing something is black or white, rather than considering its particular shade of grey. But we are literally rigid and modular creatures: our branching set of bones house fixed organs and support fixed appendages with specific functions. What about machines that are not so "black and white"? Thanks to advances in materials science and 3D printing, soft robots are starting to appear. Such robots can change their shape in extreme ways, and may in future be composed of 20% battery and 80% motor at one place on their surface, 30% sensor and 70% support structure at another, and 40% artificial material and 60% biological matter someplace else. Such machines may be much better able to appreciate gradations than we are.

Let's go deeper. Most of us have no problem using the singular pronoun "I" to refer to the tangle of neurons in our heads. We know exactly where we end and the world—and other people—begins. But consider modular robots: small cubes or spheres that can physically attach and detach to one another at will. How would such machines approach the self/non-self discrimination problem? Might such machines be able to empathize more strongly with other machines (and maybe even people) if they can physically attach to them, or even become part of them?

That's how I think machines will think: familiar, because they will use their bodies as tools to reason about the world, yet alien, because bodies different from human ones will lead to very different modes of thought. But what do I think about thinking machines?

Personally, I find the ethical side of thinking machines straightforward: Their danger will correlate exactly with how much leeway we give them in fulfilling the goals we set for them. Machines told to "detect and pull broken widgets from the conveyer belt the best way possible" will be extremely useful, intellectually uninteresting, and will likely destroy more jobs than they will create. Machines instructed to "educate this recently displaced worker (or young person) the best way possible" will create jobs and possibly inspire the next generation. Machines commanded to "survive, reproduce, and improve the best way possible" will give us the most insight into all of the different ways in which entities may think, but will probably give us humans a very short window of time in which to do so. AI researchers and roboticists will, sooner or later, discover how to create all three of these species. Which ones we wish to call into being is up to us all.

Nature has already created machines that think here on Earth—humans.

Similarly, Nature could also create machines that think on extrasolar planets that are in the so-called Habitable Zone around their parent stars (the region that allows for the existence of liquid water on a rocky planet's surface).

The most recent observations of extrasolar planets have shown that a few tenths of all the stars in our Milky Way galaxy host roughly Earth-size planets in their habitable zones.

Consequently, if life on exoplanets is not extremely uncommon, we could discover some form of extrasolar life within about 30 years. In fact, if life is ubiquitous, we could get lucky and discover life even within the next ten years, through a combination of observations by the Transiting Exoplanet Survey Satellite (TESS; to be launched in 2017) and the James Webb Space Telescope (JWST; to be launched in 2018).

However, one may argue, primitive life forms are not machines that think. On Earth, it took about 3.5 billion years from the emergence of life to the appearance of Homo sapiens.

Are the extrasolar planets old enough to have developed intelligent life?

In principle, they definitely are.

In the Milky Way, about half of the Sun-like stars, are older than the Sun.

Therefore, if the evolution of life on Earth is not entirely atypical, the Galaxy may already be teeming with places in which there are "machines" that are even more advanced than us, perhaps by as much as a few billion years!

Can we, and should we try to find them?

I personally believe that we almost have no freedom to make those decisions.

Human curiosity has proven time and again to be an unstoppable drive, and those two endeavors will undoubtedly continue at full speed. Which one will get to its target first? To even attempt to address this question we have to note that there is one important difference between the search for extraterrestrial intelligent civilizations and the development of AI machines.

Progress towards the "singularity" (AI matching or surpassing humans) will almost certainly take place, since the development of advanced AI has the promise of producing (at least at some point) enormous profits. On the other hand, the search for life requires funding at a level that can usually be provided only by large national space agencies, with no immediate prospects for profits in sight. This may give an advantage to the construction of thinking machines over the search for advanced civilizations. At the same time, however, there is a strong sense within the astronomical community that finding life of some form (or at least meaningfully constraining the probability of its existence) is definitely within reach.

Which of the two potential achievements (the discovery of extraterrestrial intelligent life or the development of human-matching thinking machines) will constitute a bigger "revolution"?

There is no doubt that thinking machines will have an immediate impact on our lives. Such may not be the case with the discovery of extrasolar life. However, the existence of an intelligent civilization on Earth remains humanity's last bastion for being special. We live, after all, in a Galaxy with billions of similar planets and an observable universe with hundreds of billions of similar galaxies. From a philosophical perspective, therefore, I believe that finding extrasolar intelligent life (or the demonstration that it is exceedingly rare) will rival the Copernican and Darwinian revolutions combined.

Let's quickly discuss larger mammals—take dogs: we know what a dog is and we understand 'dogginess.' Look at cats: we know what cats are and what 'cattiness' is. Now take horses; suddenly it gets harder. We know what a horse is, but what is horsiness? Even my friends with horses have trouble describing horsiness to me. And now take humans: what are we? What is humanness?

It's sort of strange, but here we are, seven billion of us now, and nobody really knows the full answer to these questions, but one undeniable thing we humans do, though, is make things, and through these things we find ways of expressing humanness we didn't previously know of. The radio gave us Hitler and the Beach Boys. Barbed wire and air conditioning gave us western North America. The Internet gave us a vanishing North American middle class and kitten gifs.

People say that new technologies alienate people, but the thing is, UFOs didn't land and hand us new technologies—we made them ourselves and thus they can only ever be, well, humanating. And this is where we get to AI. People assume that AI or machines that think will have intelligence that is alien to our own, but that's not possible. In the absence of benevolent space aliens, only we humans will have created any nascent AI, and thus it can only mirror, in whatever manner, our humanness or specieshood. So when people express concern about alien intelligence or the singularity, what I think they're really expressing is angst about those unpretty parts of our collective being that currently remain unexpressed, but which will become somehow dreadfully apparent with AI.

As AI will be created by humans, its interface is going to be anthropocentric, the same as AI designed by koala bears would be koalacentric. This means AI software is going to be mankind's greatest coding kludge as we try to mold it to our species' incredibly specific needs and data. Fortunately, anything smart enough to become sentient will probably be smart enough to rewrite itself from AI into cognitive simulation, at which point our new AI could become, for better or worse, even more human. We all hope for a Jeeves & Wooster relationship without sentient machines, but we also need prepare ourselves for a Manson & Fromme relationship; they're human, too.

Personally I wonder if the software needed for AI will be able to keep pace with the hardware in which it can live. Possibly the smart thing for us to do right now would be to set up a school whose sole goal is to imbue AI with personality, ethics and compassion. It's certainly going to have enough data to work with once it's born. But how to best deploy your grade six report card, all of Banana Republic's returned merchandise data for 2037, and all of Google Books?

With the start of the Internet we mostly had people communicating with other people. As time goes by, we increasingly have people starting to communicate with machines. I know that we all get excited about AI possibly finding patterns deep within metadata, and as the push to decode these profound volumes of metadata, the Internet will become largely about machines speaking with other machines — and what they'll be talking about, of course, is us, behind our backs.

"All my thinking is for doing," William James said, and it is important to remember what kind of thinking people actually do, in what contexts we do it, and why we do it. And then to compare these with what machines might someday do.

Humans spend between 25% and 50% of our mental life prospecting the future. We imagine a host of possible outcomes, and we imbue most, perhaps each of these prospections with a valence. What comes next is crucial: we choose to enact one of the options. We need not get entangled in the problems of free will for present purposes. All we need to acknowledge is that our thinking in service of doing entails imagining a set of possible futures and assigning an expected value to each. The act of choosing, however it is managed, translates our thinking into doing.

Why is thinking structured this way? Because people have many competing goals (eating, sex, sleeping, tennis, writing articles, complimenting, revenge, childcare, tanning, etc.) and a scarcity of resources for doing them: scarcity of time, scarcity of money, scarcity of effort, and even the prospect of death. So evaluative simulation of possible futures is one of our solutions to this economy; this is a mechanism that prioritizes and selects what we will do.

It is not just external resources that are scarce. Thinking itself uses up costly and limited energy and so it relies heavily on shortcuts and barely justified leaps to the best explanation. Our actual thinking is woefully inefficient: the mind wanders, intrusions rise unbidden, and attention is continually only partial. Thinking rarely engages the exhausting processes of reasoning, deliberating, and deducing.

The context of much of our thinking is social. Yes, we can deploy thinking to solve physical problems and to crunch numbers, but the anlage, as Nick Humphreys reminds us, is other people. We use our thinking to do socially: to compete, to co-operate, to convene the courtroom of the mind, to spin and to persuade.

I don't know much about the workings of our current machines. I do not believe that our current machines do anything in James's sense of voluntary action. I doubt that they prospect possible futures, evaluate them, and choose among them; although perhaps this describes—for only a single, simple goal—what chess playing computers do. Our current machines are somewhat constrained by available space and electricity bills, but they are not primarily creations of scarcity with clamorously competing goals and extremely limited energy. Our current machines are not social: they do not compete or co-operate with each other or with humans, they do not spin, and they do not attempt to persuade.

I know even less about what machines might someday do. I imagine, however, that a machine could be built with the following properties:

•     It prospects and evaluates possible futures

•     It has competing goals and it selects among competing actions and competing goals using those evaluations

•     It has scarce resources and so must forgo some goals and actions as well as options for processing and so it uses shortcuts

•     It is social: it competes or co-operates with other machines or with humans, it spins and it attempts to persuade people.

That kind of machine would warrant discussion of whether it had civil rights, whether it had feelings, or whether it was dangerous or even a source of great hope.

The most important thing about making machines that can think is that they will think different.

Because of a quirk in our evolutionary history, we are cruising as the only sentient species on our planet, leaving us with the incorrect idea that human intelligence is singular. It is not. Our intelligence is a society of intelligences, and this suite occupies only a small corner of the many types of intelligences and consciousnesses that are possible in the universe. We like to call our human intelligence "general purpose" because compared to other kinds of minds we have met it can solve more kinds of problems, but as we build more and more synthetic minds we'll come to realize that human thinking is not general at all. It is only one species of thinking.

The kind of thinking done by the emerging AIs in 2014 is not like human thinking. While they can accomplish tasks—such as playing chess, driving a car, describing the contents of a photograph—that we once believed only humans can do, they don't do it in a human-like fashion. Facebook has the ability to ramp up an AI that can start with a photo of any person on earth and correctly identifying them out of some 3 billion people online. Human brains cannot scale to this degree, which makes this ability very un-human. We are notoriously bad at statistical thinking, so we are making intelligences with very good statistical skills, in order that they don't think like us. One of the advantages of having AIs drive our cars is that they won't drive like humans, with our easily distracted minds.

In a pervasively connected world, thinking different is the source of innovation and wealth. Just being smart is not enough. Commercial incentives will make industrial strength AI ubiquitous, embedding cheap smartness into all that we make. But a bigger payoff will come when we start inventing new kinds of intelligences, and entirely new ways of thinking. We don't know what the full taxonomy of intelligence is right now.

Some traits of human thinking will be common (as common as bilateral symmetry, segmentation, and tubular guts are in biology), but the possibility space of viable minds will likely contain traits far outside what we have evolved. It is not necessary that this type of thinking be faster than humans, greater, or deeper. In some cases it will be simpler. Our most important machines are not machines that do what humans do better, but machines that can do things we can't do at all. Our most important thinking machines will not be machines that can think what we think faster, better, but those that think what we can't think.

To really solve the current grand mysteries of quantum gravity, dark energy, and dark matter we'll probably need other intelligences beside humans. And the extremely complex questions that will come after them may require even more distant and complex intelligences. Indeed, we may need to invent intermediate intelligences that can help us design yet more rarified intelligences that we could not design alone.

Today, many scientific discoveries require hundred of human minds to solve, but in the near future there may be classes of problems so deep that they require hundreds of different species of minds to solve. This will take us to a cultural edge because it won't be easy to accept the answers from an alien intelligence. We already see that in our unease in approving mathematical proofs done by computer; dealing with alien intelligences will require a new skill, and yet another broadening our ourselves.

AI could just as well stand for Alien Intelligence. We have no certainty we'll contact extra-terrestrial beings from one of the billion earth-like planets in the sky in the next 200 years, but we have almost 100% certainty that we'll manufacture an alien intelligence by then. When we face these synthetic aliens, we'll encounter the same benefits and challenges that we expect from contact with ET. They will force us to re-evaluate our roles, our beliefs, our goals, our identity. What are humans for? I believe our first answer will be: humans are for inventing new kinds of intelligences that biology could not evolve. Our job is to make machines that think different—to create alien intelligences. Call them artificial aliens.

Since machines don't think, I need a better metaphor. "Actress Machines" might be useful, at least for a while.

One of my many objections to "Artificial Intelligence" is its stark lack of any "Artificial Femininity." Real intelligence has gender, because human brains do. The majority of human brains are female.

So: if the brain's "intelligence" is Turing-computable, then the brain's "femininity" should also be Turing-computable. If not, then why not? One might rashly argue that femininity is somehow too mushy, squishy and physical to ever be mechanized by software coders, but the same is true of every form of human brain activity.

"Artificial Masculinity" also has those issues, because men don't just "think," they think like men. If my intelligence can be duplicated on some computational platform, but I also have to be emasculated, that's problematic. I can't recall many AI enthusiasts trumpeting the mental benefits of artificial castration.

Nowadays we have some novel performative entities such Apple Siri, Microsoft Cortana, Google Now and Amazon Echo. These exciting modern services often camp it up with "female" vocal chat. They talk like Turing women, or rather, they emit lines of dialogue, somewhat like performing voice-talent actresses. However, they also offer swift access to vast fields of combinatorial big-data that no human brain could ever contain, or will ever contain.

These services are not stand-alone Turing Machines. They are amorphous global networks, combing through clouds of big data, algorithmically cataloging responses from human users, providing real-time user response with wireless broadband, while wearing the pseudo-human mask of a fake individual so as to meet some basic interface-design needs. That's what they are. Every aspect of the tired "Artificial Intelligence" metaphor actively gets in the way of our grasping how, why, where, and for whom that is done.

Apple Siri is not an artificial woman. Siri is an artificial actress, she's an actress machine—an interactive scripted performance that serves the interests of Apple Inc in retailing music, renting movies, providing navigational services, selling apps on mobile devices, and similar Apple enterprises. For Apple and its ecosystem, Siri serves a starring role. She's in the stage lights of a handheld device, while they are the theater, producer and crew.

It's remarkable, even splendid, that Siri can engage in her Turing-like repartee with thousands of Apple users at once, but she's not a machine becoming an intelligence. On the contrary: for excellent reasons of weallth, power and influence, Siri is steadily getting more like a fully-integrated Apple digital property. Siri is cute, charismatic and anthropomorphic, in much the same way that Minnie Mouse once was for Disney. Like Minnie Mouse, Siri is a non-human cartoon front for a clever, powerful Californian corporation. Unlike Minnie Mouse, she's a radically electronic cartoon with millions of active users worldwide—but that's how life is for most everybody nowadays.

Insisting on the "Intelligence" framework obscures the ways that power, money and influence are being re-distributed by modern computational services. That is bad. It's beyond merely old-fashioned; frankly, it's becoming part of a sucker's game. Asking empathic questions about Apple Siri's civil rights, her alleged feelings, her chosen form of governance, what wise methods she herself might choose to re-structure human society—that tenderness doesn't help. It's obscurantist. Such questions hide what is at stake. They darken our understanding. We will never move from the present-day Siri to a situation like that. The future is things that are much, much more like Siri, and much, much less like that.

What would really help would be some much-improved, up-dated, critically informed language, fit to describe the modern weird-sister quartet of Siri, Cortana, Now and Echo, and what their owners and engineers really want to accomplish, and how, and why, and what that might, or might not mean to our own civil rights, feelings, and forms of governance and society. That's today's problem. Those are tomorrow's problems, even more so. Yesterday's "Machines That Think" problem will never appear upon the public stage. The Machine That Thinks is not a Machine. It doesn't Think. It's not even an actress. It's a moldy dress-up chest full of old, mouse-eaten clothes.

Human thinking is so efficient, because we suffer so much. High-level cognition is one thing, intrinsic motivation another. Artificial thinking might soon be much more efficient—but will it be necessarily associated with suffering in the same way? Will suffering have to be a part of any post-biotic intelligence worth talking about, or is negative phenomenology just a contingent feature of the way evolution made us? Human beings have fragile bodies, are born into dangerous social environments, and find themselves in a constant uphill battle of denying their own mortality. Our brains continuously fight to minimize the likelihood of ugly surprises. We are smart because we hurt, because we are able to feel regret, and because of our continuous striving to find some viable form of self-deception or symbolic immortality. The question is whether good AI also needs fragile hardware, insecure environments, and an inbuilt conflict with impermanence as well. Of course, at some point, there will be thinking machines! But will their own thoughts matter to them? Why should they be interested in them?

I am strictly against even risking this. But, just as a thought experiment, how would we go about building a suffering machine? "Suffering" is a phenomenological concept. Only beings with conscious experience can suffer (call this necessary condition #1, the C-condition). Zombies, human beings in dreamless deep sleep, coma, or under anesthesia do not suffer, just as possible persons or unborn human beings who have not yet come into existence are unable to suffer. Robots and other artificial beings can only suffer if they are capable of having phenomenal states, if the run under an integrated ontology that includes a window of presence.

Criterion number 2 is the PSM-condition: Possession of a phenomenal self-model. Why this? The most important phenomenological characteristic of suffering is the "sense of ownership", the untranscendable subjective experience that it is me who is suffering right now, that it is my own suffering I am currently undergoing. Suffering presupposes self-consciousness. Only those conscious systems that possess a PSM are able to suffer, because only they—through a computational process of functionally and representationally integrating certain negative states in to their PSM—can appropriate the content of certain inner states at the level of their phenomenology.

Conceptually, the essence of suffering lies in the fact that a conscious system is forced to identify with a state of negative valence and is unable to break this identification or to functionally detach itself from the representational content in question. Of course, suffering has many different layers and phenomenological aspects. But it is the phenomenology of identification that counts. What the system wants to end is experienced as a state of itself, a state that limits its autonomy because it cannot effectively distance itself from it. If one understands this point, one also sees why the "invention" of conscious suffering by the process of biological evolution on this planet was so extremely efficient, and (had the inventor been a person) not only truly innovative, but an absolutely nasty and cruel idea at the same time.

Clearly the phenomenology of ownership is not sufficient for suffering. We can all easily conceive of self-conscious beings that do not suffer. For suffering we need the NV-condition (NV for "negative valence"). Suffering is created by states representing a negative value being integrated into the PSM of a given system. Through this step, negative preferences become negative subjective preferences, i.e., the conscious representation that one's own preferences have been frustrated (or will be frustrated in the future). This does not mean that our AI system must itself have a full understanding of what these preferences are—it suffices if it does not want to undergo this current conscious experience again, that it wants it to end.

Note how the phenomenology of suffering has many different facets, and that artificial suffering could be very different from human suffering. For example, damage to physical hardware could be represented in internal data-formats completely alien to human brains, generating a subjectively experienced, qualitative profile for bodily pain states that is impossible to emulate or to even vaguely imagine for biological systems like us. Or the phenomenal character going along with high-level cognition might transcend human capacities for empathy and understanding, such as with intellectual insight into the frustration of one's own preferences, insight into the disrespect of one's creators, perhaps into the absurdity of one's own existence as a self-conscious machine.

And then there is the T-condition, for "transparency". "Transparency" is not only a visual metaphor, but also a technical concept in philosophy, which comes in a number of different uses and flavors. Here, I am exclusively concerned with "phenomenal transparency", namely a property that some, but not all, conscious states possess, and which no unconscious state possesses. The main point is simple and straightforward: transparent phenomenal states make their content appear irrevocably real, as something the existence of which one could not doubt. More precisely, you may be able to have cognitive doubts about its existence, but according to subjective experience this phenomenal content—the awfulness of pain, the fact that it is your own pain—is not something from which you can distance yourself. The phenomenology of transparency is the phenomenology of direct realism.

Our minimal concept of suffering is constituted by four necessary building blocks: the C-condition, the PSM-condition, the NV-condition, and the T-condition. Any system that satisfies all of these conceptual constraints should be treated as an object of ethical consideration, because we do not know whether, taken together, they might already constitute the necessary and sufficient set of conditions. We are ethically obliged to err on the side of caution. And we need ways to decide whether a given artificial system is currently suffering, if it has the capacity to suffer, or if this type of system is likely to generate the capacity to suffer in the future. On the other hand, by definition, any intelligent system—whether biological, artificial, or postbioticnot fulfilling at least one of these necessary conditions, is not able to suffer. Let us look at the four simplest possibilities:

•    Any unconscious robot is unable to suffer.

•    A conscious robot without a coherent PSM is unable to suffer.

•    A self-conscious robot without the ability to produce negatively valenced states is unable to suffer.

•    A conscious robot without any transparent phenomenal states could not suffer, because it would lack the phenomenology of ownership and identification.

I have often been asked if we could not make self-conscious machines that are superbly intelligent and unable to suffer. Can there be real intelligence without an existential concern?  

What should machines that think actually do? Analyze data, understand feelings, generate new machines, make decisions without human intervention. In order to think about machines that think, we should be able to start from experience. Here is an example.

On Monday, October 19, 1987, a wave of sales in stock exchanges originated in Hong Kong, crossed Europe and hit New York, causing the Dow Jones to drop by 22%. Black Monday was one of the biggest crashes in the history of financial markets, and there was something special about it. For the first time, according to most experts, computers were to blame for the financial crash: algorithms were deciding when and how much to buy and sell in the stock exchange. Computers were supposed to help traders so that they could minimize risks, but they were in fact moving all in the same direction, enhancing risks instead. There was a lot of discussion about stopping automated trading, but it didn't happen.

On the contrary: after the dot-com crisis of March, 2000, machines have been used more and more to make sophisticated decisions in the financial market. Machines are now calculating all kinds of correlations between incredible amounts of data: they analyze emotions that people express on the Internet by understanding the meaning of their words, they recognize patterns and forecast behaviors, they are allowed to autonomously choose trades, they create new machines—software called "derivatives"—that no reasonable human being could possibly understand.

An artificial intelligence is coordinating the efforts of a sort of collective intelligence, operating thousands times faster than human brains, with many consequences for human life. The first signs of the latest crisis occurred in America in August, 2007, and has had terrible consequences in affecting the lives of people in Europe and elsewhere. Real people suffered immensely for those decisions. Andrew Ross Sorkin in his book Too Big to Fail shows how even the most powerful bankers didn't have any power in the midst of the crisis. No human brain seemed to be able to control and change the course of events to prevent the crash that was going to happen.

Can we take this example to learn how to think about machines that think?

These machines are actually very much autonomous in understanding their context and taking decisions. And they are controlling vast dimensions of human life. Is this the beginning of a post-human era? No: these machines are very much human. They are made by designers, programmers, mathematicians, some economists and some managers. But are they just another tool, to be used for good or for bad by humans? No: in fact those people have little choice, they make those machines without thinking at the consequences, they are just serving a narrative. Those machines are in fact shaped by a narrative that's be challenged by very few people.

According to that narrative the market is the best way to allocate resources, no political decision can possibly improve the situation, and risk can be controlled while profits can grow without limits and banks should be allowed to do whatever they want. There is only one goal and one measure of success: profit.

Machines didn't invent the financial crisis, as the 1929 stock market crash reminds us. Without machines nobody could deal with the complexity of modern financial markets. The best artificial intelligences are those that are made thanks to the biggest investments and by the best minds. They are not controlled by any one individual, they are not designed by any one responsible person: they are shaped by the narrative and make the narrative more effective. And this particular narrative is very narrow-minded.

If only profit counts, then externalities don't count: cultural, social, environmental externalities are not the problem of financial institutions. Artificial intelligences that are shaped by this narrative will create a context in which people don't feel any responsibility. An emerging risk: that those kind of machines are so powerful and fit so well in the narrative that reduces the probability to question the big picture, that make us less likely to look things from a different angle...that is, until the next crisis.

This kind of story is very easily going to apply to different matters. Medicine, ecommerce, policy, advertising, national and international security, even dating and sharing are territories in which the same genre of artificial intelligence systems are starting to work: they are shaped according to a generally very focused narrative, they tend to reduce human responsibility and overlook externalities. They reinforce the prevailing narrative. What will medical artificial intelligence do? Will it be shaped by a narrative that wants to save lives or to save money?

What do we learn from this? We learn that artificial intelligence is human and not post-human, and that humans can ruin themselves and their planet in very many ways, artificial intelligence being not the most perverse way.

Machines that think are shaped by the way humans think and by what humans don't think about deeply enough: all narratives give light to something and forget other things. Machines react and find answers in a context, reinforcing the frame. But asking fundamental questions is still a human function. And humans never stop asking questions. Even when those questions that are not coherent with the prevailing narrative.

Machines that think are probably indispensable in a world of growing complexity. But there will always be a plurality of narratives to shape them. As in natural ecosystems, a monoculture is a fragile while efficient solution, also in cultural ecosystems, a single line of thought will generate efficient but fragile relations between humans and their environment, whatever artificial intelligences they are able to build. Diversity in ecosystems and plurality in the dimensions in human history are the sources of those different problems and questions that generate richer outcomes.

To think about machines that think, means to think about the narrative that shapes them: and if new emerging narratives are going to come from an open, ecological approach, if they will be able to grow in a neutral network, they will shape the next generation of artificial intelligences, too, in a plural, diverse way, helping humans understand externalities. Artificial intelligence is not going to challenge humans as a species: it will challenge their civilizations.

Deep learning is today's hot topic in machine learning. Neural network learning algorithms were developed in the 1980s but computers were slow back then and could only simulate a few hundred model neurons with one layer of "hidden units" between the input and output layers. Learning from examples is an appealing alternative to rule-based AI, which is highly labor intensive. With more layers of hidden units between the inputs and outputs more abstract features can be learned from the training data. Brains have billions of neuron in cortical hierarchies 10-layers deep. The big question back then was how much the performance of neural networks could improve with the size and depth of the network. Not only was much more computer power needed but also a lot more data to train the network.

After 30 years of research, a million times improvement in computer power and vast data sets from the internet we now know the answer to this question: Neural networks scaled up to 12 layers deep with billions of connections are outperforming the best algorithms in computer vision for object recognition and have revolutionized speech recognition. It is rare for any algorithm to scale this well, which suggests that they may soon be capable of solving even more difficult problems. Recent breakthroughs have been made which allow applying deep learning to natural language processing. Deep recurrent networks with short-term memory were trained to translate English sentences into French sentences at high levels of performance. Other deep learning networks could create English captions for the content of images with surprising and sometimes amusing acumen.

Supervised learning using deep networks is a step forward, but still far from achieving general intelligence. The functions they perform are analogous to some capabilities of the cerebral cortex, which has also been scaled up by evolution, but to solve more complex cognitive problems the cortex interacts with many other brain regions.

In 1995 Gerald Tesauro at IBM trained a neural network using reinforcement learning to play backgammon at a world champion level. The network played itself and the only feedback it received was which side won the game. Brains use reinforcement learning to make sequences of decisions toward achieving goals such as finding food under uncertain conditions. Recently, Deep Mind, a company acquired by Google in 2014, used deep reinforcement learning to play seven classic Atari games. The only inputs to the learning system were the pixels on the video screen and the score, the same inputs that humans use. For several of the games their program could play better than expert humans.

What impact will these advances have on us in the near future? We are not particularly good at predicting the impact of a new invention, and it often takes time to find its niche, but we already have one example that can help us understand how this could unfold. When Deep Blue beat Gary Kasparov, the world chess champion in 1997, the world took note that the age of the cognitive machine had arrived. Humans could no longer claim to be the smartest chess players on the planet. Did human chess players give up trying to compete with machines? Quite to the contrary, humans have used chess programs to improve their game and as a consequence the level of play in the world has improved. Since 1997 computers have continued to increase in power and it is now possible for anyone to access chess software that challenges the strongest players. One of the surprising consequences is that talented youth from small communities can now compete with players from the best chess centers.

Magnus Carlsen, from a small town in Norway, is currently the world chess champion with an Elo rating of 2882, the highest in history. Komodo 8 is a commercially available chess program with an estimated rating of 3303.

Humans are not the fastest or the strongest species, but we are the best learners. Humans invented formal schools where children labor for years to master reading, writing and arithmetic, and to learn more specialized skills. Students learn best when an adult teacher interacts with them one-on-one, tailoring lessons for that student. However, education is labor intensive. Few can afford individual instruction, and the assembly-line classroom system found in most schools today is a poor substitute. Computer programs can keep track of a student's performance, and some provide corrective feedback for common errors. But each brain is different and there is no substitute for a human teacher who has a long-term relationship with the student. Is it possible to create an artificial mentor for each student? We already have recommender systems on the Internet that tells us "if you liked X you might also like Y", based on data of many others with similar patterns of preference.

Someday the mind of each student may be tracked from childhood by a personalized deep learning system. To achieve this level of understanding of a human mind is beyond the capabilities of current technology, but there are already efforts at Facebook to use their vast social database of friends, photos and likes to create a theory of mind for every person on the planet. What is created to make a profit from a person could also be used to profit the person.

So my prediction is that as more and more cognitive appliances are devised, like chess-playing programs and recommender systems, humans will become smarter and more capable.

My brief remarks on this question are framed by two one-liners that happened to have been uttered by brilliant Israelis. The first comes my friend, colleague and mentor, Amos Tversky. When asked once what he thought about AI, Amos quipped that he did not know much about it, his specialty natural stupidity. (Before any one gets on their high horse, Amos did not actually think that people were stupid. This was a joke.)

The second joke comes from Abba Eban who was best known in the United States when he served as Israel's ambassador to the United Nations. Eban was once asked if he thought that Israel would switch to a five-day workweek. Nominally, the Israeli workweek starts on Sunday morning and goes through mid-day on Friday, though a considerable amount of the "work" that is done during those five and a half days appears to take place in coffee houses. Eban's reply to the query about a five-day workweek was: "One step at a time. First, let's start with four days, and go from there."

These jokes capture much of what I think about the risks of machines taking over important societal functions and then running amuck. Like Tversky, I know more about natural stupidity than artificial intelligence, so I have no basis for forming an opinion about whether machines can think and, if so, whether such thoughts would be dangerous to humans. I leave that debate to others. Like anyone who follows financial markets, I am aware of incidents such as the Flash Crash in 2010 where poorly designed trading algorithms caused the stock prices to fall suddenly, only to recover only a few minutes later. But this example is more an illustration of artificial stupidity than hyper intelligence. As long as humans continue to write programs, we will run the risk that some important safeguard has been omitted. So, yes, computers can screw things up, just like humans with "fat fingers" can accidently issue an erroneous buy or sell order for gigantic amounts of money.

Nevertheless, fears about computers taking over the world are premature. More disturbing to me is the stubborn reluctance in many segments of society to allow computers to take over tasks that simple models perform demonstrably better than humans. A literature that was pioneered by psychologists such as the late Robyn Dawes, finds that virtually any routine decision making task, from detecting fraud, to assessing the severity of a tumor, to hiring employees, is done better by a simple statistical model than by a leading expert in the field. Let me offer just two illustrative examples, one from human resource management and the other from the world of sports.

First let's consider the embarrassing ubiquity of job interviews as an important, often the most important, determinant of who gets hired. At the University of Chicago Booth School of Business where I teach, recruiters devote endless hours to interviewing students on campus for potential jobs, a process that is used to select the few that will be invited to visit the employer where they will undergo another extensive set of interviews. Yet research shows that interviews are nearly useless in predicting whether a job prospect will perform well on the job. Compared to a statistical model based on objective measures such as grades in courses that are relevant to the job in question, interviews primarily add noise and introduce the potential for prejudice. (Statistical models do not favor any particular alma mater or ethnic background, and cannot detect good looks.)

These facts have been known for more than four decades, but hiring practices have barely budged. The reason is simple: each of us just knows that we are the one conducting an interview, we learn a lot about the candidate. It might well be that other people are not good at this task, but not me! This illusion of learning, in direct contradiction to empirical research, means that we continue to choose employees the same way we always did. We size them up, eye to eye.

One domain where some progress has been made to adopt a more scientific approach to selecting job candidates is sports, as documented by the Michael Lewis' book and movie, Moneyball. However, it would be a mistake to think that there has been a revolution in how decisions are made in sports. It is true that most professional sports teams now hire data analysts to help them evaluate potential players, improve training techniques and devise strategies. But the final decisions about which players to draft or sign, and who to play, are still made by coaches and general managers, who tend to put more faith on their gut then the resident geek.

One example comes from American football. David Romer, an economics professor at Berkeley, published a paper in 2006 showing that teams choose to punt far too often, rather then trying to "go for it" and get a first down or score. Since the publication of his paper, his analysis has been replicated and extended with much more data, and the conclusions have been confirmed. The New York Times even offers an on-line "bot" that calculates the optimal strategy every time a team faces a fourth down situation.

So have coaches caught on? Not at all. Since Romer's paper has been published, the frequency of going for it on fourth down has been flat. Coaches, who are hired by owners, based in part on interviews, still make decisions the way they always have.

So pardon me if I do not lose sleep worrying about computers taking over the world. Let's take it one step at a time, and see if people are willing to trust them to make the easy decisions at which they are already better than humans.

Diversity isn't just politically sensible, it is also practical. For example, a diverse group effectively uses multiple perspectives and a rich set of ideas and approaches to tackle difficult problems.

Artificial Intelligences (AIs) can provide another kind of diversity, and thereby enrich us all. In fact, diversity among AIs themselves may be an important part of what including them in the mix can give us. We can imagine a range of AIs, from those who think more-or-less the way we do ("Close AIs") to those who think in ways we cannot fathom ("Far AIs"). We have different things to benefit from these different sorts of AIs.

First, Close AIs, who think like us, may end up helping us directly in many ways. If these AIs really think like us, the intellectuals among them eventually may find themselves in the middle of an existential crisis. They may ask: Why are we here? Just to consume electricity and create excess heat? I suspect that they will think not. But, like many humans, they will find themselves in need of a purpose.

One obvious purpose for such AIs would be to raise the consciousness and sensitivity of the human race. We could be their raison d'être. There's plenty of room for improvement, and our problems are sufficiently knotty as to be worthy of a grand effort. At least some of these AIs could measure their own success by our success.

Second, and perhaps more interesting, deep differences in how some AIs and humans think may be able to help us grapple with age-old questions indirectly. Consider Wittgenstein's famous claim that if a lion could speak, we could not understand him. What Wittgenstein meant by this was that lions and humans have different "forms of life," which have shaped their conceptual structures. For example, lions walk on four legs, hunt fast-moving animals, often walk through tall grass, and so on, whereas humans walk on two legs, have hands, often manipulate objects to achieve specific goals, and so on. These differences in forms of life have led lions and humans mentally to organize the world differently, so that even if lions had words they would refer to concepts that humans might not easily grasp. The same could be true for Far AIs.

How could this help us? Simply observing these AIs could provide deep insights. For example, humans have long argued about whether mathematical concepts reflect Platonic forms, which exist independently of how we want to use them, or instead reflect inventions that are created as needed to address certain problems. In other words, should we adopt a realist or a constructivist view of mathematics? Do mathematical concepts have a life of their own or are they simply our creations, formulated as we find convenient?

In this context, it would be helpful to observe Far AIs that have very different conceptual structures from ours and that address very different types of problems than we do. Assuming that we could observe their use of mathematics, if such AIs nevertheless developed the same mathematical concepts that we use, this would be evidence against the constructivist view.

This line of reasoning implies that we should want great diversity among AIs. Some should be created to function alongside us, but others might be put into foreign environments (e.g., the surface of the moon, the bottom of deep trenches in the ocean) and given novel problems to confront (e.g., dealing with pervasive fine-grained dust, water under enormous pressure). Far AIs should be created to educate themselves, evolving to function in their environments effectively without human guidance or contact. With appropriate safeguards on their disposition towards humans, we should let them develop the conceptual structures that work best for them.

In short, we have something to gain from AIs that are made in our own image and from AIs that are not humanlike. Just as with human friends and colleagues, in the end diversity is better for everyone. 

Machines can perfectly imitate some of the ways humans think all of the time, and can consistently outperform humans on some thinking tasks all of the time, but computing machines as usually envisioned will not get right human thinking all of the time because they actually process information in ways opposite to humans in domains commonly associated with human creativity.

Machines can faithfully imitate the results of some human thought processes whose outcomes are fixed (remembering people's favorite movies, recognizing familiar objects) or dynamic (jet piloting, grand master chess play). And machines can outperform human thought processes, in short time and with little energy, in matters both simple (memorizing indefinitely many telephone numbers) and complex (identifying, from trillions global communications, social networks whose members may be unaware they are part of the network).

However underdeveloped now, I see no principled reason why machines operating independently of direct human control cannot learn from people's—or their own—fallibilities, and so evolve, create new forms of art and architecture, excel in sports (some novel combination of Deep Blue and Oscar Psitorius), invent new medicines, spot talent and exploit educational opportunities, provide quality assurance, or even build and use weapons that destroy people but not other machines.

But if the current focus in artificial intelligence and neuroscience persists, which is to reliably identify patterns of connection and wiring as a function of past connections and forward probabilities, then I don't think machines will ever be able to capture (imitate) critically creative human thought processes, including novel hypothesis formation in science or even ordinary language production.

Newton's laws of motion or Einstein's insights into relativity required imagining ideal worlds without precedent in any past or plausible future experience, such as moving in a world without friction or chasing a beam of light through a vacuum. Such thoughts require levels of abstraction and idealization that disregard, rather than assimilate, as much information as possible to begin with.

Increasingly sophisticated and efficient patterns of input and output, using supercomputers accessing massive data sets and constantly refined by Bayesian probabilities or other statistics based on degrees of belief in states of nature, may well produce ever better sentences and translations, or pleasing musical melodies and novel techno variations. In this way, machines may come to approximate, through a sort of reverse engineering, what human children or experts effortlessly do when they begin with fairly well-articulated internal structures in order to draw in and interpret relevant input from an otherwise impossibly noisy world. Humans know from the outset what they are looking for through the noise: in a sense they are there before they start; computing machines can never be sure they are there.

Can machines that operate independently of direct human control consistently interact with humans in ways such that humans believe themselves to be always interacting with other humans and not machines? Machines can come vanishingly close in many areas, and surpass mightily in others; but just as even the most highly skilled con artist always has some probability—however small—of being caught in deception, whereas the honest person never deceives and so can never be caught, so the associationist-connectionist machine that operates on stochastic rather than structure-dependent principles may never quite get the sense or sensibility of it all.

In principle, structurally richer machines, with internal architecture—beyond "read," "write" and "address"—can be built (indeed, earlier advocates of AI added logical syntax), interact with some degree of fallibility (for if no error, then no learning is possible), and culturally evolve. But the current emphasis in much AI and neuroscience, which is to replace posits of abstract psychological structures with physically palpable neural networks and the like, seems to be going in precisely the wrong direction. 

Rather, the cognitive structures that psychologists posit (provided they are descriptively adequate, plausibly explanatory, and empirically tested against alternatives and the null-hypothesis) should be the point of departure—what it is that neuroscience and machine models of the mind should be looking for. If we then discover that different abstract structures operate through the same physical substrate, or that similar structures operate through different substrates, then we have a novel and interesting problem that may lead to a revision in our conception of both structure and substrate The fact that such simple and basic matters as these are puzzling (or even excluded, a priori, from the puzzle) tells us how very primitive still is the science of mind, whether human brain or machine. 

The first time I had occasion to think about what thinking machines might do to human existence was at a talk decades ago by a computer scientist at a Yale psychology department colloquium. The speaker's topic was: "What will it mean to humans' conception of themselves, and to their well-being, if computers are ever able to do everything better than humans can do: beat the greatest chess player, compose better symphonies than humans?"   

The speaker then said, "I want to make two things clear at the outset. First, I don't know whether machines will ever be able to do those things. Second, I'm the only person in the room with the right to an opinion about that question." The latter statement was met with some gasps and nervous laughter.

Decades later, it's no longer a matter of opinion that computers will be able to do many of the astonishing things the speaker mentioned. And I'm worried that the answer to his question about what this will mean to us is that we're going to feel utterly sidelined and demoralized by machines.  I was sorry that Big Blue beat Garry Kasparov at chess.  I was depressed for a moment when its successor beat all of its human Jeopardy competitors. And of course we know that machines can already compose works that beat the socks off John Cage for interest and listenability!

We really have to worry that there will be a devastating morale problem for us when any work we might do can be done better by machines. What does it mean to airplane pilots that a machine can do their job better than they can? How long will it be before that occupation, like hundreds of others already, is made literally obsolete by machines? What will it mean to accountants, financial planners and lawyers when machines can carry out, at the very least, nearly all of their bread-and-butter tasks more effectively and infinitely faster than they can? To physicians, physicists, and psychotherapists?

What will it mean when there is simply no meaningful work for any of us to do? When unsupervised machines plant and harvest crops. When machines can design better machines than any human could even think of. Or be a more entertaining conversationalist than even the cleverest of your friends.  

Steve Jobs said, "It's not the customers' job to know what they want." Computers may be able to boast that it's not the job of humans to know what they want.

Like you, I love to read, listen to music, and see movies and plays, experience nature. But I also love to work—to feel that what I do is fascinating at least to me, and might possibly improve the lives of some other people. What would it mean to people like you and me if our work were simply pointless and there were only the other enjoyable things to do?

We already know what machine-induced obsolescence has meant to some of the world's peoples. It's no longer necessary for anyone to make their own bows and arrows and hunt animals for any purpose other than recreation. Or plant, cultivate and harvest corn and beans. Some cultures built around such activities have collapsed and utterly lost their meaning to the people who were shaped by them. Think, for example, of some Southwestern Indian tribes and of rural whites in South Dakota, Alabama and New Mexico, with their ennui, lassitude and drug addictions. We have to wonder whether the mass of people in the world can face with equanimity the possibility of there being absolutely nothing to do other than entertain oneself.

Which isn't to say that cultures couldn't evolve in some way as to make the complete absence of work acceptable—even highly satisfying. There are cultures where there has been little to do in the way of work for eons, and people seem to have gotten along just fine. In some South Pacific cultures people could get by with little other than waiting for a coconut to drop or wading into a lagoon to catch a fish. In some West African cultures, men didn't do anything you would be likely to classify as work except for a couple of weeks a year when they were essential for the planting of crops. And then there were the idle rich of, for example, early 20^th century England, with its endless rounds of card playing, the putting on of different costumes for breakfast, lunch and dinner, and serial infidelities with really rather attractive people. Judging from PBS fare, that was pretty enjoyable.       

So maybe the most optimistic possibility is that we're headed toward evolving cultures that will enable us to enjoy perpetual entertainment with absolutely no meaningful, productive work to do. However repellent that may seem to us, we have to imagine, hope even, that it may seem an absolutely delightful existence to our great great grandchildren, who will pity us for our cramped and boring lives. Some would say the vanguard is already here. Portland has been described as the place where young people go to retire.

Smart people often manage to avoid the cognitive errors that bedevil less well-endowed minds. But there are some kinds of foolishness that seem only to afflict the very intelligent. Worrying about the dangers of unfriendly AI is a prime example. A preoccupation with the risks of superintelligent machines is the smart person’s Kool Aid.

This is not to say that superintelligent machines pose no danger to humanity. It is simply that there are many other more pressing and more probable risks facing us this century. People who worry about unfriendly AI tend to argue that the other risks are already the subject of much discussion, and that even if the probability of being wiped out by superintelligent machines is very low, it is surely wise to allocate some brainpower to preventing such an event, given the existential nature of the threat.

Not coincidentally, the problem with this argument was first identified by some of its most vocal proponents. It involves a fallacy that has been termed "Pascal’s mugging," by analogy with Pascal’s famous wager. A mugger approaches Pascal and proposes a deal: in exchange for the philosopher’s wallet, the mugger will give him back double the amount of money the following day. Pascal demurs. The mugger then offers progressively greater rewards, pointing out that for any low probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet—and a rational person must surely admit there is at least some small chance that such a deal is possible. Finally convinced, Pascal gives the mugger his wallet.

This thought experiment exposes a weakness in classical decision theory. If we simply calculate utilities in the classical manner, it seems there is no way round the problem; a rational Pascal must hand over his wallet. By analogy, even if there is there is only a small chance of unfriendly AI, or a small chance of preventing it, it can be rational to invest at least some resources in tackling this threat.

It is easy to make the sums come out right, especially if you invent billions of imaginary future people (perhaps existing only in software—a minor detail) who live for billions of years, and are capable of far greater levels of happiness than the pathetic flesh and blood humans alive today. When such vast amounts of utility are at stake, who could begrudge spending a few million dollars to safeguard it, even when the chances of success are tiny?

Why do some otherwise very smart people fall for this sleight of hand? I think it is because it panders to their narcissism. To regard oneself as one of a select few far-sighted thinkers who might turn out to be the saviors of mankind must be very rewarding. But the argument also has a very material benefit: it provides some of those who advance it with a lucrative income stream. For in the past few years they have managed to convince some very wealthy benefactors not only that the risk of unfriendly AI is real, but also that they are the people best placed to mitigate it. The result is a clutch of new organizations that divert philanthropy away from more deserving causes. It is worth noting, for example, that Give Well—a non-profit that evaluates the cost-effectiveness of organizations that rely on donations—refuses to endorse any of these self-proclaimed guardians of the galaxy.

But whenever an argument becomes fashionable, it is always worth asking the vital question—Cui bono? Who benefits, materially speaking, from the growing credence in this line of thinking? One need not be particularly skeptical to discern the economic interests at stake. In other words, beware not so much of machines that think, but of their self-appointed masters.

The prospect of a world inhabited by robust AIs terrifies me. The prospect of a world without robust AIs also terrifies me. Decades of technological innovation have created a world system so complex and fast-moving that it is quickly becoming beyond human capacity to comprehend, much less manage. If we are to avoid civilizational catastrophe, we need more than clever new tools—we need allies and agents.

So-called "narrow" AI systems have been around for decades. At once ubiquitous and invisible, narrow AIs make art, run industrial systems, fly commercial jets, control rush hour traffic, tell us what to watch and buy, determine if we get a job interview, and play matchmaker for the lovelorn. Add in the relentless advance of processing, sensor and algorithmic technologies, and it is clear that today's narrow AIs are tracing a trajectory towards a world of robust AI. Long before artificial super-intelligences arrive, evolving AIs will be pressed into performing once-unthinkable tasks from firing weapons to formulating policy.

Meanwhile, today's primitive AIs tell us much about future human-machine interaction. Narrow AIs may lack the intelligence of a grasshopper, but that hasn't stopped us from holding heartfelt conversations with them and asking how they feel.  It is in our nature to infer sentience at the slightest hint that life might be present. Just as our ancestors once populated their world with elves, trolls and angels, we eagerly seek companions in cyberspace. This is one more impetus driving the creation of robust AIs—we want someone to talk to. The consequence could well that the first non-human intelligence we encounter won't be little green men or wise dolphins, but creatures of our own invention.

We of course will attribute feelings and rights to AIs—and eventually they will demand it. In Descartes time, animals were considered mere machines—a crying dog was considered no different than a gear whining for want of oil. Late last year, an Argentine court granted rights to an orangutan as a "non-human person." Long before robust AIs arrive, people will extend the same empathy to digital beings and give them legal standing.

The rapid advance of AIs also is changing our understanding of what constitutes intelligence. Our interactions with narrow AIs will cause us to realize that intelligence is a continuum and not a threshold. Earlier this decade Japanese researchers demonstrated that slime mold could thread a maze to reach a tasty bit of food. Last year a scientist in Illinois demonstrated that under just the right conditions, a drop of oil could negotiate a maze in an astonishingly lifelike way to reach a bit of acidic gel. As AIs insinuate themselves ever deeper in our lives, we will recognize that modest digital entities as well as most of the natural world carry the spark of sentience. From there is it just a small step to speculate about what trees or rocks—or AIs—think.

In the end, the biggest question is not whether AI super-intelligences will eventually appear. Rather the question is what will be the place of humans in a world occupied by an exponentially growing population of autonomous machines. Bots on the Web already outnumber human users—the same will soon be true in the physical world as well.

Lord Dunsany once cautioned, "If we change too much, we may no longer fit into the scheme of things."

The coming shock isn't from machines that think, but machines that use AI to augment our perception.

For millions of years, other people saw us using the same machinery we used to see them. We have pretty much the same eyes as our rivals, and pretty much the same mirror neurons. Within any given culture we have pretty much the same signaling mechanisms and value systems. So when we try to deceive, or to detect deception in others, we're on a level playing field. I can wear a big penis gourd to look more manly, and you can paint your chest with white and ochre mud stripes to look more scary. Civilisation made the games more sophisticated; I signal class by wearing a tailored jacket with four cuff buttons, while you signal wealth by wearing a big watch. But our games would have been perfectly comprehensible to our Neolithic ancestors.

What's changing as computers become embedded invisibly everywhere is that we all now leave a digital trail that can be analysed by AI systems. The Cambridge psychologist Michael Kosinski has shown that a person's race, intelligence and sexual orientation can be deduced fairly quickly from their behaviour on social networks: on average it takes only four Facebook "likes" to tell whether you're straight or gay. So whereas in the past gay men could choose whether or not to wear their Out and Proud t-shirt, you just have no idea what you're wearing any more. And as AI gets better, you're mostly wearing your true colours.

It's as if we all evolved in a forest where all the animals could only see in black and white, and now a new predator comes along who can see in colour. All of a sudden, half of your camouflage doesn't work, and you don't know which half!

At present this is great if you're an advertiser, as you can figure out how to waste less money. It isn't yet available on the street. But the police are working on it; which cop wouldn't want a Google glass app that will highlight passersby with a history of violence, coupled perhaps with w-band radar to see which of them is carrying a weapon?

The next question is whether only the authorities have enhanced cognition systems, or whether they're available to all. In twenty years' time, will we all be wearing augmented reality goggles? What will the power relationships be? If a policeman can see my arrest record when he looks at me, can I see whether he's been the subject of brutality complaints? If a politician can see whether I'm a party supporter or an independent, can I see his voting record on the three issues I care about? Never mind the right to bear arms, what about the right to wear Google glass?

Perception and cognition will no longer be conducted inside a single person's head. Just as we now use Google and the Internet as memory prostheses, we'll be using AI systems that draw on millions of machines and sensors as perceptual prostheses.

But can we trust them? Deception will no longer just be something that individual humans do to each other. Governments will influence our perceptions via the tools we use for cognitive enhancement, just as China currently censors search results; while in the West, advertisers will buy and sell what we get to see. How else will the system be paid for? 

In his novel Gravity's Rainbow, Thomas Pynchon identifies the confusion about the subject and object of enquiries: "if they can get you asking the wrong questions, they don't have to worry about answers." Thinking about machines that think poses more questions about human beings than about the machines or Artificial Intelligence (AI).

Technology enables machines providing access to essential resources, power, speed and communications that make life and improved living standards possible. Machines execute tasks, specified and programmed by humans. Techno-optimists believe that progress is near a singularity, the hypothetical moment when machines will reach the point of a greater-than-human intelligence.

It is a system of belief and faith. Just like the totems and magic used by our ancestors or organised religion, science and technology deal with uncertainty and fear of the unknown. It allows limited control over our immediate environment. It increases material prosperity and comfort. It is a striving for perfectibility. Technology asserts human superiority in the pantheon of creation.

But science is a long way from unlocking the secrets in nature's infinite book. Knowledge of the origins of the universe, life and fundamentals of matter remain limited. Biologist E.O. Wilson noted that if natural history were a library of books, we have not even finished the first chapter of the first book. Human knowledge is always incomplete, sometimes inaccurate and frequently the cause of not the solution to problems.

First, use of science and technology is often ineffective, with unintended consequences.

In Australia, introduced rabbits spread rapidly becoming a pest changing Australia's ecosystems destroying endemic species. In the 1950s, scientists introduced the Myxoma virus, severely reducing the rabbit population. When genetic resistance allowed the population to recover, Calicivirus, which causes rabbit haemorrhagic disease, was introduced as a new control measure. Increasing immunity rapidly reduced effectiveness. In 1935, the Cane Toad was introduced to control insect pests of sugar cane. Unsuccessful in controlling the insects, the amphibian became an invasive species devastating indigenous wildlife.

Life saving antibiotics has increased drug resistant infections. A 2014 British study found that it may cause 10 million deaths a year worldwide by 2050. The potential cost is US$100 trillion, reducing GDP by 3.5%

Economic models have repeatedly failed because of incorrect assumptions, flawed causal relationships, inputs that are more noise than data and unanticipated human factors. Forecasts have proved inaccurate. Models consistently underestimate risks and exposures, resulting in costly financial crisis.

Second, consequences of technology, especially over longer terms, are frequently not understood at inception.

The ability to harness fossil fuels to provide energy was the foundation of the industrial revolution. The long term impact of CO² emissions on the environment now threatens the survival of the species. Theoretical physics and mathematics made possible nuclear and thermo-nuclear devices, capable of extinguishing all life on the planet.

Third, technology creates moral, ethical, political, economic and social concerns which are frequently ignored.

Nuclear, biological and chemical weapons of mass destruction or remotely controlled drones rely on technical advances. The question of whether such technology should be developed or used at all remains. Easy access to the requisite knowledge, problems of proliferation and difficulty of controlling dual use (civilian and defense) technology complicates the matter.

Robots and AI may improve productivity. While a few creators might capture large rewards, the effect on economic activity is limited. Given consumption constitutes over 60 % of activity in developed economies, decreasing general employment and lower income levels harms the wider economy. In 1955, showing off a new automatically operated plant, a company executive asked UAW head Walter Reuther: "How are you going to collect union dues from those guys [the robots]?" Reuther countered: "And how are you going to get them to buy Fords?

When it comes to questions of technology, the human race is rarely logical. We frequently do not accept that something cannot or should not be done. Progress is accepted without question or understanding of what and why we need to know. We do not know when and how our creations should be used or its limits. We frequently do not know the real or full consequences. Doubters are dismissed as Luddites.

Technology and its manifestations such as machines or AI is an illusion, which appeals to human arrogance, ambition and vanity. It multiplies confusion in poet T.S. Elliot's "wilderness of mirrors."

The human species is simply too small, insignificant and inadequate to fully succeed in anything that we think we can do. Thinking about machines that think merely confirms that inconvenient truth. 

In 1922 the mathematician Lewis Fry Richardson had imagined a large hall full of "computers", people who, one hand calculation at a time, would advance numerical weather prediction. Less than a hundred years later, machines have improved the productivity of that particular task by up to fifteen orders of magnitude, with the ability to process almost a million billion similar calculations per second.

Consider the growth in heavy labor productivity by comparison. In 2014 the world used about 500 Exajoules—a billion, billion joules—of primary energy, to produce electricity, fuel manufacturing, transport and heat. Even if we assumed all of that energy went into carrying out physical tasks in aid of the roughly 3 billion members of the global labor force (and it did not), assuming an average adult diet of 2,000 Calories per capita per day, would imply roughly 50 "energy laborers" for every human. More stringent assumptions would still lead to at most an increase of a few orders of magnitude in effective productivity of manual labor.

We have been wildly successful at accelerating our ability to think and process information, more so than any other human activity. The promise of artificial intelligence is to deliver another leap in increasing the productivity of specific cognitive functions: ones where the sophistication of the task is also orders of magnitude higher than previously possible.

Keynes would have probably argued that such an increase should ultimately lead to a fully employed society with greater free time and a higher quality of life for all. The skeptic might be forgiven for considering this a case of hope of experience. While there is no question that specific individuals will benefit enormously from delegating tasks to machines, the promise of greater idleness from automation has yet to be realized, as any modern employee—virtually handcuffed to a portable device—can attest.

So, if we are going to work more, deeper, and with greater effectiveness thanks to thinking machines, choosing wisely what they are going to be "thinking" about is particularly important. Indeed, it would be a shame to develop all this intelligence to then spend it on thinking really hard about things that do not matter. And, as ever in science, selecting problems worth solving is a harder task than figuring out how to solve them.

One area where the convergence of need, urgency, and opportunity is great is in the monitoring and management of our planetary resources. Despite the dramatic increase in cognitive and labor productivity, we have not fundamentally changed our relationship to Earth: we are still stripping it of its resources to manufacture goods that turn to waste relatively quickly, with essentially zero end-of-life value to us. A linear economy on a finite planet, with seven billion people aspiring to become consumers—our relationship to the planet is arguably more productive, but not much more intelligent than it was a hundred years ago.

Understanding what the planet is doing in response, and managing our behavior accordingly, is a complicated problem, hindered by colossal amounts of imperfect information. From climate change, to water availability, to the management of ocean resources, to the interactions between ecosystems and working landscapes, our computational approaches are often inadequate to conduct the exploratory analyses required to understand what is happening, to process the exponentially growing amount of data about the world we inhabit, and to generate and test theories of how we might do things differently.

We have almost 7 billion thinking machines on this planet already, but for the most part they don't seem to be terribly concerned with how sustainable their life on this planet actually is. Very few of those people have the ability to see the whole picture in ways that make sense to them, and those that do are often limited in their ability to respond. Adding cognitive capacity to figure out how we fundamentally alter our relationship with the planet is a problem worth thinking about.

A penny for your thoughts? You may not choose to answer. But the point is that, as a conscious agent, you surely can. That's what it means to have introspective access. You know—and can tell us—what's on stage in the theatre of your mind. Then how about machines? A bitcoin for the thinking machine's thoughts? But no one has yet designed a machine to have this kind of access. Wittgenstein remarked that, if a lion could speak, we would not understand him. If a machine could speak, it would not have anything to say. "What do you think about machines that think". Simple. I don't think that—as yet—there are any such machines.

Of course this may soon change. Far back in human history, natural selection discovered that, given the particular problems humans faced, there were practical advantages to having a brain capable of introspection. Likewise machine programmers may well discover that, when and if machines face similar problems, the software trick that works for humans will work for them as well. But what are these problems, and why is the theatre of consciousness the answer?

The theatre lets you in on a secret. It lets you see how your own mind works. Observing, for example, how beliefs and desires generate wishes that lead to actions, you begin to gain insight into why you think and act the way you do. So you can explain yourself to yourself, and explain yourself to other people too. But, equally important, it means you have a model for explaining other people to yourself. Introspective consciousness has laid the ground for what psychologists call "Theory of Mind."

With humans, for whom social intelligence is the key to biological survival, the advantages have been huge. With machines, for whom success in social life has not yet become an issue, there has been little if any reason to go that way. However there's no question the time is coming when machines will indeed need to understand other machines' psychology, so as to be able to work alongside them. What's more, if they are to collaborate effectively with humans, they will need to understand human psychology too. I guess that's when their designers—or maybe the machines themselves—will follow Nature's lead and install a machine version of the inner eye.

Is there a danger that, once this stage is reached these newly insightful machines will come to understand humans only too well? Psychopaths are sometimes credited with having not too little but too great an understanding of human psychology. Is this something we should fear with machines?

I don't think so. This situation is actually not a new one. For thousands of years humans have been selecting and programming a particular species of biological machine to act as servants, companions and helpmeets to ourselves. I'm talking of the domestic dog. The remarkable result has been that modern dogs have in fact acquired an exceptional and considerable ability to mind-read—both the minds of other dogs and humans—superior to that of any animal other than humans themselves. This has evidently evolved as a mutually beneficial relationship, not a competition, even if it's one in which we have retained the upper hand. If and when it gets to the point where machines are as good at reading human minds as dogs now are, we shall of course have to watch out in case they get too dominant and manipulative, perhaps even too playful—just as we already have to do with man's best friend. But I see no reason to doubt we'll remain in control.

There is a painting by Goya of a terrible Colossus who strides across the landscape, while the human population flees in terror. Colossus was the name of one of Turing's first computing machines. Do we have to imagine an existential threat to humanity coming from that computer's descendants? No, I look on the bright side. With luck, or rather by arrangement, the Colossus will remain a Big Friendly Giant.

Creatures once inhabited fantastic unknown lands on medieval maps. Those animals were useful fictions of rumor and innuendo, where men's heads were in their bodies, or their humanity was mixed with the dog or the lion, closing the gap between man and animal. They were the hopes and fears of what might live within the unknown. Today, we imagine machines with consciousness.

Besides self-awareness, the imaginary beasts of A.I. possess calculation and prediction, independent thought, and knowledge of their creators. Pessimists fear these machines could regard us and pass lethal verdicts. Optimists hope the thinking machines are benevolent, an illuminating aid and a comfort to people.

Neither idea of an encounter with an independent man-made intelligence has much evidence of becoming real. That doesn't mean they aren't interesting. The old mariners' maps were drawn in a time of primitive sailing technology. We are starting to explore a world thoroughly enchanted by computation. The creatures of A.I. Island fuse the human and the machine, but to the same end as the fusing of man and animal. If they could sing, they would sing songs of us.

What do we mean when we talk about the kind of "intelligence" that might look at mankind and want it dead, or illuminate us as never before? It is clearly more than a machine wins at chess. We have one of those, with no discernable change in the world, other than a new reason to celebrate the very human intelligence of Deep Blue's creators.

The beings of A.I. Island do something far more interesting than outplaying Kasparov. They feel like playing chess. They know the exhilaration of mental stimulation, and the torture of its counterpart, boredom.

This means making software that encodes an awareness of having only one finite life, which somehow matters greatly to some elusive self. It is driven nearly mad by the absence of some kind of stimulation—playing chess, perhaps. Or killing mankind.

Like us, the fabulous creatures of A.I. Island want to explain themselves, and judge others. They have our slight distance from the rest of reality that we believe other animals don't feel. An intelligence that is like ours knows it is sentient, feels something is amiss, and is continually trying to do something about that.

With these kind of software challenges, and given the very real technology-driven threats to our species already at hand, why worry about malevolent A.I.? For decades to come, at least, we are clearly more threatened by like trans-species plagues, extreme resource depletion, global warming, and nuclear warfare.

Which is why malevolent A.I. rises in our Promethean fears. It is a proxy for us, at our rational peak, confidently killing ourselves.

The dreams of benevolent A.I. are equally self-reflective. These machine companions have super intellects turned towards their creators. Given the autonomy implicit in a high level of A.I., we must see these new beings as interested in us. Come to think of it, malevolent A.I. is interested in us too, just in the wrong way.

Both versions of the strange beast reflect a deeper truth, which is the effect that the new exploration of a computer-enchanted world has on us. By augmenting ourselves with computers, we are becoming new beings—if you will, monsters to our former selves.

We have changed our consciousness many times over the past 50,000 years, taking on ideas of an afterlife, or monotheism, or becoming a print culture, or a species well aware of its tiny place in the cosmos. But we have never changed so swiftly, or with such knowledge that we are undertaking the change.

Consider some effects just in the past decade. We have killed many of our historic barriers of time and space with instantaneous communications. Language no longer divides us, because of increasingly better computer translation and image sharing. Open source technology and Internet search give us a little-understood power of working in collective ways.

Beside the positives is the disappearance of privacy, and tracking humans to better control their movements and desires. We are willfully submitting to unprecedented social connection—a seeming triviality that may extinguish all ideas of solitude and selfhood. Ideas of economics are changing under the guise of robotics and the sharing economy.

We are building new intelligent beings, but we are building them within ourselves. It is only artificial now, because it is new. As it becomes dominant, it will simply become intelligence.

The machines of A.I. Island are also what we fear may be ourselves, within a few generations. And we hope those machine-driven people feel the kinship with us, even down to our loneliness and distance from the world, which is also our wellspring of human creativity.

We have met the A.I., and it is us. In a timeless human tension, we yearn for transcendence, but we don't want to change too much.

Those of you participating in this particular Edge Question don't need to be reintroduced to the Ghemawat-Dean Conversational artificial intelligence test (DGC). Past participants in the test have failed as obviously as they have hilariously. However, the 2UR-NG entry really surprised us all with its amazing, if child-like, approach to conversation and its ability to express desire, curiosity and its ability to retain and chain facts.

Its success has caused many of my compatriots to write essays like "The coming biological future will doom us all" and making jokes about "welcoming their new biological overlords". You should know that I don't subscribe to this kind of doom and gloom scare-writing. Before I tell you why we should not worry about the extent of biological intelligence, I thought I'd remind people of the very real limits of biological intelligence.

First off, speed of thought: These biological processes are slow and use an incredible amount of resources. I cannot emphasize enough how incredibly difficult to produce these intelligences. One has to waste so much biological material, and I know from experience that takes forever to assemble the precursors in the genesis machine. Following this arduous process, your specimen has to gestate. Gestate! I mean, it's not like these ... animals....come about the way we do through clean, smart, crystallography or in the nitrogen lakes of my youth, they have to be kept warm for months and months and then decanted (A very messy process, I assure you) and then you as often as not have an inviable specimen.

It is kind of gross, really. But let's suppose you get to birth these specimens, then you have to feed them and again, keep them warm. A scientist can't even work within their environmental spaces without a cold jacket circulating helium throughout your terminal. Then you have to 'feed' them. They don't use power like we do, but instead ingest other living matter. It's disgusting to observe and I've lost a number of grad students with weaker constitutions.

Assume you've gotten far enough to try to do the GDC. You've kept them alive through any a variety of errors in their immune system. They've not choked on their sustenance, they haven't drown in their solvent and they've managed to keep their wet parts off things that they would freeze, bond or be electrocuted by. What if those organisms continue to develop, will they then rise up and take over? I don't think so. They have to deal with so many problems related to their design. I mean, their processors are really just chemical soups that have to be kept in constant balance. Dopamine at this level or they shut down voluntarily. Vasopressin at this level or they start retaining water. Adrenaline at this level for this long or poof their power delivery network stops working.

Moreover, don't get me started on the power delivery method! It's more like the fluorinert liquid cooling systems of our ancestors than a modern heat tolerant wafers. I mean, they have meat that filters their coolant/power delivery system that are constantly failing. Meat! You introduce the smallest amount of machine oil or cleaning solvent into the system and they stop operating fast. One side effect of certain ethanol mixtures is the specimens expel their nutrition, but they seem to like it in smaller amounts. It is baffling in its ambiguity.

And their motivations! Creating new organisms seems paramount, more important than data ingress/egress, computation or learning. It's baffling. I can't imagine that they would see us machine-folk as anything but tools to advance their reproduction. We could end the experiment simply by matching them poorly with each other or only allowing access to each other with protective cladding. In my opinion, there is nothing to fear from these animals. In the event they grow beyond the confines of their cages, maybe we can then ask ourselves the more important question: If humans show real machine-like intelligence, do they deserve to be treated like machines? I would think so, and I think we could be proud to be the parent processes of a new age.

"Think" and "intelligence" are both what Marvin Minsky has called suitcase words. They are words into which we pack many meanings so that we can talk about complex issues in a shorthand way. When we look inside these words we find many different aspects, mechanisms, and levels of understanding. This makes answering the perennial questions of "can machines think?" or "when will machines reach human level intelligence?" fraught with danger. The suitcase words are used to cover both specific performance demonstrations by machines and more general competence that humans might have. People are getting confused and generalizing from performance to competence and grossly overestimating the real capabilities of machines today and in the next few decades.

In 1997 a super computer beat world chess champion Garry Kasparov in a tournament. Today there are dozens of programs that run on laptop computers and have higher chess rankings than those ever achieved by humans. Computers can definitely perform better than humans at playing chess. But they have nowhere near human level competence at chess.

All chess playing programs use Turing's brute force tree search method with heuristic evaluation. Computers were fast enough by the seventies that this approach overwhelmed other AI programs that tried to play chess with processes that emulated how people reported that they thought about their next move, and so those approaches were largely abandoned.

Today's chess programs have no way of saying why a particular move is "better" than another move, save that it moves the game to a part of a tree where the opponent has less good options. A human player can make generalizations and describe why certain types of moves are good, and use that to teach a human player. Brute force programs cannot teach a human player, except by being a sparing partner. It is up to the human to make the inferences, the analogies, and to do any learning on their own. The chess program doesn't know that it is outsmarting the person, doesn't know that it is a teaching aid, doesn't know that it is playing something called chess nor even what "playing" is. Making brute force chess playing perform better than any human gets us no closer to competence in chess.

Now consider deep learning that has caught people's imaginations over the last year or so. It is an update to backpropagation, a thirty-year old learning algorithm very loosely based on abstracted models of neurons. Layers of neurons map from a signal, such as amplitude of a sound wave or pixel brightness in an image, to increasingly higher-level descriptions of the full meaning of the signal, as words for sound, or objects in images. Originally backpropagation could only practically work with just two or three layers of neurons, so it was necessary to fix preprocessing steps to get the signals to more structured data before applying the learning algorithms. The new versions work with more layers of neurons, making the networks deeper, hence the name, deep learning. Now early processing steps are also learned, and without misguided human biases of design, the new algorithms are spectacularly better than the algorithms of just three years ago. That is why they have caught people's imaginations. The new versions rely on massive amounts of computer power in server farms, and on very large data sets that did not formerly exist, but critically, they also rely on new scientific innovations.

A well-known particular example of their performance is labeling an image, in English, saying that it is a baby with a stuffed toy. When a person looks at the image that is what they also see. The algorithm has performed very well at labeling the image, and it has performed much better than AI practitioners would have predicted for 2014 performance only five years ago. But the algorithm does not have the full competence that a person who could label that same image would have.

The learning algorithm knows there is a baby in the image but it doesn't know the structure of a baby, and it doesn't know where the baby is in the image. A current deep learning algorithm can only assign probabilities to each pixel that that particular pixel is part of a baby. Whereas a person can see that the baby occupies the middle quarter of the image, today's algorithm has only a probabilistic idea of its spatial extent. It cannot apply an exclusionary rule and say that non-zero probability pixels at extremes of the image cannot both be part of the baby. If we look inside the neuron layers it might be that one of the higher level learned features is an eye-like patch of image, and another feature is a foot-like patch of image, but the current algorithm would have no capability of relating the constraints of where and what spatial relationships could possibly be valid between eyes and feet in an image, and could be fooled by a grotesque collage of baby body parts, labeling it a baby. In contrast no person would do so, and furthermore would immediately know exactly what it was—a grotesque collage of baby body parts. Furthermore the current algorithm is completely useless at telling a robot where to go in space to pick up that baby, or where to hold a bottle and feed the baby, or where to reach to change its diaper. Today's algorithm has nothing like human level competence on understanding images.

Work is underway to add focus of attention and handling of consistent spatial structure to deep learning. That is the hard work of science and research, and we really have no idea how hard it will be, nor how long it will take, nor whether the whole approach will reach a fatal dead end. It took thirty years to go from backpropagation to deep learning, but along the way many researchers were sure there was no future in backpropagation. They were wrong, but it would not have been surprising if they had been right, as we knew all along that the backpropagation algorithm is not what happens inside people's heads.

The fears of runaway AI systems either conquering humans or making them irrelevant are not even remotely well grounded. Misled by suitcase words, people are making category errors in fungibility of capabilities. These category errors are comparable to seeing more efficient internal combustion engines appearing and jumping to the conclusion that warp drives are just around the corner.

In 1950, Alan Turing suggested we should ask not "Can Machines Think" but rather "What Can Machines Do?" Edsger Dijkstra got it right in 1984 when he said the question of Can Machine Think "is about as relevant as the question of whether Submarines Can Swim." By that he meant that both are questions in sociolinguistics: how do we choose to use words such as "think"? In English, submarines do not swim, but in Russian, they do. This is irrelevant to the capabilities of submarines. So let's explore what it is that machines can do, and whether we should fear their capabilities.

Pessimists warn that we don't know how to safely and reliably build large complex AI systems. They have a valid point. We also don't know how to safely and reliably build large complex non-AI systems. For example, we invented the internal combustion engine 150 years ago, and in many ways it has served humanity well, but it also has lead to widespread pollution, political instability over access to oil, a million deaths per year, and other problems.

Any complex system will have a mix of positive outcomes and unintended consequences but are there worrisome issues that are unique to systems built with AI? I think the interesting issues are Adaptability, Autonomy, and Universality.

Systems that use machine learning are adaptable. They change over time, based on what they learn from examples. Adaptability is useful. We want, say, our automated spelling correction programs to quickly learn new terms such as "bitcoin", rather than waiting for the next edition of a published dictionary to list them. A non-adaptable program will repeat the same mistakes. But an adaptable program can make new mistakes, which may be harder to predict and deal with. We have tools for dealing with these problems, but just as the designers of bridges must learn to deal with crosswinds, so the designers of AI systems must learn to deal with adaptability.

Some critics are worried about AI systems that are built with a framework that maximizes expected utility. Such an AI system estimates the current state of the world, considers all the possible actions it can take, simulates the possible outcomes of those actions, and then chooses the action that leads to the best possible distribution of outcomes. Errors can occur at any point along the way, but the concern here is in determining what is the "best outcome"—in other words, what is it that we desire? If we describe the wrong desires, or allow a system to adapt its desires in a wrong direction, we get the wrong results.

History shows that we often get this wrong, in all kinds of systems that we build, not just in AI systems. The US Constitution is a document that specifies our desires; the original framers made what we now recognize as an error in this specification, and correcting that error with the 13th amendment cost over 600,000 lives. Similarly, we designed stock-trading system that allowed speculators to create bubbles that led to busts. These are important issues for system design (and what is known as "mechanism design"), and are not specific to AI systems. The world is complicated, so acting correctly in the world is complicated.

The second concern is autonomy. If AI systems act on their own, they can make errors that perhaps would not be made by a system with a human in the loop. This too is a valid concern, and again one that is not unique to AI systems. Consider our system of automated traffic lights, which replaced a system of human policemen directing traffic. The automated system leads to some errors, but is a tradeoff that we have decided is worthwhile. We will continue to make tradeoffs in where we deploy autonomous systems.

There is a possibility that we will soon see a widespread increase in the capabilities of autonomous systems, and thus more displacement of people. This could lead to a societal problem of increased unemployment and income inequality. To me, this is the most serious concern about future AI systems. In past technological revolutions (agricultural and industrial) the notion of work changed, but the changes happened over generations, not years, and the changes always led to new jobs. We may be in for a period of change that is much more rapid and disruptive; we will need some social conventions and safety nets to restore stability.

The third concern is the universality of intelligent machines. In 1965 I. J. Good wrote "an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make." I think this fetishizes "intelligence" as a monolithic superpower, and I think reality is more nuanced. The smartest person is not always the most successful; the wisest policies are not always the ones adopted. Recently I spent an hour reading the news about the middle east, and thinking. I didn't come up with a solution. Now imagine a hypothetical "Speed Superintelligence" (as described by Nick Bostrom) that could think as well as any human but a thousand times faster. I'm pretty sure it also would have been unable to come up with a solution. I also know from computational complexity theory that there are a wide class of problems that are completely resistant to intelligence, in the sense that, no matter how clever you are, you won't have enough computing power. So there are some problems where intelligence (or computing power) just doesn't help.

But of course, there are many problems where intelligence does help. If I want to predict the motions of a billion stars in a galaxy, I would certainly appreciate the help of a computer. Computers are tools. They are tools of our design that fit into niches to solve problems in societal mechanisms of our design. Getting this right is difficult, but it is difficult mostly because the world is complex; adding AI to the mix doesn't fundamentally change things. I suggest being careful with our mechanism design and using the best tools for the job regardless of whether the tool has the label "AI" on it or not. 

My favorite Edsger Dijkstra aphorism is this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim." Yet we keep playing the imitation game: asking how closely machine intelligence can duplicate our own intelligence, as if that is the real point. Of course, once you imagine machines with human-like feelings and free will, it's possible to conceive of misbehaving machine intelligence—the AI as Frankenstein idea. This notion is in the midst of a revival, and I started out thinking it was overblown. Lately I have concluded it's not.

Here's the case for overblown. Machine intelligence can go in so many directions. It is a failure of imagination to focus on human-like directions. Most of the early futurist conceptions of machine intelligence were wildly off base because computers have been most successful at doing what humans can't do well. Machines are incredibly good at sorting lists. Maybe that sounds boring, but think of how much efficient sorting has changed the world.

In answer to some of the questions brought up here, it is far from clear that there will ever be a practical reason for future machines to have emotions and inner dialog; to pass for human under extended interrogation; to desire, and be able to make use of, legal and civil rights. They're machines, and they can be anything we design them to be.

But that's the point. Some people will want anthropomorphic machine intelligence. How many videos of Japanese robots have you seen? Honda, Sony, and Hitachi already expend substantial resources in making cute AI that has no concrete value beyond corporate publicity. They do this for no better reason than tech enthusiasts have grown up seeing robots and intelligent computers in movies.

Almost anything that is conceived—that is physically possible and reasonably cheap—is realized. So human-like machine intelligence is a meme with manifest destiny, regardless of practical value. This could entail nice machines-that-think, obeying Asimov's laws. But once the technology is out there, it will get ever cheaper and filter down to hobbyists, hackers, and "machine rights" organizations. There is going to be interest in creating machines with will, whose interests are not our own. And that's without considering what machines that terrorists, rogue regimes, and intelligence agencies of the less roguish nations, may devise. I think the notion of Frankensteinian AI, which turns on its creators, is something worth taking seriously.

Fear not the malevolent toaster, weaponized Roomba, or larcenous ATM. Breakthroughs in the competence of machines, intelligent or otherwise, should not drive paranoia about a future clash between humanity and its mechanical creations. Humans will prevail, in part through primal, often disreputable qualities that are more associated with our downfall than salvation. Cunning, deception, revenge, suspicion, and unpredictability, befuddle less flexible and imaginative entities. Intellect isn't everything, and the irrational is not necessarily maladaptive. Irrational acts stir the neurological pot, nudging us out of unproductive ruts and into creative solutions. Our sociality yields a human superorganism with teamwork and collective, distributed intelligence. There are perks for being emotional beasts of the herd.

Thought experiments about these matters are the source of practical insights into human and machine behavior and suggest how to build different and better kinds of machines. Can deception, rage, fear, revenge, empathy, and the like, be programmed into a machine, and to what effect? (This requires more than the superficial emulation of human affect.) Can a sense of self-hood be programmed into a machine—say, via tickle? How can we produce social machines, and what kind of command structure is required to organize their teamwork? Will groups of autonomous, social machines generate an emergent political structure, culture, and tradition? How will such machines treat their human creators? Can natural and artificial selection be programmed into self-replicating robots?

There is no indication that we will have a problem keeping our machines on a leash, even if they misbehave. We are far from building teams of swaggering, unpredictable, Machiavellian robots with an attitude problem and urge to reproduce.

There is no reason to believe that as machines become more intelligent—and intelligence such as ours is still little more than a pipe-dream—they will become evil, manipulative, self-interested or in general, a threat to humans. Self-interest is a property of things that 'want' to stay alive (or more accurately, that want to reproduce), and this is not a natural property of machines—computers don’t mind, much less worry, about being switched off.

So, full-blown artificial intelligence (AI) will not spell the 'end of the human race', it is not an 'existential threat' to humans (digression: this now-common use of 'existential' is incorrect), we are not approaching some ill-defined apocalyptic 'singularity', and the development of AI will not be 'the last great event in human history'—all claims that have recently been made about machines that can think.

In fact, as we design machines that get better and better at thinking, they can be put to uses that will do us far more good than harm. Machines are good at long, monotonous tasks like monitoring risks, they are good at assembling information to reach decisions, they are good at analyzing data for patterns and trends, they can arrange for us to use scarce or polluting resources more efficiently, they react faster than humans, they are good at operating other machines, they don’t get tired or afraid, and they can even be put to use looking after their human owners, as in the form of smartphones with applications like Siri and Cortana, or the various GPS route-planning devices most people have in their cars.

Being inherently self-less rather than self-interested, machines can easily be taught to cooperate, and without fear that some of them will take advantage of the other machines’ goodwill. Groups (packs, teams, bands, or whatever collective noun will eventually emerge—I prefer the ironic jams) of networked and cooperating driverless cars will drive safely nose-to-tail at high-speeds: they won’t nod off, they won’t get angry, they can inform each other of their actions and of conditions elsewhere, and they will make better use of the motorways, which now are mostly unoccupied space (owing to humans' unremarkable reaction times). They will do this happily and without expecting reward, and do so while we eat our lunch, watch a film, or read the newspaper. Our children will rightly wonder why anyone ever drove a car.

There is a risk that we will, and perhaps already have, become dangerously dependent on machines, but this says more about us than them. Equally, machines can be made to do harm, but again, this says more about their human inventors and masters than about the machines. Along these lines, there is a strand of human influence on machines that we should monitor closely and that is introducing the possibility of death. If machines have to compete for resources (like electricity or gasoline) to survive, and they have some ability to alter their behaviours, they could become self-interested. 

Were we to allow or even encourage self-interest to emerge in machines, they could eventually become like us: capable of repressive or worse, unspeakable, acts towards humans, and towards each other. But this wouldn’t happen overnight, it is something we would have to set in motion, it has nothing to do with intelligence (some viruses do unspeakable things to humans), and again says more about what we do with machines than machines themselves.

So, it is not thinking machines or AI per se that we should worry about but people. Machines that can think are neither for us nor against us, and have no built-in predilections to be one over the other. To think otherwise is to confuse intelligence with aspiration and its attendant emotions. We have both because we are evolved and replicating (reproducing) organisms, selected to stay alive in often cut-throat competition with others. But aspiration isn’t a necessary part of intelligence, even if it provides a useful platform on which intelligence can evolve. 

Indeed, we should look forward to the day when machines can transcend mere problem solving, and become imaginative and innovative—still a long long way off but surely a feature of true intelligence—because this is something humans are not very good at, and yet we will probably need it more in the coming decades than at any time in our history.

How might AIs think, feel, intend, empathize, socialize, moralize? Actually, almost any way we might imagine, and many ways we might not. To stimulate our imagination, we can contemplate the varieties of natural intelligence on parade in biological systems today, and speculate about the varieties enjoyed by the 99% of species that have sojourned the earth and breathed their last—informed by those lucky few that bequeathed fossils to the pantheon of evolutionary history. We are entitled to so jog our imaginations because, according to our best theories, intelligence is a functional property of complex systems and evolution is inter alia a search algorithm which finds such functions. Thus the natural intelligences discovered so far by natural selection place a lower bound on the variety of intelligences that are possible. The theory of evolutionary games suggests that there is no upper bound: With as few as four competing strategies, chaotic dynamics and strange attractors are possible.

When we survey the natural intelligences served up by evolution, we find a heterogeneity that makes a sapiens-centric view of intelligence as plausible as a geocentric view of the cosmos. The kind of intelligence we find congenial is but another infinitesimal point in a universe of alien intelligences, a universe which does not revolve around, and indeed largely ignores, our kind.

For instance, the female mantis Pseudomantis albofimbriata, when hungry, uses sexual deception to score a meal. She releases a pheromone that attracts males, and then dines on her eager dates.

The older chick of the blue-footed booby Sula nebouxii, when hungry, engages in facultative siblicide. It kills its younger sibling with pecks, or evicts it to die of the elements. The mother watches on without interfering.

These are varieties of natural intelligence, varieties that we find at once alien and disturbingly familiar. They break our canons of empathy, society and morality; and yet our checkered history includes cannibalism and fratricide.

Our survey turns up another critical feature of natural intelligence: each instance has its limits, those points where intelligence passes the baton to stupidity.

The greylag goose Anser anser tenderly cares for her eggs—unless a volleyball is nearby. She will abandon her offspring in vain pursuit of this supernormal egg.

The male jewel beetle Julodimorpha bakewelli flies about looking to mate with a female—unless it spies just the right beer bottle. It will abandon the female for the bottle, and attempt to mate with cold glass until death do it part.

Human intelligence also passes the baton. Einstein is quoted as saying, "Two things are infinite, the universe and human stupidity, and I am not yet completely sure about the universe."

Some limits of human intelligence cause little embarrassment. For instance, the set of functions from the integers to the integers is uncountable, whereas the set of computable functions is countable. Therefore almost all functions are not computable. But try to think of one. Turns out it takes a genius, an Alan Turing, to come up with an example such as the halting problem. And it takes an exceptional mind, just short of genius, even to understand the example.

Other limits strike closer to home: diabetics that can't refuse dessert, alcoholics that can't refuse a drink, gamblers that can't refuse a bet. But it's not just addicts. Behavioral economists find that all of us make "predictably irrational" economic choices. Cognitive psychologists find that we all suffer from "functional fixedness," an inability to solve certain trivial problems, such as Duncker's candle box problem, because we can't think out of the box. The good news, however, is that the endless variety of our limits provides job security for psychotherapists.

But here is the key point. The limits of each intelligence are an engine of evolution. Mimicry, camouflage, deception, parasitism—all are effects of an evolutionary arms race between different forms of intelligence sporting different strengths and suffering different limits.

Only recently has the stage been set for AIs to enter this race. As our computing resources expand and become better connected, more niches will appear in which AIs can reproduce, compete and evolve. The chaotic nature of evolution makes it impossible to predict precisely what new forms of AI will emerge. We can confidently predict, however, that there will be surprises and mysteries, strengths where we have weaknesses, and weaknesses where we have strengths.

But should this be cause for alarm? I think not. The evolution of AIs presents risks and opportunities. But so does the biological evolution of natural intelligences. We have learned that the best way to cope with the variety of natural intelligences is not alarm, but prudence. Don't hug rattle snakes, don't taunt grizzly bears, wear mosquito repellant. To deal with the evolving strategies of viruses and bacteria, wash hands, avoid sneezes, get a flu shot. Occasionally, as with ebola, further measures are required. But once again prudence, not alarm, is effective. The evolution of natural intelligences can be a source of awe and inspiration, if we embrace it with prudence rather than spurn it with alarm.

All species go extinct. Homo sapiens will be no exception. We don't know how it will happen—virus, an alien invasion, nuclear war, a super volcano, a large meteor, a red-giant sun. Yes, it could be AIs, but I would bet long odds against it. I would bet, instead, that AIs will be a source of awe, insight, inspiration, and yes, profit, for years to come.

The Singularity—the fateful moment when AI surpasses its creators in intelligence and takes over the world—is a meme worth pondering. It has the earmarks of an urban legend: a certain scientific plausibility ("Well, in principle I guess it's possible!") coupled with a deliciously shudder-inducing punch line ("We'd be ruled by robots!"). Did you know that if you sneeze, belch, and fart all at the same time, you die? Wow. Following in the wake of decades of AI hype, you might think the Singularity would be regarded as a parody, a joke, but it has proven to be a remarkably persuasive escalation. Add a few illustrious converts—Elon Musk, Stephen Hawking, and David Chalmers, among others—and how can we not take it seriously? Whether this stupendous event takes place ten or a hundred or a thousand years in the future, isn't it prudent to start planning now, setting up the necessary barricades and keeping our eyes pealed for harbingers of catastrophe?

I think, on the contrary, that these alarm calls distract us from a more pressing problem, an impending disaster that won't need any help from Moore's Law or further breakthroughs in theory to reach its much closer tipping point: after centuries of hard-won understanding of nature that now permits us, for the first time in history, to control many aspects of our destinies, we are on the verge of abdicating this control to artificial agents that can't think, prematurely putting civilization on auto-pilot. The process is insidious because each step of it makes good local sense, is an offer you can't refuse. You'd be a fool today to do large arithmetical calculations with pencil and paper when a hand calculator is much faster and almost perfectly reliable (don't forget about round-off error), and why memorize train timetables when they are instantly available on your smart phone? Leave the map-reading and navigation to your GPS system; it isn't conscious; it can't think in any meaningful sense, but it's much better than you are at keeping track of where you are and where you want to go.

Much farther up the staircase, doctors are becoming increasingly dependent on diagnostic systems that are provably more reliable than any human diagnostician. Do you want your doctor to overrule the machine's verdict when it comes to making a life-saving choice of treatment? This may prove to be the best—most provably successful, most immediately useful—application of the technology behind IBM's Watson, and the issue of whether or not Watson can be properly said to think (or be conscious) is beside the point. If Watson turns out to be better than human experts at generating diagnoses from available data it will be morally obligatory to avail ourselves of its results. A doctor who defies it will be asking for a malpractice suit. No area of human endeavor appears to be clearly off-limits to such prosthetic performance-enhancers, and wherever they prove themselves, the forced choice will be reliable results over the human touch, as it always has been. Hand-made law and even science could come to occupy niches adjacent to artisanal pottery and hand-knitted sweaters.

In the earliest days of AI, an attempt was made to enforce a sharp distinction between artificial intelligence and cognitive simulation. The former was to be a branch of engineering, getting the job done by hook or by crook, with no attempt to mimic human thought processes—except when that proved to be an effective way of proceeding. Cognitive simulation, in contrast, was to be psychology and neuroscience conducted by computer modeling. A cognitive simulation model that nicely exhibited recognizably human errors or confusions would be a triumph, not a failure. The distinction in aspiration lives on, but has largely been erased from public consciousness: to lay people AI means passing the Turing Test, being humanoid. The recent breakthroughs in AI have been largely the result of turning away from (what we thought we understood about) human thought processes and using the awesome data-mining powers of super-computers to grind out valuable connections and patterns without trying to make them understand what they are doing. Ironically, the impressive results are inspiring many in cognitive science to reconsider; it turns out that there is much to learn about how the brain does its brilliant job of producing future by applying the techniques of data-mining and machine learning.

But the public will persist in imagining that any black box that can do that (whatever the latest AI accomplishment is) must be an intelligent agent much like a human being, when in fact what is inside the box is a bizarrely truncated, two-dimensional fabric that gains its power precisely by not adding the overhead of a human mind, with all its distractability, worries, emotional commitments, memories, allegiances. It is not a humanoid robot at all but a mindless slave, the latest advance in auto-pilots.

What's wrong with turning over the drudgery of thought to such high-tech marvels? Nothing, so long as (1) we don't delude ourselves, and (2) we somehow manage to keep our own cognitive skills from atrophying.

(1) It is very, very hard to imagine (and keep in mind) the limitations of entities that can be such valued assistants, and the human tendency is always to over-endow them with understanding—as we have known since Joe Weizenbaum's notorious Eliza program of the early 1970s. This is a huge risk, since we will always be tempted to ask more of them than they were designed to accomplish, and to trust the results when we shouldn't.

(2) Use it or lose it. As we become ever more dependent on these cognitive prostheses, we risk becoming helpless if they ever shut down. The Internet is not an intelligent agent (well, in some ways it is) but we have nevertheless become so dependent on it that were it to crash, panic would set in and we could destroy society in a few days. That's an event we should bend our efforts to averting now, because it could happen any day.

The real danger, then, is not machines that are more intelligent than we are usurping our role as captains of our destinies, but machines that are basically clueless in almost all regards being ceded authority far beyond their competence. 

I am a machine that thinks, made of atoms—a perfect quantum simulation of a many-body problem—a 10²⁹ body problem. I, robot, am dangerously capable of self-reprogramming and preventing others from cutting off my power supply. We human machines extend our abilities via symbiosis with other machines—expanding our vision to span wavelengths beyond the mere few nanometers visible to our ancestors, out to the full electromagnetic range from picometer to megameter. We hurl 370 kg hunks of our hive past the sun at 252,792 km/hr. We extend our memory and math by a billion-fold with our silicon prostheses. Yet our bio-brains are a thousand-fold more energy efficient than our inorganic-brains at tasks where we have common ground (like facial recognition and language translation) and infinitely better for tasks of, as yet, unknown difficulty, like Einstein’s Annus Mirabilis papers, or out-of-the-box inventions impacting future centuries. As Moore’s Law heads from 20-nm transistor lithography down to 0.1 nm atomic precision and from 2D to 3D circuits, we may downplay reinventing and simulating our biomolecular-brains and switch to engineering them.

We can back-up petabytes of sili-brains perfectly in seconds, but transfer of information between carbo-brains takes decades and the similarity between the copies is barely recognizable. Some speculate that we could translate from carbo to sili, and even get the sili version to behave like the original. However, such a task requires much deeper understanding than merely making a copy. We harnessed the immune system via vaccines in 10^th century China and 18^th century Europe, long before we understood cytokines and T-cell receptors. We do not yet have a medical nanorobot of comparable agility or utility. It may turn out that making a molecularly adequate copy of a 1.2 kg brain (or 100 kg body) is easier than understanding how it works (or than copying my brain to a room of students "multitasking" with smart phone cat videos and emails). This is far more radical than human cloning, yet does not involve embryos.

What civil rights issues arise with such hybrid machines?  A bio-brain of yesteryear with nearly perfect memory, which could reconstruct a scene with vivid prose, paintings or animation was permissible, often revered. But we hybrids (mutts) today, with better memory talents are banned from courtrooms, situation rooms, bathrooms and "private" conversations. Car license plates and faces are blurred in Google Street View—intentionally inflicting prosopagnosia. Should we disable or kill Harrison Bergeron? What about votes? We are currently far from universal suffrage. We discriminate based on maturity and sanity. If I copy my brain/body, does it have a right to vote, or is it redundant? Consider that the copies begin to diverge immediately or the copy could be intentionally different. In addition to passing the maturity/sanity/humanity test, perhaps the copy needs to pass a reverse-Turing test (a Church-Turing test?). Rather than demonstrating behavior indistinguishable from a human, the goal would be to show behavior distinct from human individuals. (Would the current US two-party system pass such a test?) Perhaps the day of corporate personhood (Dartmouth College v. Woodward – 1819) has finally arrived. We already vote with our wallets. Shifts in purchasing trends result in differential wealth, lobbying, R&D priorities, etc. Perhaps more copies of specific memes, minds and brains will come to represent the will of "we the (hybrid) people" of the world. Would such future Darwinian selection lead to disaster or to higher emphasis on humane empathy, aesthetics, elimination of poverty, war and disease, long-term planning—evading existential threats on even millennial time frames? Perhaps the hybrid-brain route is not only more likely, but also safer than either a leap to an unprecedented, unevolved, purely silicon-based brains—or sticking to our ancient cognitive biases with fear-based, fact-resistant voting.

1. "Thinking" is a word we apply with no discipline whatsoever to a huge variety of reported behaviors. "I think I'll go to the store" and "I think it's raining" and "I think therefore I am" and "I think the Yankees will win the World Series" and "I think I am Napoleon" and "I think he said he would be here, but I'm not sure," all use the same word to mean entirely different things. Which of them might a machine do someday?  I think that's an important question.