I was born behind the Iron Curtain, in Soviet-occupied Estonia, and looked forward to a pretty bleak life in some scientific institute trying to figure out how to kill more Americans. Luckily though, the Soviet Union collapsed shortly before I was ready for independent life. The year 1990, when I went to university, was also in the middle of big turmoil, where the Soviet Union collapsed and various countries, including Estonia, became independent.
When I went to university, I studied physics there. The reason I studied physics was that I was into computer programming already, since high school or even a little bit earlier, so I thought I should expand my horizons a little bit. And I do think it has helped me quite a lot. If you look around in the so‑called existential risk ecosystem that I support, there's, I would say, an over-representation of physicists, because physics helps you to see the world in a neutral manner. You have a curiosity that helps you to create the world model rather than you try to model the world in a way that suits your predispositions.
After having studied physics, I worked with computers throughout my entire period of university. We jokingly called ourselves the "computer games industry of Estonia," because we were really the only commercial computer games development studio in Estonia. After spending a decade developing computer games, I gave one talk where I described my life as surfing the Moore's Law. Interesting turning points in my life have coincided with things that Moore's Law has made possible or has made no longer feasible.
For example, we exited the computer games industry when graphics cards came along, thus enabling much more powerful storytelling capabilities and therefore reducing the importance that programming played in computer games. Because we, being mostly good at programming, didn't have a good comparative advantage in this new world, we ended up exiting the computer games business and going into Internet programming. At this point we met Niklas Zennström and Janus Friis, who eventually became the main founders of Skype. Together with them, we first did the Kazaa file sharing application, which got us into a bunch of legal trouble. After Kazaa, we did a few smaller projects, and eventually ended up doing Skype.
The way we got into the games industry was almost by accident. The nice thing about starting your computer career with computers that are really slow is that you have to do work in order to make them do something interesting. These days I see that my children, for example, have a tough time starting programming because YouTube is just one click away. Just trying to figure out how to make interesting games was a natural step of evolution for a programmer.
It was in 1989 when I teamed up with a couple of my classmates to develop a simple graphical action-based computer game, and we managed to sell that game to Sweden. We earned a little hard currency—Swedish kronor—as a result, which, at the time of the collapse of the Soviet Union when the Russian ruble was in freefall, was a fortune. I think we made $5,000 based on that, which was an incredibly big sum back then. Hence, we were hooked and thought, okay, we can do something that other people are willing to pay money for, and ended up developing bigger and bigger games and spent about one decade in games.
People do ask me quite a lot, what is the secret of Estonia when it comes to advancing digital ideas, and programming and technology in general. It's hard to track this down to one cause, but there are a few contributing factors. One thing was that even during Soviet times there was this big scientific center called the Institute of Cybernetics, in Tallinn, that hosts many scientists who developed things like expert systems and early precursors of AI.
I'm proud to say that Skype has played quite a big role in the Estonian startup ecosphere, for various reasons. One is that Estonia is a small place, so people know each other. I half-jokingly say that quite a lot of people just knew "the Skype boys," as we were called in Estonia, and they think, if they can do it, well, so can I. The other nice side effect of Skype is that it's a fairly big company in Estonian context, so it works as a training ground. A lot of people meet there and get their experience working there, working in an international context. Skype is no longer a startup, but we used to have a strong startup culture there. Even now, I have invested in three or four companies that are just made by Skype alumni, so there's a strong startup culture there.
Finally, the Estonian government has gotten into a nice positive feedback loop, where they have done a few digital innovations in the domain of e-governance. They had gotten very good positive feedback based on their achievements in things like digital voting and paperless government office. Whenever humans get into a positive feedback loop, they want to do more of the things that they get praise for. Indeed, the latest project was called Estonian digital E-residency, so you can go to an Estonian consulate—as far as I understand—and get a chip card that will give you the ability to give digital signatures that have the power of law in Estonia, and hence in the EU.
Skype started as a project within another company. The other company was called Joltid, founded by Niklas Zennström and Janus Friis. Within that company, we first did various projects, including the back end for Kazaa file sharing network. Skype was started in late 2002 as a project within that company, but just a few months later it was spun off into a separate company, and seven people got founding shares in this new company called Skyper Limited. The name Skyper came from "sky peer", because the original idea wasn't actually to do a Voice over IP client. The original idea was to do Wi-Fi sharing, but Skyper.net ... or was it Skyper.com? ... was taken, so we ended up chopping the 'R' off from the end.
Skype was not the first VoIP client. In fact, when we started with this idea of developing a Wi-Fi sharing network—Skyper—our thinking was that, clearly there is VoIP software out there that we should import and implement on top of our Wi-Fi sharing network to give people incentive to join our network. However, after having evaluated the existing offerings that were out there, we determined that none of them worked properly behind firewalls. The state of Voice over IP back then was that the latest thing, which was called SIP—Session Initiation Protocol—was a new standard that was roughly modeled after email. The problem was that, just like with email, you need an ISP or some third party to set you up with this. You can't start just an email program and be immediately connected; you need to connect that email program to some server, which creates the chicken and egg situation.
In the VoIP world, where there were no Web servers at that point, we figured out that we needed a peer-to-peer solution where people could bootstrap the network without being reliant on their ISPs and installing some gateways or things. After having empirically determined that the existing Voice over IP, although sufficient for our purposes technically, wouldn't work because of the architectural requirements—the chicken and egg situation—we decided, okay, let's do our own Voice over IP program. Eventually we ended up dropping this Wi-Fi sharing network idea altogether and just focused on the Voice over IP.
Skype has been sold three or four times, depending on how you count. Myself and the founders sold our shares during the first sale, which was to eBay in September 2005—two years after we launched Skype. After that, eBay sold the majority of the shares to a private equity company and a consortium of VCs. It was 2010 or 2011 when Microsoft bought the whole thing.
From Skype, I eased out gradually. There was no sharp point where I left Skype. One moment where I significantly reduced my involvement in Skype was 2009, when there was this big lawsuit between the founders of Skype and the private equity companies that bought shares of Skype from eBay. There was some technology licensing issue. Because I ended up on the other side of that lawsuit than Skype, my day-to-day activities in Skype were hindered. When I came back to Skype half a year later, the company had moved along quite a lot, so it was hard for me to fit right back in. I ended up just gradually easing out from the day-to-day activities.
Already, during the lawsuit that Skype had, I was looking out for what other important things there might be to do in the world, and I ended up reading the writings of Eliezer Yudkowsky, an AI researcher in California. I found him very interesting, but also he was making a very important argument that the default outcome from AI is not necessarily good. We have to put in effort in order to make the outcomes from AI, once they get powerful enough, good. Once I got interested in these topics, I was more and more willing to contribute my time and money to advancing what's called the existential risk ecosystem—people thinking about not just the risks from technology, like AI technology, but also from other technologies such as synthetic biology and nanotechnology.
When it comes to existential risks, there are two big categories. One category is natural existential risks such as super volcanoes, for example, or asteroid impacts. Every 10- to 100 million years, a big enough asteroid comes along that just potentially destroys the planet. Now, the nice thing about natural existential risk is that these are risks that we have lived with for our entire history, so they're not necessarily getting bigger over time.
The other category is technological existential risks—risks from either deliberate or accidental misapplication of technology of ever-increasing power. As we know, there's an exponential increase in the power of computers and other technology. We are entering uncharted territory in this century, and therefore, foresight is important to have. We need to figure out how to steer the technological progress to ensure safe outcomes.
I started engaging with existential risk ecosystem in 2009 already. I remember meeting up with Yudkowsky, then starting to engage with other people in the existential risk community and seeing how I could help. First, I started by donating money, but eventually ended up supporting more and more organizations and doing a "cross-pollination" between those organizations by introducing people and making sure that their activities are more coordinated than they otherwise would be. Also, I finally ended up co-founding two new organizations. At Cambridge University, we have an organization called the Centre for the Study of Existential Risk, co-founded with Huw Price, who is a professor of philosophy at Cambridge University, and Martin Rees, who back then used to be the Master of Trinity College, and is very well-known scientist, who has written a book about existential risks himself. The other organization that I helped to co-found is at MIT, here in the US. It's called the Future of Life Institute, and it's led by Max Tegmark, who is a well-known physicist at MIT.
There's an interesting point about what is the role of computer science. Obviously I'm biased because I'm a computer person, but I have found that there is a very fertile intersection of computer science and philosophy. The reason is that throughout history philosophy has leaned on human intuitions. Analytic philosophy tries to make concepts precise, but when doing so they come up with examples and counterexamples to delineate the concepts that lean on human intuitions.
We know from psychological research that human intuitions aren't fundamental entities in the world. If you do different experiments in different cultures, for example, people have completely different intuitions. They even see different visual illusions. Daniel Dennett has said that computers keep philosophy honest. When you make a philosophical argument and you don't lean on intuitions, you lean on programs, you basically point to a program and say, "This is what I mean," you're on much, much more solid ground because you're no longer influenced on what intuition tells humans and how they differ from culture to culture.
Human philosophy has had thousands of years to come up with interesting passages of thought and explore the thought space, but now we need answers. And these answers have to be there in a decade or two. These answers have to be in the form of computer code.
Elon Musk said at his interview at the TED conference a couple of years ago, that there are two kinds of thinking. All of humanity, most of the time, engages in what you call metaphorical thinking, or analog-based thinking. They bring in metaphors from different domains and then apply them to a domain that they want to analyze, which is like things that they do intuitively. It's quick, cheap, but it's imprecise. The other kind of thinking is that you reason from first principles. It's slow, painful, and most people don't do it, but reasoning from first principles is really the only way we can deal with unforeseen things in a sufficiently rigorous manner. For example, sending a man to the moon, or creating a rocket. If it hasn't been done before, we can't just use our knowledge. We can't just think about "how would I behave if I were a rocket" and then go from there. You have to do the calculations. The thing with existential risks is it's the same. It's hard to reason about them, these things that have never happened. But they're incredibly important, and you have to engage in this slow and laborious process of listening to the arguments and not pattern-matching them to things that you think might be relevant.
The reasons why I'm engaged in trying to lower the existential risks has to do with the fact that I'm a convinced consequentialist. We have to take responsibility for modeling the consequences of our actions, and then pick the actions that yield the best outcomes. Moreover, when you start thinking about—in the pallet of actions that you have—what are the things that you should pay special attention to, one argument that can be made is that you should pay attention to areas where you expect your marginal impact to be the highest. There are clearly very important issues about inequality in the world, or global warming, but I couldn't make a significant difference in these areas.
When I found that there is this massively underappreciated topic of existential risks, I saw immediately that I could make a significant difference there, first by bringing my reputation to these arguments and more credibility to those arguments. I basically started with taking those arguments, internalizing them, then repackaging them in my own words, and then using my street credibility to give talks and discussions and meet with people to talk about these issues. As a result of that activity, now we're in much better position in the world, where we do have very strong organizations, reputationally, at Cambridge University and MIT, and organizations that are associated with advancing those topics.
Over the last six years or so there has been an interesting evolution of the existential risk arguments and perception of those arguments. While it is true, especially in the beginning, that these kinds of arguments tend to attract cranks, there is an important scientific argument there, which is basically saying that technology is getting more and more powerful. Technology is neutral. The only reason why we see technology being good is that there is a feedback mechanism between technology and the market. If you develop technology that's aligned with human values, the market rewards you. However, once technology gets more and more powerful, or if it's developed outside of market context, for example in the military, then you cannot automatically rely on this market mechanism to steer the course of technology. You have to think ahead. This is a general argument that can apply to both synthetic biology, artificial intelligence, nanotechnology, and so on.
One good example is the report LA-602, that was developed by the Manhattan Project. During the Manhattan project, it was six months before the first nuclear test. They did a scientific analysis of what is the probability, what are the chances of creating a runaway process in the atmosphere that would burn up the atmosphere and thus destroy the earth? It’s the first solid example of existential risk research that humanity has done.
Really, what I am trying to advance is more reports like that. Nuclear technology is not the last potentially disastrous technology that humans are going to invent. In my view, it's very, very dangerous when people say, "Oh, these people are cranks." You’re basically lumping together those Manhattan Project scientists who developed solid scientific analysis that's clearly beneficial for humanity, and some people who are just clearly crazy and are predicting the end of the world for no reason at all.
It’s too early to tell right now what kind of societal structures we need to contain the technology once the market mechanism is no longer powerful enough to contain them. At this stage, we need more research. There's a research agenda coming out pretty soon that represents a consensus between the AI safety community and the AI research community, of things that are not necessarily commercially motivated research, but the research that needs to be done if you want to steer the course, if you want to make sure that the technology is beneficial in the sense that it's aligned with human values, and thus giving us a better future the way we think the future should be. The AI should also be robust in the sense that it wouldn't accidentally create situations where, even though we developed it with the best intentions, it would still veer off the course and give us a disaster.
There are several technological existential risks. An example was the nuclear weapons before the first nuclear test was done. It wasn't clear whether this was something safe to do on this planet or not. Similarly, as we get more and more powerful technology, we want to think about the potentially catastrophic side effects. It's fairly easy for everyone to imagine that once we get synthetic biology, it becomes much easier to construct organisms or viruses that might be much more robust against human defenses.
I was just talking about technological existential risks in general. One of those technological existential risks could be potentially, artificial intelligence. When I talk about AI risks to people I sometimes ask them two questions. First, do you have children? Second, can you program computers? To people who have children, I can make the point that their children are part of humanity, hence, they can't treat humanity as an abstract object and say things like, "Perhaps humanity doesn't deserve to survive," because their children are part of it, and they're saying that their children don't deserve to survive, which is hardly what they mean.
But the reason why I ask them whether they can program computers is that, can I talk to them about AI in the language of what it really is—it's a computer program. People who are not computer programmers, I can't tell them in the exact language. I have to use metaphors, which are necessarily imprecise. People who don't program don't know what computer programs, and hence AI, really are.
One of the easiest arguments to make is, look around. What you see in the world is, you see a world of human designs and human dominance. The reason why humans dominate this planet has nothing to do with our speed or our manipulators. It has to do with intelligence, however we define it. The thing about AI is, if you're creating machines that are more and more intelligent, you don't want to inadvertently end up in a situation that gorillas are these days, for example, that you have a smarter agent than you dominating the environment.
As Stuart Russell points out in his commentary on edge.org, the worry with AIs isn't necessarily that they would be malevolent or angry at humans. The worry that we need to think through and do research about is that they will get more and more competent. If we have a system that's very competent in steering the world toward something that we don't exactly want, how do we prevent the world ending up in a place that we don't exactly want? We need to solve two challenges. One is to ensure that AI is beneficial, in the sense that in using it, increasing its competence would contribute to the best outcomes as we humans see it. Second, we have to ensure that AI is robust, meaning that once it starts developing its own technologies, once it starts developing further next generations of AIs, it wouldn't drift from the course that we want it to stick to.
When I say we, a lot of times I really mean humanity. It's not chimpanzees who are developing these technologies. It's humans who are developing the technology. If I want to zoom in and narrow it down, then I would say technology developers and people who are funding technologies, people who are regulating technologies. More generally, everyone who is on a causal path of new technologies being developed, is in some way responsible for making sure that the new technologies that are brought into existence as a result of their efforts, they are responsible for ensuring that they are beneficial in the long term for humanity.
I would say that I don't have any favorites, or any particular techniques within the domain of AI that I'm particularly worried about. First of all, I'm much more calm about these things. Perhaps by virtue of just having longer exposure to AI companies and people who develop AI. I know that they are well-meaning and people with good integrity.
Personally, I think the biggest research that we need to advance is how to analyze the consequences of bringing about very competent decision-making systems to always ensure that we have some degree of control over them, and we won't just end up in a situation where this thing is loose and there's nothing we can do now.
There is some research that can be done and has been proposed. The technical term for this is corrigibility. Most of the technology these days is developed in an iterative manner: you create the first version of technology, you see what's wrong with it, you create the next version, and the next version, next version. Each next version tends to be better, in some dimension at least. But the thing is that once you create autonomous systems, and once those autonomous systems get powerful enough to model the activities of their creators, to put it simply, once they figure out that there's an off switch, they have instrumental reasons to disable that off switch. We need to think through how we construct ever more competent systems to ensure that the outcomes are beneficial.
When it comes to control of the future, it eventually ends up in philosophy and moral philosophy, and thinking about topics like how should conflicting interests be reconciled when there are 7 billion, perhaps 10 billion people on this planet. And how should we take into account the interests of animals, for example, and the ecosystem in general. Humanity does not know the answer to the question: what do we really want from the long-term future? And again, in a situation where we might hand off the control to machines, it's something that we need to get right.
