【Economist】Artificial intelligence: Shall we play a game?

Why AI researchers are so keen on video games 

LAST year Artur Filipowicz, a computer scientist at Princeton University, had a stop-sign problem. Dr Filipowicz is teaching cars how to see and interpret the world, with a view to them being able to drive themselves around unaided. One quality they will need is an ability to recognise stop signs. To that end, he was trying to train an appropriate algorithm. Such training meant showing this algorithm (or, rather, the computer running it) lots of pictures of lots of stop signs in lots of different circumstances: old signs and new signs; clean signs and dirty signs; signs partly obscured by lorries or buildings; signs in sunny places, in rainy places and in foggy ones; signs in the day, at dusk and at night.

Obtaining all these images from photo libraries would have been hard. Going out into the world and shooting them in person would have been tedious. Instead, Dr Filipowicz turned to “Grand Theft Auto V”, the most recent release of a well-known series of video games. “Grand Theft Auto V” is controversial because of its realistic portrayal of crime and violence—but from Dr Filipowicz’s point of view it was ideal, because it also features realistic stop signs. By tinkering with the game’s software, he persuaded it to spit out thousands of pictures of these signs, in all sorts of situations, for his algorithm to digest.

Dr Filipowicz’s stop signs are one instance of the fondness that students of artificial intelligence (AI, of which machine vision is an example) have for video games. There are several reasons for this popularity. Some people, such as Dr Filipowicz, use games as training grounds for the real world. Others, observing that different games require different cognitive skills, think games can help them understand how the problem of intelligence may be broken down into smaller, more manageable chunks. Others still, building on these two observations, think games can help them develop a proper theory of artificial (and perhaps even natural) intelligence.

Learner driver

For all of this to happen, though, the games themselves have first to be tweaked so that they can be played directly by another computer program, rather than by a human being watching the action on a screen. “Grand Theft Auto V”, for instance, can be turned from a source of pictures of road signs into a driving simulator for autonomous vehicles by bolting onto it a piece of software called “Deep Drive”. This lets the driving and navigation programs of such vehicles take control—a cheaper and safer way of testing driving software than letting it loose on roads.

Games companies are beginning to understand this. In June 2015, for instance, Microsoft started Project Malmo, an AI-development platform based on a popular “world-building” game called “Minecraft” that it had recently purchased. In November 2016 Activision Blizzard, owners of “Starcraft II”, a science-fiction strategy game in which players build and command human and alien armies, announced something similar in collaboration with DeepMind, an AI firm owned by Alphabet, Google’s holding company.

The following month, with the permission of the owners involved, a privately financed research group in San Francisco, called OpenAI, released “Universe”. This is a piece of software, free for all to use, which features hundreds of games presented in ways that mean they can be played directly by appropriate programs. The offerings in “Universe” range from bestselling, big-budget titles such as “Portal 2” (a physics-based puzzle game) to cheap-and-cheerful web games like “Bubble Hit Pony Parade” and “James the Space Zebra”.

One of Microsoft’s hopes in starting Project Malmo was to teach AI software to collaborate with people. To this end, Katja Hofman, the project’s head, is trying to use “Minecraft” to create an advanced personal assistant. Her goal is software that can anticipate what its human operator wants, and help him achieve it. “Minecraft”, which is simpler than the real world but still complicated enough to be interesting, makes the perfect testing-ground. Dr Hofman and her colleagues are, for instance, using it to try to teach a computer to work out that it must co-operate with a human player in order to catch a virtual pig. Since the machine is incapable of understanding written instructions, it must learn co-operation purely by watching the actions of its human confrères in the game.

Acting as training wheels for the real world is not, however, the only thing video games can do for AI. The fact that different games require different talents helps researchers chop up the problem of intelligence. In 2015 DeepMind released a paper describing how its researchers had trained an artificial neural network—a program based loosely on the structure of a biological brain—to play dozens of different games released in the 1970s and 1980s by Atari, a pioneering video-games company.

Some games proved harder than others for the network to master. “Breakout”, which is a bit like a single-player version of tennis, was easy. The objective is to smash floating blocks by hitting them with a bouncing ball. A player can do one of two things: move the “racket” left or move it right. Failure is punished instantly (missing the ball costs a life). Similarly, success is instantly rewarded (each smashed block adds to the score). This combination of simplicity and immediate feedback suited DeepMind’s neural network, which learnt to play “Breakout” so well that it reached scores more than ten times those a professional human games-tester can manage.

Other games were less straightforward. In “Montezuma’s Revenge” the goal is to retrieve treasure buried deep inside a danger-filled pyramid. To do this players must first achieve lots of sub-goals, such as finding keys to open doors. Feedback is less immediate than in “Breakout”—for instance, a key that turns up in one area might open a door in another, far away. And the ultimate reward, reaching the treasure, is the consequence of thousands of previous actions. This meant that the network found it hard to connect cause and effect. In contrast to its virtuoso performance at “Breakout”, it was able to make almost no headway at all with “Montezuma’s Revenge”.

Since then, DeepMind’s researchers have tweaked their algorithms to make the system more curious about things, by giving it bigger rewards for exploration and experimentation. This makes it more likely to stumble across good strategies which have payouts that are not immediately apparent. That approach is not limit to mastering skills in a virtual world—it can be applied to the real one, as well. DeepMind’s algorithms have, for instance, been put to use in Google’s data centres, where they have developed ways to cut energy use by 40%. Indeed, it is possible to view tasks like that as games in themselves. To cut energy use in a data centre, a network can tweak things like coolant-pump settings and load distributions while keeping an eye on energy use. The lower it can get the “score”, the better it is doing.

Embodiments of truth

At the moment, repurposing a games-playing program to run a data centre’s energy budget really is like teaching it a new game from scratch. That is because DeepMind’s original neural network could learn to play only one game at a time. In order to understand “Breakout”, for example, it would have to forget everything it knew about “Space Invaders”. Such amnesia is in the nature of artificial neural networks—and is something that distinguishes them from real brains. They learn by system-wide adjustments of the strengths of the connections between the virtual neurons of which they are composed. Change the task to be learned, and the old web of connections will gradually be overwritten. Now, however, as they describe in a paper published in March, DeepMind’s programmers have worked out how to overcome this and let a network master many games at once, in the way that a real brain can. That is a step towards transfer learning—the ability to put to use in one context patterns of behaviour learned in another—which is a hot topic in AI research.

Like displaying curiosity and delaying rewards, transferring learning from one task to another is something humans do effortlessly but machines struggle to manage. Here again, games are playing an important role in research. For example, Julian Togelius of New York University has organised a challenge called the General Video Game AI Competition. Entrants must create a single program that can play, with reasonable competence, ten different video games that neither it nor its programmers have come across. This requires the software to master many skills—planning, exploration, decision-making and so on—and apply them to problems it has not previously encountered.

Even when transfer learning is mastered, though, constructing useful artificial intelligence will remain a piecemeal activity. What researchers would really like is an underlying theory of how to do so systematically. One candidate to be such a theory, called embodied cognition, argues that, instead of trying to design intelligence into a program from the beginning, it needs to be learned entirely from experience.

Dr Hofman, in particular, backs this approach. She reckons video games are perfect platforms on which to explore the idea. Previous attempts to study embodied cognition, carried out in the 1980s, involved fitting robots with sensors and letting them learn, by running around and bumping into things, how the real world works. Researchers back then did have some success with this approach, but they ran into problems scaling their experiments up. As David Silver, who works at DeepMind, observes: “Robots have gears and wheels and motors, and all sorts of fiddly things like that. You end up spending a lot of time doing maintenance work.”

Play up, play up and play the game

Video games can streamline this process. A virtual robot in a virtual world is weightless. It has no moving parts, so needs no maintenance. Adjusting it to change its specifications does not require breaking out the spanners and taking it to bits. A few strokes on a keyboard will suffice.

Its environment can be altered easily, too. Rerouting a maze no longer means welding sheets of metal together or gluing plastic walls. And a computer can run thousands of such simulations at a time, allowing legions of virtual robots to try tasks again and again, learning with each attempt. That kind of large-scale testing, which permits the learning process itself to be monitored and understood, is simply not practical using real machines.

The important thing, according to Demis Hassabis, DeepMind’s founder, is to make sure the virtual robot cannot cheat. It must navigate using only the information its virtual sensors can gather. There can be no peeking behind the scenes of the simulation. If such a robot wants to learn its way around the danger-filled pyramid in “Montezuma’s Revenge”, or the fictional city of Los Santos in “Grand Theft Auto”, it must work out where it is and what is happening from what it can “see”, not by asking the computer which is running the game to give it co-ordinates. This is the approach DeepMind takes when it teaches programs to play video games.

Studying embodied cognition in this way is a logical conclusion of the games-playing approach to AI. It seems an appropriate one. Watch the young of any intelligent creature, from dogs to humans, and you will see them building up something that looks suspiciously like embodied cognition by playing. Evolution did not have the assistance of computers when it arrived at this process. But the fundamental point of such activity, in both the artificial and the natural worlds, is to prepare players for the biggest game of all—reality.

——

May 11th 2017 | Science and technology | 1948 words