Recently, Niantic, the maker of Pokémon Go, invited me to its London offices to see its new mobile pet sim Peridot. I saw a presentation, tried the game — which will be available on iOS and Android on May 9 — and got to hang out with some of the developers, including the computer scientist leading the small research team developing Peridot’s potentially groundbreaking new augmented reality technology with machine learning.
My experience with Peridot a few days later, when I was given a test version to try at home, revealed the potential of the game better than any demo I’d seen thus far. When my two young children (ages 4 and 6) saw Niantic’s invention, called a Peridot, hatch on my phone screen and run around my house, they were instantly charmed. Petted it, fed it, threw a ball for it that bounced off the walls, and watched it run around and behind the furniture, only slightly erratically.
When I checked the baby Dot’s desires, I noticed that it wanted to eat a dandelion, which is found in grassy areas. There’s grass across the street, it’s a pleasant spring evening, and we have a few minutes to kill before bath time – let’s go! As we set out, the kids squealed with delight when they saw Peridot running ahead of us down the driveway. In addition, the creature was able to forage different items from grass, foliage, and paving stones, which was impressive to them.
In this game, Habitats are map-based points of interest, which are common to all Niantic’s AR games (in Pokémon Go, they’re called PokéStops). I noticed my Peridot wanted to forage from the Habitat. Seeing this was only a short stroll away and not having been there before (I’m not a Pokémon Go player) – why not? The two of us headed over there. My interest turned out to be a Victorian stink pipe (I live in London, can you tell? ), which looks like a lamppost without a lamp, but is actually a giant straw that was designed to release noxious gasses from the sewers below, well above the heads of the good Queen’s subjects. It’s the first time I’ve heard of these things or noticed this one. Our pet leveled up (and we had a jolly time) after returning home after having an unplanned adventure, getting some exercise, and learning about local history. It couldn’t have been scripted better by Niantic.
Petridot is a typical pet sim in the style of Nintendogs, combined with Niantic’s vast mapping data and a new generation of AR technology. You interact with your pet to keep it happy and earn growth points, allowing it to level up from baby to teen to adult. When the creature reaches adulthood, it will want to be “released” into a habitat where it can breed with other Peridots (the animals are genderless) and sire a new baby for you to care for. Initially, Ziah Fogel stated that players would have to give up their adult Peridots permanently, but testers got upset, so you get to keep them.) Niantic has built enough variables into the creatures’ DNA that each one is genetically unique, so every pairing will result in a new, equally unique child.
Imagine the location-based features Niantic has built into this genre of game; taking your pet for walks, foraging from different environments, breeding with other local players. As for the customization features (yes, your pet can wear hats), you can imagine the gentle but insistent trickle of objectives, progress, currencies, and rewards built into a free-to-play live service game such as this.
With Peridot, Niantic wants to push the boundaries of AR and change the way people think about it. Ideally, your pet should be able to understand the real world that it (or your phone camera) is looking at and respond accordingly. Gabe Brostow, Niantic’s chief research scientist, leads Niantic’s R&D team, which occupies six floors of a narrow office building in London’s Covent Garden area.
The AI research begins with accurate mapping of the 3D space and pathfinding around real-world objects, as well as obstacle occlusion so that your pet disappears realistically behind chairs and tree trunks. (Brostow is frustrated that the designers asked for a shadow showing the Peridot’s position behind the object to be added back into the game, lest players get too stressed out about losing their charges.) With the help of machine learning, Brostow trains the code to differentiate grass from water, recognize a TV screen so that your Peridot can warn you about screen time, and identify classes of objects, such as pets, people, and food plates, from the video feed on your phone. Even five years ago, Niantic says, neural networks weren’t possible on mobile phones. According to its effect on battery life, Peridot certainly exercises your phone’s CPU and GPU.
Niantic’s efforts on Peridot began as a tech demo, and although the pet sim genre is a natural fit for Niantic’s efforts, you can tell. While it may be a labor of love from a game developer, it still feels like a skunkworks project from a tech company. Through Niantic’s Lightship suite of development tools, Brostow’s team will share its findings with the academic community and license the code to third parties. When Brostow performs home improvement, he holds his phone up to see the location of pipes behind walls and under floors, which he imagines could be useful for robotics or other assistive technologies. This combination of machine learning of environments and Niantic’s industrial scale geopositional data farming probably has more sinister uses, too, but don’t think about it – look at these cute little critters!
The technology is not perfect in practice: It has trouble detecting food reliably, can see water outside better than inside, and has trouble with windows. Watching your Peridot scamper around your home, jump up onto tables, and notice your cat is undeniably magical. As the game is released and players begin training Niantic’s neural nets in mass, it will feel even more alive. In Fogel’s view, a game about growing little creatures and introducing them to the world is a good thematic match for technology, which itself will improve its understanding of the world and its realism.
It’s impressive, in my opinion. It’s not my reaction that matters, or that tells you whether Peridot will succeed. Within minutes, my 4-year-old was wandering around the house calling for “pennydot.”. In addition, we all know about Victorian stink pipes.
