Google's Gemini AI Is Making Robots Smarter

While skeptics may say the AI revolution looks like it's all about chatbots and amazing, if weird, digital artwork creation, there's actually a lot more going on behind the scenes. Google just delightfully demonstrated the depth of the technology's promise with its AI-powered robots. As part of a research project, some distinctly nonhuman-looking robots are roaming the corridors of Google's DeepMind AI division, busily learning how to navigate and interact with the Googler employees they encountered. The experiment gives us a tantalizing glimpse of what the machines that might make up a looming robot revolution will be capable of when they're put to work in our homes, factories and offices--in their millions and billions, if you believe some prominent futurists. In a research paper, Google explains it's been examining "Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs." Behind the tech jargon, it all boils down to using AI to move around the spaces in Google's offices and interact with humans using "long context" prompts. The long context bit is very important--it relates to how much information the AI model, Gemini 1.5 Pro, can take in and process in one input session using natural language. Essentially it's about giving the robot a sense of context, so it can remember lots of details about its interaction with people and what they've said or asked the robot to do. Think about how you ask a very simple AI like Amazon's Alexa a question, only to realize that a moment later she's "forgotten" it, and can't carry on a human-like conversation--this is part of what the Google experiment is tackling. In videos documenting the project, Google shows some examples of how the AI-powered robots function in the workplace, website TechCrunch notes. One example shows the robots being asked by a user to take him somewhere he can draw something--after a moment the robot matches up this request with what it knows about objects that can be drawn on, and where they are, and it leads the Googler to a whiteboard. Though it sounds simple, this is actually a higher level of reasoning that's much more human-like than many earlier AI/robot systems have been capable of. The Alexa example is good here again: Alexa is clever, but only understands very specific commands, and if you've used her natural language system you'll have encountered Alexa's very limited reasoning when she complains she doesn't understand, until you tweak your wording. Another part of the Google project involved teaching the robots about the environment they were going to be navigating. While earlier robot systems may have been trained using very precisely input maps of office or factory floors, or even being initially tele-operated around the space by a human so their sensors learn the layout of their surroundings, the new Google robots were trained by having their AI "watch" a walkthrough video made on a smartphone. The video showed that the AI bot could identify objects, like furniture or electrical sockets, remember where they are, and then reason what a user meant when it asked the robot to, for example, help them charge their smartphone. Or, demonstrating even more smarts, they knew what to do when a user asked for more of "this," pointing to soda cans on the person's desk, and knowing it should go and check the office fridge for supplies. While Google's robots in the video are very artificial looking (the machines themselves were actually left over from an earlier research project, TechCrunch noted) and there is a definite delay issue, with up to a minute of "thinking time" between being the robot receiving the request and then acting on it, Google's project is still an exciting potential preview of what's to come. It tracks with recent news that a different startup, Skild, has raised $300 million in funding to help build a universal AI brain for all sorts of robots. And it supports thinking by robot tech enthusiasts like Bill Gates, Jeff Bezos and Elon Musk who are certain that we'll all be buying AI-powered humanoid robots pretty soon, ready to welcome them into our homes and workspaces. That has been a promise made every year since the mid-20th century, though. Remember Robbie the Robot? He'd have some pithy things to say about Google's spindly, slow-thinking robots. BY KIT EATON @KITEATON