Google has released a new open source benchmark that allows robots to perform tasks by writing their own code in response to instructions written by humans.
The company has a new website (opens in new tab) to reveal “Code as Policies” (CAP), where prompts written in plain English can be interpreted in language model-generated programs (LMPs) written in Python code.
CAP is the successor of PaLM-SayCan (opens in new tab), a project that similarly allowed a physical helper robot to be controlled by means of commands in plain English. CAP promises to make it possible to perform more complex tasks with greater accuracy, in part by allowing machines to write their own code.
In a blog post (opens in new tab) discussing the release of CAP, Google Research Intern Jacky Liang and Research Scientist Andy Zeng describe the motivation for the technology and what it could mean for the future.
“What if robots, when instructed by humans, could autonomously write their own code to communicate with the world? […] Given instructions in natural language, current language models are very adept at writing not only generic code, but, as we have discovered, code that can also control robotic actions.”
But it might not be the time to throw your laptop away for programming just yet. In testing, Google’s researchers demonstrated simple commands with a similar structure. Test robots were able to “draw a 5 cm hexagon around the center” [of a whiteboard]and “put the blocks in a horizontal line at the top” [of a square boundary].
In the accompanying paper (opens in new tab), titled “Code as Policies: Language Model Programs for Embodied Control,” the project team admits that CAP is currently unable to handle particularly abstract or complex commands or perceive trajectory descriptions. The team’s approach also did not take into account impossible assignments given through CAP.
In theory, the open source nature of Google’s “robot-centered” implementation of LMPs in Python could result in much faster implementation of solutions to these problems. The CAP website also includes: releases (opens in new tab) via Github, and an interactive demo (opens in new tab) via Google Colab to describe how robots “write” code in response to commands.