Princeton Robotics Seminar: Physical Intelligence as API

Large Language Models (LLMs) are unprecedented in their ability to go beyond application-specific software and promise a one-stop solution to several digital tasks. With such advances, robotic agents are able to convert complex natural language commands into step-wise instructions. However, accurate and reliable execution of sensorimotor skills (e.g., locomotion, opening doors, object manipulation, etc.) remains elusive. I will discuss a framework that allows robots to learn new, complex, and generalizable behaviors while reducing the human effort in designing such behaviors to easily scale to many tasks. This framework is a stepping stone towards building Physical Intelligence as API  -- a one-stop robotics solution (i.e., an API) that can perform many manipulation and locomotion tasks that humans perform. I will elaborate on the framework using the following case studies:

(i) a dexterous manipulation system capable of re-orienting novel objects and tool use such as peeling vegetables.
(ii) a quadruped robot capable of fast locomotion and manipulation on diverse natural terrains.
(iii) object re-arrangement system tested on manipulating out-of-distribution object configurations.

Bio: Pulkit Agrawal is an Assistant Professor in the Department of Electrical Engineering and Computer Science at MIT, where he directs the Improbable AI Lab. His research interests span robotics, computer vision, and reinforcement learning. Pulkit's work received the Best Paper Award at Conference on Robot Learning 2021 and the Best Student Paper Award at the Conference on Computer Supported Collaborative Learning 2011. He is a recipient of the Sony Faculty Research Award, Salesforce Research Award, Amazon Research Award, a Fulbright fellowship, etc. Before joining MIT, he received his Ph.D. from UC Berkeley and Bachelor's degree from IIT Kanpur, where he was awarded the Directors Gold Medal.