Princeton Robotics Seminar

Multi-robot systems are becoming more pervasive all around us, in the form of fleets of autonomous vehicles, future delivery drones, and robotic teammates for search and rescue.  As a result, it becomes increasingly critical to question the robustness of their coordination algorithms to reliable information exchange, security threats and/or corrupted data. This talk will focus on the role of control and information exchange for enhancing situational awareness and security of multirobot systems. An example is the consensus problem where classical results hold that agreement cannot be reached when malicious agents make up more than half of the network connectivity; this quickly leads to limitations in the practicality of many multi-robot coordination tasks. However, with the growing prevalence of cyber-physical systems comes novel opportunities for detecting attacks by using cross-validation with physical channels of information. In this talk we consider the class of problems where the probability of a particular (i,j) link being trustworthy is available as a random variable. We refer to these as “stochastic observations of trust.” We show that under this model, strong performance guarantees such as convergence for the consensus problem can be recovered, even in the case where the number of malicious agents is greater than ½ of the network connectivity and consensus would otherwise fail. We will present both a theoretical framework, and experimental results, for provably securing multi-robot distributed algorithms through careful use of communication.  Lastly, we will present promising results on new communication-centric methods for learning and sequential decision-making in tomorrow’s multi-robot systems.

Bio: Stephanie is an Assistant Professor in the John A. Paulson School of Engineering and Applied Sciences (SEAS) at Harvard University. Her work centers around trust and coordination in multi-robot systems for which she has received the Office of Naval Research Young Investigator award (2021) and the National Science Foundation CAREER award (2019). She has also been selected as a 2020 Sloan Research Fellow for her contributions at the intersection of robotics and communication. She has held a Visiting Assistant Professor position at Stanford University during the summer of 2019, and an Assistant Professorship at Arizona State University from 2018-2020. She completed her Ph.D. work (2014) on multi-robot coordination and control and her M.S. work (2009) on system identification and model learning. At MIT she collaborated extensively with the wireless communications group NetMIT, the result of which were two U.S. patents recently awarded in adaptive heterogeneous networks for multi-robot systems and accurate indoor positioning using Wi-Fi.  She completed her B.S. at Cornell University.

At the Toyota Research Institute (TRI), we've been working on behavior cloning for dexterous manipulation. Building on the Diffusion Policy framework that we've recently developed in collaboration with Shuran Song, we now have a very solid pipeline for taking ~50-100 bimanual haptic teleop demonstrations and turning that into a surprisingly effective visuomotor (+tactile) policy. Because there is no explicit state representation required, these skills work equally well manipulating deformable, liquid, or other difficult to model tasks as they do for more traditional rigid-object manipulation. We're actively scaling this up into the multi-task setting and now see a plausible path towards "Large Behavior Models."

Bio: Russ Tedrake is the Toyota Professor at the Massachusetts Institute of Technology (MIT) in the Department of Electrical Engineering and Computer Science, Mechanical Engineering, and Aero/Astro, and he is a member of MIT’s Computer Science and Artificial Intelligence Lab (CSAIL). He is also the Vice President of Robotics Research at Toyota Research Institute (TRI). He received a B.S.E. in Computer Engineering from the University of Michigan in 1999, and a Ph.D. in Electrical Engineering and Computer Science from MIT in 2004. Dr. Tedrake is the Director of the MIT CSAIL Center for Robotics and was the leader of MIT’s entry in the DARPA Robotics Challenge. He is a recipient of the NSF CAREER Award, the MIT Jerome Saltzer Award for undergraduate teaching, the DARPA Young Faculty Award in Mathematics, the 2012 Ruth and Joel Spira Teaching Award, and was named a Microsoft Research New Faculty Fellow. His research has been recognized with numerous conference best paper awards, including ICRA, Robotics: Science and Systems, Humanoids, Hybrid Systems: Computation and Control, as well as the inaugural best paper award from the IEEE RAS Technical Committee on Whole-Body Control.

Robots hold the promise of serving human needs, like helping older adults live independently at home or assisting drivers in preventing crashes. For these robots to integrate seamlessly into people's lives, they must provide proactive assistance that is responsive to their human partners' needs. Often, these needs are a result of underlying mental states like intent or awareness. Conversely, it is also useful for people to have an accurate mental model of their robot assistant's policy and knowledge. Mental states may be revealed implicitly through actions the agents take, such as gazing at a certain object or moving in a certain way. This talk describes research on developing collaborative robots that infer people's needs through interaction, adapt to people's individual preferences, and communicate their own models to make the interaction more explainable. These robots are evaluated in a range of human-robot interaction domains, such as manipulation and driving.

Bio: Dr. Henny Admoni is an Associate Professor in the Robotics Institute at Carnegie Mellon University, where she leads the Human And Robot Partners (HARP) Lab. Dr. Admoni’s research interests include human-robot interaction, assistive robotics, and nonverbal communication. Dr. Admoni holds a PhD in Computer Science from Yale University, and a BA/MA joint degree in Computer Science from Wesleyan University.

Over the last decade, a variety of paradigms have sought to teach robots complex and dexterous behaviors in real-world environments. On one end of the spectrum we have nativist approaches that bake in fundamental human knowledge through physics models, simulators and knowledge graphs. While on the other end of the spectrum we have tabula-rasa approaches that teach robots from scratch. In this talk I will argue for the need for better constructivist approaches to robotics, i.e. techniques that take guidance from humans while allowing robots to continuously adapt in changing scenarios. The constructivist guide I propose will focus on three lessons. First, creating physical interfaces to allow humans to provide robots with rich and dexterous data. Second, developing adaptive learning mechanisms to allow robots to continually fine-tune in their environments. Third, architecting models that allow robots to learn from un-curated play. Applications of such a learning paradigm will be demonstrated on mobile manipulators in home environments, industrial robots on precision tasks, and multi-fingered hands on dexterous manipulation.

Bio:  Lerrel Pinto is an Assistant Professor of Computer Science at NYU. His research interests focus on creating general purpose robotic systems. He received a PhD degree from CMU in 2019; prior to that he received an MS degree from CMU in 2016, and a B.Tech in Mechanical Engineering from IIT-Guwahati. His work on robotics received paper awards at ICRA 2016 and RSS 2023, and finalist awards at IROS 2019 and CoRL 2022. He is a recipient of grants and awards from Amazon, Honda, Hyundai, Meta, LG and Google. More recently, he was named a TR35 innovator under 35 for 2023. Several of his works have been featured in popular media like TechCrunch, MIT Tech Review, Wired, and BuzzFeed among others. His recent work can be found on www.lerrelpinto.com.

Large Language Models (LLMs) are unprecedented in their ability to go beyond application-specific software and promise a one-stop solution to several digital tasks. With such advances, robotic agents are able to convert complex natural language commands into step-wise instructions. However, accurate and reliable execution of sensorimotor skills (e.g., locomotion, opening doors, object manipulation, etc.) remains elusive. I will discuss a framework that allows robots to learn new, complex, and generalizable behaviors while reducing the human effort in designing such behaviors to easily scale to many tasks. This framework is a stepping stone towards building Physical Intelligence as API  -- a one-stop robotics solution (i.e., an API) that can perform many manipulation and locomotion tasks that humans perform. I will elaborate on the framework using the following case studies:

(i) a dexterous manipulation system capable of re-orienting novel objects and tool use such as peeling vegetables.
(ii) a quadruped robot capable of fast locomotion and manipulation on diverse natural terrains.
(iii) object re-arrangement system tested on manipulating out-of-distribution object configurations.

Bio: Pulkit Agrawal is an Assistant Professor in the Department of Electrical Engineering and Computer Science at MIT, where he directs the Improbable AI Lab. His research interests span robotics, computer vision, and reinforcement learning. Pulkit's work received the Best Paper Award at Conference on Robot Learning 2021 and the Best Student Paper Award at the Conference on Computer Supported Collaborative Learning 2011. He is a recipient of the Sony Faculty Research Award, Salesforce Research Award, Amazon Research Award, a Fulbright fellowship, etc. Before joining MIT, he received his Ph.D. from UC Berkeley and Bachelor's degree from IIT Kanpur, where he was awarded the Directors Gold Medal.

Zoom link: https://princeton.zoom.us/my/robotics

---

To become useful in human-centric environments, robots must demonstrate language comprehension, semantic understanding and logical reasoning capabilities working in concert with low-level physical skills. With the advent of modern "foundation models" trained on massive datasets, the algorithmic path to developing general-purpose “robot brains” is (arguably) becoming clearer, though many challenges remain.  In the first part of this talk, I will attempt to give a flavor of  how state-of-the-art multimodal foundation models are built, and how they can be bridged with low-level control. In the second part of the talk, I will summarize a few surprising lessons on control synthesis observed while solving a collection of Robotics benchmarks at Google. I will end with some emerging open problems and opportunities at the intersection of dynamics, control and foundation models.

Bio: Vikas Sindhwani is Research Scientist at Google Deepmind in New York where he leads a research group focused on solving a range of planning, perception, learning, and control problems arising in Robotics.  His interests are broadly in core mathematical foundations of statistical machine learning, and in end-to-end design aspects of building large-scale and robust AI systems. He received the best paper award at Uncertainty in Artificial Intelligence (UAI-2013), the IBM Pat Goldberg Memorial Award in 2014, and was finalist for Outstanding Planning Paper Award at ICRA-2022. He serves on the  editorial board of Transactions on Machine Learning Research (TMLR) and IEEE Transactions on Pattern Analysis and Machine Intelligence; he has been area chair and senior program committee member for NeurIPS, International Conference on Learning Representations (ICLR) and Knowledge Discovery and Data Mining (KDD). He previously headed the Machine Learning group at IBM Research, NY. He has a PhD in Computer Science from the University of Chicago and a B.Tech in Engineering Physics from Indian Institute of Technology (IIT) Mumbai. His publications are available at: http://vikas.sindhwani.org/

This talk introduces dynamic game theory as a natural modeling tool for multi-agent interactions ranging from large, abstract systems such as ride-hailing networks to more concrete, physically-embodied robotic settings such as collision-avoidance in traffic. We present the key theoretical underpinnings of dynamic game models for these varied situations and draw attention to the subtleties of information structure, i.e., what information is implicitly made available to each agent in a game. Thus equipped, the talk presents a state-of-the-art technique for solving these games, as well as a set of “dual” techniques for the inverse problem of identifying players’ objectives based on observations of strategic behavior.

Bio: David Fridovich-Keil is an assistant professor at the University of Texas at Austin. David’s research spans optimal control, dynamic game theory, learning for control, and robot safety. While he has also worked on problems in distributed control, reinforcement learning, and active search, he is currently investigating the role of dynamic game theory in multi-agent interactive settings such as traffic. David’s work also focuses on the interplay between machine learning and classical ideas from robust, adaptive, and geometric control theory.

In this talk, I will introduce our recent work on origami mechanisms and actuation strategies for applications spanning from biomedical devices to foldable space structures. The first topic is magnetically actuated millimeter-scale origami medical robots for effective amphibious locomotion in severely confined spaces or aqueous environments. The origami robots are based on the Kresling origami, whose thin shell structure 1) provides an internal cavity for drug storage, 2) permits torsion-induced contraction as a crawling mechanism and a pumping mechanism for controllable liquid medicine dispensing, 3) serves as propellers that spin for propulsion to swim, 4) offers anisotropic stiffness to overcome the large resistance from the severely confined spaces in biomedical environments. For the second part of my talk, the concept of hexagonal ring origami folding mechanism will be introduced as a strategy for deployable/foldable structures for space applications. The hexagonal rings can tessellate 2D/3D surfaces and each ring can snap to its stable folded configuration with only 10.6% of the initial area. Through finite-element analysis and the rod model, snap-folding of the hexagonal ring with slight geometric modification and residual strain are studied for easy folding of the ring to facilitate the design and actuation of hexagonal ring origami assemblies for functional foldable structures with extreme packing ratio.

Bio: Renee Zhao is an Assistant Professor of Mechanical Engineering at Stanford University. Renee received her PhD degree in Solid Mechanics from Brown University in 2016. She spent two years as a postdoc associate at MIT working on modeling of soft composites. Before Renee joined Stanford, she was an Assistant Professor at The Ohio State University from 2018 to 2021. Renee’s research concerns the development of stimuli-responsive soft composites and shape morning mechanisms for multifunctional robotic systems. Renee is a recipient of the NSF Career Award (2020), AFOSR YIP (2023), ASME Journal of Applied Mechanics award (2021), the 2022 ASME Pi Tau Sigma Gold Medal, and the 2022 ASME Henry Hess Early Career Publication Award.

In this talk, I will provide some insights and observations from our recent work on Atlas, the world's most dynamic humanoid robot. I'll cover the core technical ideas that have made an impact for us over the past few years and share my thoughts about the future for robots like Atlas.

Bio: Scott Kuindersma is the Senior Director of Robotics Research at Boston Dynamics where he leads behavior research on Atlas. Prior to joining Boston Dynamics, he was an Assistant Professor of Engineering and Computer Science at Harvard. Scott’s research explores intersections of machine learning and model-based control to improve the capabilities of humanoids and other dynamic mobile manipulators.

There have been significant advances in the field of robot learning in the past decade. However, many challenges still remain when considering how robot learning can advance interactive agents such as robots that collaborate with humans. In this talk, I will be discussing the role of learning representations for robots that interact with humans and robots that interactively learn from humans through a few different vignettes. I will first discuss how bounded rationality of humans guided us towards developing learned latent action spaces for shared autonomy. It turns out this “bounded rationality” is not a bug and a feature — i.e. we can develop extremely efficient coordination algorithms by learning latent representations of partner strategies and operating in this low dimensional space. I will then discuss how we can go about actively learning such representations capturing human preferences including our recent work on how large language models can help design human preference reward functions. Finally, I will end the talk with a discussion of the type of representations useful for learning a robotics foundation model and some preliminary results on a new model that leverages language supervision to shape representations.

Bio: Dorsa Sadigh is an assistant professor in Computer Science and Electrical Engineering at Stanford University.  Her research interests lie in the intersection of robotics, learning, and control theory. Specifically, she is interested in developing algorithms for safe and adaptive human-robot and human-AI interaction. Dorsa received her doctoral degree in Electrical Engineering and Computer Sciences (EECS) from UC Berkeley in 2017, and received her bachelor’s degree in EECS from UC Berkeley in 2012.  She is awarded the Sloan Fellowship, NSF CAREER, ONR Young Investigator Award, AFOSR Young Investigator Award, DARPA Young Faculty Award, Okawa Foundation Fellowship, MIT TR35, and the IEEE RAS Early Academic Career Award.