Towards Generalist Robots:
Learning Paradigms for Scalable Skill Acquisition

Conference on Robot Learning 2023, Atlanta, USA

Monday, Nov 6th, 2023

Location: Sequoia 1

Recording available at: https://www.youtube.com/watch?v=3-FkI6Qlj6s

We have witnessed very impressive progress in large-scale and multi-modal foundation/generative models in recent months. We believe making use of such models in a reasonable way could really enable robots to acquire diverse skills. In the recent white paper, we discussed how we can automate the whole pipeline for robotic skill learning, from low-level asset generation, texture generation, to high-level scene, task and reward generation. Once we obtain such a diverse suite of tasks and environments, we can offload policy training to RL of trajectory optimization to solve all the generated low-level tasks, and finally distill all the learned closed-loop policy into a unified policy model. Apart from scaling up in simulation, how to use real-world data more effectively is another promising research direction. Real-world human demonstrations can be found at scale, but typically only provides spatial trajectory information and doesn’t advise how to recover from error compounding during policy rollout.

Motivated by these observations and thoughts, this workshop seeks to discuss and compare the advantages and limitations of different paradigms for scaling up skill learning: scaling up simulation, leveraging generative models, exploiting unstructured passive human demonstration, scaling up structured demonstration collection in the real world, etc.

We aim to discuss questions including, but not limited to:

Is automating task and scene generation in simulation a valid paradigm to pursue?
Are today’s simulators capable of simulating diverse physical phenomena that would be encountered in real-world robotic tasks?
Is sim2real transfer a major bottleneck? What are the potential solutions to it?
What are the pros and cons of collecting learning from real-world data compared to simulated data?
How to make collecting and learning from real-world data more efficient and effective?
Is there a way to combine the strengths of both simulated and real-world data for learning generalist robots?

Call for papers

We invite submissions including but not limited to the following topics:

Scaling up robotic data with generative AI
Exploiting knowledge from vision and language foundation models for robotics
Scalable skill learning in simulation and sim-to-real transfer
Scalable real-world data collection and robot learning
Learning robotic foundation models by leveraging internet-scale data

Important Dates

Paper submission open: 2023/09/07
Paper submission deadline: 2023/10/06 (Extended to 2023/10/08 AoE)
Notification of acceptance: 2023/10/20
Workshop date: 2023/11/06
Submission portal: CoRL 2023 Workshop TGR (OpenReview).

We expect submissions with 4 - 8 pages for the main content, with no limit on references/appendices. Submissions are suggested to use the CoRL template and must be anonymized. All papers will be peer-reviewed in a double-blind manner. We welcome both unpublished original contributions and recently published relevant works. Accepted papers will be presented in the form of posters, with several papers being selected for spotlight sessions.

Contact

If you have any questions, please contact us at: generalistrobots@gmail.com.