What Do We Learn from a Large-Scale Study of Pre-Trained Visual Representations in Sim and Real Environments?
We evaluate 5 different PVR on 5 sim and 5 real benchmarksAbstract
We present a large empirical investigation on the use of pre-trained visual representations (PVRs) for training downstream policies that execute real-world tasks. Our study spans five different PVRs, two different policy-learning paradigms (imitation and reinforcement learning), and three different robots for 5 distinct manipulation and indoor navigation tasks. From this effort, we can arrive at three insights: 1) the performance trends of PVRs in the simulation are generally indicative of their trends in the real world, 2) the use of PVRs enables a first-of-its-kind result with indoor ImageNav (zero-shot transfer to a held-out scene in the real world), and 3) the benefits from variations in PVRs, primarily data-augmentation and fine-tuning, also transfer to the real-world performance. See project website for additional details and visuals.
Karmesh Yadav
Ph.D. Student at Georgia Tech
My research interests include Embodied AI, Robotics and Reinforcement Learning.