Karmesh Yadav

Karmesh Yadav

Ph.D. Student at Georgia Tech

Georgia Tech

Biography

Hi! I am Karmesh, a PhD student at Georgia Tech, advised by Prof. Dhruv Batra and Prof. Zsolt Kira.

During my PhD I am interested in creating better pretraining strategies for Embodied AI agents. Previously, I was an AI Resident at FAIR, working with the Habitat and Cortex team under the supervision of Dr. Oleksandr Maksymets and Prof. Batra. Before that, I worked as a Senior Robotics Engineer at ISEE, an autonomous vehicles startup working on automating yard trucks. I completed my Masters in Robotics Systems Development (MRSD) at the CMU Robotics Institute in 2020.

Download my resumé.

Interests
  • Embodied AI
  • Robot Learning
  • Reinforcement Learning
Education
  • Ph.D. in Computer Science, 2026

    Georgia Institute of Technology

  • Masters in Robotic Systems Development, 2020

    Carnegie Mellon University

  • B.Tech in Mechanical Engineering, 2017

    Indian Institute of Technology, Guwahati

Experience

 
 
 
 
 
AI Resident
Aug 2021 – Jun 2023 Menlo Park
  • Worked on self-supervised pretraining techniques for learning useful representations for embodied agents.
  • Released the HM3D-Semantics dataset and the Open-Vocabulary Mobile Manipulation benchmark based of Habitat Simulator.
  • Organised multiple challenges on Embodied Navigation and Rearrangement in CVPR and NeurIPS.
 
 
 
 
 
Robotics Engineer
Jul 2020 – Aug 2021 Pittsburgh
  • Explored deep uncertainty estimation techniques for predicting the closed loop tracking performance of an autonomous vehicle controller. Estimated the collision probability of the AV with respect to obstacles in an occupancy grid.
  • Improved the trajectory optimization planner and robustified its collision checking. This led to an increased confidence in its performance and resulted in its deployment on the AV.
  • Developed the speed planning module for safely achieving three-fold increase in the operating speed of the AV.
 
 
 
 
 
Software Development Intern
May 2019 – Aug 2019 Boston
  • Built toolboxes to automate the system identification and calibration procedure of Isee’s vehicles.
  • Researched and implemented various vehicle and tire models for control application in AVs.

Publications & Preprint

Quickly discover relevant content by filtering publications.
OpenEQA: Embodied Question Answering in the Era of Foundation Models
We present a new embodied question answering (EQA) dataset with open vocabulary questions.
OpenEQA: Embodied Question Answering in the Era of Foundation Models
What Do We Learn from a Large-Scale Study of Pre-Trained Visual Representations in Sim and Real Environments?
We conduct a study on using pre-trained visual representations (PVRs) to train robots for real-world tasks.
What Do We Learn from a Large-Scale Study of Pre-Trained Visual Representations in Sim and Real Environments?
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
We present the largest and most comprehensive empirical study of visual foundation models for Embodied AI (EAI).
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
HomeRobot: Open-Vocabulary Mobile Manipulation
We propose a combined simulation and real-world benchmark on the problem of Open-Vocabulary Mobile Manipulation (OVMM).
HomeRobot: Open-Vocabulary Mobile Manipulation
Navigating to Objects Specified by Images
We present a modular system that can perform well on the Instance ImageNav task in both simulation and the real world.
Navigating to Objects Specified by Images
Habitat-matterport 3d semantics dataset
We present Habitat-Matterport 3D Semantics (HM3DSEM), the largest dataset of 3D real-world spaces with densely annotated semantics.
Habitat-matterport 3d semantics dataset

Recent & Upcoming Talks

NeurIPS 2020 Oral Presentation
An Oral Presentation on Look-Ahead MAML
NeurIPS 2020 Oral Presentation