UROP Openings

Have a UROP opening you would like to submit?

Please fill out the form.

Submit your UROP opening

Deep-Learning based Object Tracking for Human-Robot Interactive Learning




CSAIL: Computer Science and Artificial Intelligence Lab

Faculty Supervisor:

Julie Shah

Faculty email:


Apply by:



Nadia Figueroa, nadiafig@mit.edu

Project Description

A key component in robot learning from demonstration and interactive learning is tracking the 6D pose of objects that are being handled/manipulated by the human teacher. This is normally achieved by using marker-based approaches in which external camera systems and bulky markers are placed on the objects. However, relying on external marker-based tracking systems becomes a nuisance when such interactive learning algorithms are sought to be deployed outside of the lab. To this end, in this project, the UROP researcher will develop and implement a marker-less object tracking algorithm using state-of-the-art deep learning approaches on RGB-D data. While deep learning approaches for object recognition on static images have shown promising results, applying them to track continuous motion of objects (in real-time) is still an open problem. The goal of this work is to extend existing deep learning object tracking approaches to provide a robust tracking of objects being manipulated by a human demonstrator. The UROP researcher is expected to have significant influence in shaping these extensions, which can include (but not limited to): physics-based predictive models for object dynamics, Bayesian filtering and/or trajectory prediction strategies. The UROP researcher can work partially remote/in-person. If eligible (https://urop.mit.edu/%3Cnolink%3E/alert-covid-19-urop-updates) limited access to our lab in CSAIL Building 32 will be granted in October to test algorithms on dedicated hardware. Credit option (with tuition waiver) is guaranteed and a pay option is also possible upon further discussion.


The student researcher should be comfortable with numerical computation in Python or MATLAB and have a background in machine learning at 6.036 (undergraduate ML) or equivalent/higher level, or ability to quickly pick it up, as evidenced by strong mathematical background. Background in related topics, such as computer vision, statistics, bayesian modeling and robotics is preferred. Experience with ROS (Robot Operating System) and C++ is appreciated.