UROP Openings

Have a UROP opening you would like to submit?

Please fill out the form.

Submit your UROP opening

Multi-Modal Sensor Fusion for Robust Hand Motion Tracking for Human Robot Interaction (Sensor Fusion and Deep Learning)




CSAIL: Computer Science and Artificial Intelligence Lab

Faculty Supervisor:

Julie Shah

Faculty email:


Apply by:



Nadia Figueroa, nadiafig@mit.edu

Project Description

The success of many human robot interaction (HRI) applications, where physical collaboration between human and robot is necessary, relies heavily on proper tracking of the human hand’s motion. This is normally achieved by tracking the human hand with a glove, equipped with either (i) an external optical marker-based camera system or (ii) wearable IMU sensors. While (i) yield measurements in a global reference frame that are robust to drift, measurements can often be lost due to occlusions and lighting effects. On the other hand, measurements from (ii) are robust to occlusions yet prone to drift and use a local reference frame -- needing an initial external calibration. This first part of this project involves implementing a robust state estimator that fuses the sensor measurements from (i) and (ii) to provide reliable estimates of the human hand’s motion. The UROP researcher will perform a literature review on existing sensor fusion approaches for multi-modal measurements and develop a robust algorithm to fuse measurements from (i) and (ii). The second part of this project focuses on replacing the measurements from (ii) with recently developed (iii) glove-less deep learning based hand-tracking approaches for RGB-D data. While preliminary results from these approaches seem promising, applying them to track a human’s hand motion during an interactive human-robot scenario is yet to be verified. The UROP researcher is expected to have significant influence in shaping the algorithm for sensor fusion in both scenarios. This project can also be performed in tandem between two UROP researchers. The UROP researcher/s will work partially remote/in-person. If eligible (https://urop.mit.edu/%3Cnolink%3E/alert-covid-19-urop-updates) limited access to our lab in CSAIL Building 32 or 31 will be granted in October to test algorithms on dedicated hardware. Credit option (with tuition waiver) is guaranteed and pay option is also possible upon further discussion.


The student researcher should be comfortable with numerical computation in Python or MATLAB and have background in state estimation and bayesian filtering, or strong mathematical background. Background in related topics, such as machine learning, control theory, signal processing, statistics, robotics and computer vision is preferred. Experience with ROS (Robot Operating System) and C++ is appreciated.