UROP Openings

Have a UROP opening you would like to submit?

Please fill out the form.

Submit your UROP opening

Multi-Modal Perception for Building Situation Models in Human-Robot Interactive Tasks




CSAIL: Computer Science and Artificial Intelligence Lab

Faculty Supervisor:

Julie Shah

Faculty email:


Apply by:



Nadia Figueroa, nadiafig@mit.edu

Project Description

Humans perceive the world as a unified set of sensations from multiple sensory modalities; i.e., our perception is multi-modal. Most research in human multi-modal perception suggests that in the perceptual processing pipeline the information from these multiple sensory modalities is combined and treated as a unitary representation of the world. Such representation of the world, which we will refer to as a “Situation Model”, can be seen as a cognitive representation of events, object location/motion and actions from other agents. The goal of this project is to build a situation model of the world, formulated as an aggregation of coupled beliefs over object and human states. Such beliefs will be updated by the robot’s multi-modal sensory inputs, which are provided by state-of-the-art (i) object tracking algorithm (from RGB-D images) and (ii) human hand tracking algorithm (from IMU+tracking data). The objective of the project is not the implementation of (i) or (ii), but the development of an amodal probabilistic representation of the objects and human states in the scene. Such a situation model will be used in the future to ground human demonstrated tasks on the perceptual inputs and will allow the incorporation of feedback mechanisms between robot and human for improving state representation.The UROP researcher is expected to have significant influence in shaping the representation of the situation model, which may involve (but not limited to): Bayesian networks, context-aware motion prediction, spatial reasoning, object physics model priors and inspiration from action-based theories of perception. The UROP researcher can work fully remotely on this project. However, depending on the progress (and eligibility https://urop.mit.edu/%3Cnolink%3E/alert-covid-19-urop-updates), limited access can be granted to our lab in CSAIL Building 32 from October to test algorithms on dedicated hardware. Credit option (with tuition waiver) is guaranteed and pay option is also possible upon further discussion.


The student researcher should be comfortable with numerical computation in Python or MATLAB and have background in machine learning at 6.036 (undergraduate ML) or equivalent/higher level, or ability to quickly pick it up, as evidenced by a strong mathematical background. Background in related topics, such as statistics, probabilistic modelling and robotics is preferred. Experience with ROS (Robot Operating System) and C++ is appreciated, yet not required