UROP Openings

Have a UROP opening you would like to submit?

Please fill out the form.

Submit your UROP opening

Painting Light and Style with a Generative Adversarial Network


Term:

Fall

Department:

6: Electrical Engineering and Computer Science

Faculty Supervisor:

Antonio Torralba

Faculty email:

torralba@mit.edu

Apply by:

9/7/2020

Contact:

David Bau: davidbau@csail.mit.edu

Project Description

How does a neural network, trained to imitate real photographs of the world, reason about light? In previous work (https://ganpaint.io/), we found that, when a generative adversarial network (GAN) is trained to synthesize real-looking photos, it learns sets of neurons that can be used to control the presence of object classes. Remarkably, manipulating neurons to add a light source such as a lamp or window, will also add reflected light, far from the object. In this project, we will investigate this phenomenon and improve our control of it. Our goal is to be able to improve our understanding of a GAN's model of attributes, and to create an interactive app that disentangles the factors of variation learned by a GAN (e.g., inspired by and collaborating with authors of https://ali-design.github.io/gan_steerability/). We want to be able to cleanly turn and off light sources, or change their color and intensity, without changing the presence of lamps or other objects in the scene. We also aim to cleanly control other attributes, such as style, size, and orientation of objects within a scene.

Pre-requisites

Computer vision knowledge (6.869 or similar); coursework or experience in machine learning, linear algebra, javascript, python.