This semester I've been taking Interactive Installations, a class in NYU's Tandon School of Engineering. The focus of the course has been learning Max/MSP/Jitter, a family of visual programming tools for creating interactive audio/visual projects.
At first I was frustrated working with Max, because I'm used to writing lines of code. Instead of writing code, Max has you drag and drop nodes and connect them with cords to control the sequence of commands. It reminds me of Scratch, an introductory visual programming language I learned as a kid. You can trigger actions to happen on mouse clicks, on key presses, or when the program is opened.
The Scratch UI, where the colored blocks are different commands chained together
This is a Max patch I wrote for a video-based project. It's a big 2D plane of messy, interconnected nodes. For my own sanity, I organized chunks of nodes into logical units with colored backgrounds. Each of these colored sections accomplishes a task, like handling mouse movement or crossfading between two images. Eventually I got used to thinking about Max in a different way from how I approach writing code. Max is great for having multiple things happen at the same time, and constantly adapting to user input. On the other hand, when I write code I think of events happening linearly and behavior organized into discrete functions.
An example of the Max UI. This is called a "patch" in Max
While it's possible to write code within Max, this class has been entirely focused on using the visual editor. Here's some projects I've completed in the past few weeks.
Collagescape
This Interactive Installations class is described as "bring your own media." Meaning that the assignments are very open-ended and can work with whatever themes or content you want. For a video-based assignment, I decided to work with the collage pieces on my bedroom wall. I cover my walls in photos, magazine cutouts, postcards, polaroids, lots of paper-like pieces that I've scavenged throughout the years. I took a photo of my wall and used Photoshop to create several frames where the collage pieces animate out one by one. I exported this into a video so it looks like the collage is gradually disappearing to leave behind an empty wall.
The result of over 100 Photoshopped frames, each one removing a single photo from the wall
I wrote a Max patch to have this video's playback be controlled via mouse position. Moving the computer's mouse from left to right advances the video frame by frame, so it feels like the mouse is brushing over the wall and erasing collage pieces. That was my original vision, but things evolved from there. I added more controls to edit RGB color, video mixing, contrast, saturation, and more. You can read more about my process here. This let me get a pretty wide range of visual effects:
Interacting with Collagescape
Finally I added a keyboard shortcut to let me take a screenshot. This let me preserve output very quickly and produce some cool artwork!
Sample images generated with Collagescape
Over the Moon
The midterm project had a more specific prompt:
Using Max/MSP/Jitter and openCV, create an interactive experience which explores multi-temporality and memory. Develop an interactive system which explores these notions of time, creating an experience with the past and the present simultaneously
I decided to reimagine the 1969 Apollo 11 moon landing as an interactive experience to help keep the event alive in cultural memory. Using restored NASA footage as a background, I green-screened participants into the scene to witness man's first steps on the moon. Periodically, I recorded short videos of the participant and pasted their videos into the scene alongside their live green-screened video avatar. These videos pop around to various locations in the scene as time passes. Read more about my design and build process here.
The experience includes original audio from NASA, in which you can hear Neil Armstrong communicate with ground control and have a phone call with President Nixon. I also included a remix of Fly Me To the Moon by Frank Sinatra, since this was supposedly the first song played on the moon via a cassette player in the lunar module.
I had a fun time installing this piece in the NYU Media Commons Ballroom, even though I was only able to have it up for one day.