Can you remember, let alone depict, your dreams? Dreams have traditionally been difficult to express, due to their abstract and intangible nature. However, with the recent emergence of generative art AI technology such as DALL-E 2, people can now create images, music, and even videos about their dreams with just a few clicks and prompts.

Ruihan Zhang, a graduate student in the MIT Media Lab’s Molecular Machines group, was lead organizer of the recent MIT AI for Filmmaking Hackathon, an event created by the MIT Film Makers Association that aimed to bring dreams to life through AI and film. The hackathon brought together participants—including MIT students and staff, as well as high-profile speakers and judges—from various backgrounds and disciplines to utilize AI in the filmmaking process and produce short films. 

Ruihan Zhang was one of six event organizers from the MIT Film Makers Association, each from different MIT departments: Shangdi Yu and Honghao Cao of MIT Electrical Engineering and Computer Science (EECS), Yiming Zhang of MIT Biological Engineering, Zhibo Chen of MIT Aeronautics and Astronautics, and Songchen Tan of MIT Computational Science and Engineering.

“Tell me your dream” was the theme behind the hackathon, which occurred this past January as part of MIT’s Independent Activities Period (IAP). Participants were challenged to create a two-minute film within eight hours, and the films were evaluated by judges from academia and industry based on creative execution and effective use of AI techniques. “Everyone has a tale to tell, and with the advancements in AI technology, individuals now have the means to bring their dreams to life,” said hackathon co-organizer Shangdi Yu, a PhD student in MIT EECS. 

Unlike other hackathons on the MIT campus, the AI for Filmmaking Hackathon was the first event that focused specifically on film creation. The complexity of filmmaking offers numerous opportunities for AI to showcase its capabilities, such as using ChatGPT for script generation, DALL-E 2 and Midjourney for image generation, and Deforum for video synthesis. The films produced in the hackathon surpassed many people's expectations.

“DOG: Dream Of Galaxy” won best overall film, patching together colorful, AI-generated moving images to tell a touching story of a dog cadet selected to travel to space. The turning point of this dog’s life is when an anonymous man asked the stray dog, “Do you want to dream a different dream?” The team leader behind this film, Liu Yang, is a visiting student from Tsinghua University and a research affiliate in the Media Lab’s Tangible Media group.

In “Dreams Across the Stars,” an elderly man dreams about becoming a little boy again and meets outlandish aliens, dancing spacemen, and sees numerous stars in the universe. The vivid and imaginative paintings were generated by DALL-E and Midjourney. Another film titled “The Scoop” follows a team of researchers on a serious mission, watched over by a mysterious man escorted by the alert MIT rockstar mini-cheetah—yet they turned out to be creating the perfect scoop of ice cream with liquid nitrogen. 

The most popular video from the hackathon was "Once upon a spacetime," which skillfully blends elements of sci-fi westerns and dream-like AI art, generated by Deforum. And the film “Infinity Corridor” recreates the experience of students' first arrival at MIT, surrounded by the breadth and depth of its knowledge. AI was utilized to create a mesmerizing world filled with strange and diverse objects, capturing the awe and overwhelming feeling.

The highlight of this hackathon was a panel discussion where people from academia and industry jointly discussed the future of AI for filmmaking. The contest’s panel included television producer and director Greg Daniels (who worked on shows like Saturday Night Live, Parks and Recreation, Upload, Spaceforce, and The Simpsons), MIT Media Lab researcher Pat Pataranutaporn, Brown University Assistant Professor in AI James Tompkin, production designer and set decorator Rachel O'Toole, content strategist and former YouTube head of comedy Ben Relles, and television director and producer David Rogers (who has won two Emmys for his work in The Office).

The panelists believed AI will rapidly change job descriptions and the required skills in the film industry. Some expressed concerns that the increasing use of personalized generative AI may compromise shared memories of films that have a lasting cultural impact.

“AI algorithms enable more amateurs to express their ideas without professional training, and lead to the de-privileging and popularizing of artistic creation. Focus more on creativity, leave the rest to AI,” said Liu Yang. Many participants echoed the same idea, and the yearning for AI for art persisted in many participants long after the hackathon. Given the popularity of this year’s event, the MIT Film Makers Association plans to organize another hackathon next year. “In a technology-driven era of art creation,” said Ruihan Zhang, “I am eager to see how MIT students leverage their technical skills to bring their creative visions to life.”