An Ambitious Vision for the Virtual World

At a time when bitter conflict seems to prevail over civil discourse, might virtual adventures make it easier for people to see eye to eye? That’s the hope of Çağrı Hakan Zaman SM ’14, PhD ’20, who believes that his work designing and disseminating new media technologies could help illuminate tough topics and foster common ground.

“Bringing people together is very difficult,” says Zaman. “By expanding their perceptual, cognitive, and imaginary worlds, we can enable them to experience things from someone else’s perspective and forge a mutual point of view.” This is one of the objectives Zaman is pursuing in the Department of Architecture, where he conducts crucial research with MIT’s multi-department Transmedia Storytelling Initiative (TSI). Launched in 2019 by the School of Architecture and Planning (SA+P) working closely with faculty in the MIT School of Humanities, Arts, and Social Sciences (SHASS) and others across the Institute, TSI aims to open up new avenues of research focused on the applications, impacts, and opportunities of emerging interactive media tools. The initiative, under the direction of Caroline A. Jones, professor and associate dean of SA+P, carries forward some of the energies from the school’s film program from the ’60s through the ’80s. “We are makers but also work to theorize and alter the media tools on offer,” Jones says.

Zaman, an MIT lecturer in architecture and director of the MIT Virtual Experience Design Lab, collaborates with students and faculty partners from across MIT, investigating the use of immersive media to enlighten, unsettle, and open minds. With interactive, 3-D digital experiences that interweave audio, video, and still photography, his lab takes up such issues as ethnic conflict and racial violence.

Zaman puts his digital media tools to use in a wider range of applications as well. He analyzes and improves architectural spaces in the built world, creates spatial simulations to assist visually impaired people, and produces scientific visualizations of processes normally invisible to the human eye. But for Zaman, these diverse ventures share a common theme: “The tools we are innovating can potentially have a huge impact on how people access information and on how they comprehend what’s going on in the world,” he says. “Eventually they could promote a more humane society.”

Spatial Storytelling

Fundamental to TSI’s mission, and Zaman’s research, is a commitment to the narrative form. “My mentor Patrick Winston [an alumnus, professor, and former director of the MIT Computer Science and Artificial Intelligence Laboratory, who died in 2019] believed human intelligence is all about storytelling,” says Zaman. “When stories tap into our perceptual selves, we can become much more engaged, even where difficult issues are involved.”

In 2017, while still a graduate student, Zaman created “September 1955,” an interactive installation about a pogrom waged against Greeks and other minorities in Istanbul, Turkey. Integrating a library of archival photos of the episode and first-person audio accounts, Zaman reconstructed a spatially accurate photography studio from the period. Users wearing virtual reality headsets could walk around a room to explore and activate different scenes, gaining the feeling that they were bearing witness to events as they unfolded.

“When data becomes tangible, it speaks to you,” says Zaman. “Being in a place similar to an everyday lived experience is a more powerful way of conveying information than simply telling things.”

This is because human memory is spatially wired, says Zaman, an architect by training. “When we create a story in space, we’re not just conveying the story better but allowing it to stick with people much longer.”

Zaman continues to plumb the potential of storytelling to alter perceptions, shift perspectives, and perhaps change beliefs through his teaching—work that supports the TSI’s main goals: to create new partnerships among faculty across schools and offer pioneering pedagogy to students.

Jones, who directs TSI with help from steering committee members from across the Institute, says, “Çağrı Zaman is truly breaking new ground in innovating tools and approaches, such as opening digital documentary media for archival scrutiny, using machine learning to reconstruct locations for fiction films, and helping our team fundraise for student support. His research, teaching, and mentoring give MIT students new ways of engaging with urgent social issues through spatialized storytelling. That’s what TSI is all about.”

For example, a fall 2020 class, co-taught by Zaman and D. Fox Harrell, a professor in both the Computer Science and Artificial Intelligence Laboratory and Comparative Media Studies/Writing in SHASS, yielded an interactive app to tell the nonfiction story of Occupy City Hall, a 2020 Black Lives Matter protest in New York City triggered by George Floyd’s murder.

A quartet of graduate students (Megan Prakash ’18, Wonki Kang, Wuyahuang Li SM ’21, and Kwan Queenie Li) developed a cellphone-based, augmented reality interface that could be projected onto the surfaces of a virtual reality living room, where strings of different colors connected to audiovisual objects on walls and furniture. By following these strings and activating the objects (to play video interviews, for instance), users could identify themes from the Occupy tent city: police brutality, media coverage, racial inequity, and artistic expression. The team’s intent was to create a digital bridge between a real-world experience comprising a multiplicity of perspectives and physically distanced observers seeking intimate access to that experience.

“This could be a new form of journalism, where you move beyond the mainstream discussion that looks at a place and what happened there,” says Zaman. The students have published a paper based on this interactive method and are presenting their work at conferences. “We want to get students interested in telling stories using these media and get them as skilled as we can so they can move out into different venues.”

Discipline-Spanning Applications

Zaman sees a growing role for immersive media in liberal arts curricula and notes that the TSI is spearheading MIT efforts to expand this work worldwide. One recent project he supported, “Digital Archaeology, Virtual Narratives: The Case of Lifta,” brought together biblical archaeologists from Ben Gurion University and MIT architecture faculty and students to create a digital tapestry of a Palestinian village notable for its ancient archaeological and architectural remains. By representing the richly varied history of a now politically contested village, the exhibition aimed to offer a more complex view of the site.

In a new venture, “The Latent Archive,” Zaman will exploit artificial intelligence to repair rare and delicate historical moving images. Zaman’s media tools are also proving useful in science and engineering. He collaborated with Admir Masic, assistant professor in the Department of Civil and Environmental Engineering, on an exhibition that used augmented reality tools to show the properties of the first artificial pigment, Egyptian blue.

As he continues to widen the scope of his research and the range of TSI’s collaborations, Zaman envisions immersive media techniques evolving into a fundamental platform for teaching and scholarship in virtually all fields, with a new generation of digital storytelling tools providing a pathway for addressing humankind’s most complex challenges, whether racism or climate change. “We can tackle these issues because our technology can squeeze larger periods of time into seconds, and great distances into small spaces,” he says. “These experiences can expose more people to different angles of a problem and allow us to engage in constructive conversation with each other.”

This story originally appeared in the Fall 2021 issue of MIT Spectrum. 

Photo: Courtesy of Çağrı Hakan Zaman.