In many ways, AR still feels like it’s in its infancy. But some artists are gradually exploring its possibilities.
Experimenting with virtual reality
(VR) in film production — (Image courtesy of Framestock)
Over the past decade, extended reality (XR) has gone from a specialized niche to a thriving scene with many more users and creators. But much of the critical and popular attention for these projects have gone to virtual reality, while augmented reality (AR) is comparatively ignored. Let’s take a closer look.
The difference between the two forms is that virtual reality (VR) involves complete immersion in a created environment, while AR layers virtual elements onto the real world. Professors Elizabeth Miller and Patricia R. Zimmermann describe the distinction as VR being “place-agnostic,” versus AR (or “augmented documentary,” as they put it) being “place-centric.” In many ways, the latter still feels like it’s in its infancy, and this gap is particularly pronounced in the realm of documentary.
Within fiction AR there are at least a few breakout hits like Pokémon GO, while there’s no comparable blockbuster project for documentary AR.
Surveying existing AR docs, a certain paucity of imagination becomes evident. Nearly all of them are some variation on historical education projects, often produced by or in conjunction with museums and similar institutions — what Sue Ding calls “location-based media” or “participatory documentary.” They follow a similar format: using the relevant device (almost always a smartphone), a viewer can survey a specific real-world location with options to engage some kind of text, video, or audio-based piece of exposition or mini-experience.
For example, the Chicago 00 Project grants access to materials from the Chicago History Museum at various areas of the city. Emerging Radiance uses Spark AR and Instagram filters to layer additional elements onto portraits of Japanese-American farmers who lived and worked in Bellevue, Washington, before World War II. The Japanese American National Museum’s app BeHere / 1942 lets visitors “witness” the forced removal of interned Japanese Americans after Pearl Harbor; looking through their phone camera, one can watch reenactments of the event layered onto the space outside the museum. The Freedom Fighter app lets one view an animated rendering of civil rights leader Dr. Lillie May Carroll Jackson talking about her life and activism at locations around Baltimore. More of these experiences exist for everything from the sites of Spain’s Camino de Santiago pilgrimage to San Jose’s Japantown.
Befitting their educational origins, these projects feel very much like extensions of the traditional informational placards which accompany museum exhibits and artifacts. Rather than meaningfully transforming one’s understanding of a site, they tend to simply add more information to absorb. Often, they require more passive activity than XR usually entails — holding up one’s phone to watch or read, or having it at your side to listen. Sometimes they seem simply to exist out of convenience, an alternative to erecting physical installations on a site.
Notably, many of these apps are designed as methods of historical corrective, highlighting the experiences of groups whose stories have traditionally been ignored by mainstream educational curricula and institutions. This use seems paired to the idea of XR being an “empathy machine.” However, that concept requires a more immersive and evocative experience. The assumption that audiovisual materials beyond standard text have more perceived power and can draw in an audience presumed to otherwise have less interest in these stories is fraught, and one that could easily step over the line of good taste. Already there are Holocaust-based AR experiences that toe the line of historical misery rubbernecking.
Looking at the history of writing about AR over the last 20 years, scholars repeatedly speak to the “promise” it holds, without always being able to cite many concrete examples of work that demonstrate that promise. That’s shifting somewhat. Comparing two articles addressing AR at Documentary.org from 2018 by Sue Ding and then 2022 by Bedatri Choudhury, the newer one can at least point to more projects.
This isn’t to suggest that AR doesn’t hold great promise. This is perhaps more a symptom of the form still being in its infancy and something we will explore as we develop more expansive technology and ways of interacting with the medium. Some AR documentary projects do more to engage users directly, rather than passively pushing them “through” their surroundings.
The NYC-based edtech company Movers and Shakers is devoted to these kinds of projects, with their app Kinfolk allowing one to digitally place monuments to Black, Brown, Indigenous, and/or queer historical figures onto real spaces, combining education and documentary with activist reclamation of public space. Miller and Zimmermann cite, among other projects, Tamiko Thiel’s 2018 piece “Unexpected Growth,” which evoked the threat of ocean pollution by animating a digital tumor on the Whitney Museum that would evolve and spread in response to both weather conditions and the gaze of visitors. Nancy Baker’s Mushroom Cloud NYC / RISE projects a nuclear explosion over New York’s skyline, an intimidating and visceral evocation of the danger of climate change.
There’s also some interesting potential in the AR features of social media apps. As Chase DiBenedetto explains, businesswoman Kristen Cuneo used her husband’s TikTok account to share her experiences as a new mother. Her series of videos featured heavily quantified information she collected around feedings, diaper changes, and the like — all of which her coworkers could interact with in real time by scanning embedded QR codes.
Gradually but surely, we are seeing this format blossom, as creators get a grasp on the possibilities offered by interactive elements. One key appears to be recognizing that a “place-centric” medium can be explored in ways beyond merely adding digital facets to a location that could just as easily be physical ones. Kinfolk, Baker, and Thiel’s projects actively transform public spaces. Cuneo’s social sharing invites participation from users at their own pace. Pokemon GO became hugely popular by letting players reimagine the mundane outdoors as a science fiction hunting ground. One could imagine the documentary form similarly taking advantage of augmented reality.
For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.
Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.