The Body in the Machine

Sundance’s New Frontier is a stimulating space to explore how artists and technologists are collaborating to create novel narrative forms. From virtual reality to mixed reality combined with artificial intelligence developments, these immersive media makers are not only experimenting with the future of storytelling but testing new embodied spatial interfaces beyond our 2D black mirrored screens. As VR pioneer Nonny de la Peña pointed out in a panel discussion, “If the world is not flat, then why should media be?”

In the pursuit of bringing media closer to a lived world experience, the body becomes a central space of exploration. The projects Interlooped and Mica are particularly interesting to understand the emergence of hybrid spaces where human bodies are digitally rendered inside 3D virtual worlds and artificial intelligences are virtually embodied in the physical world, respectively. These “dynamics of the biodigital looping” — as New Frontier’s chief curator Shari Frilot described the program — open new interrogations of these extensions of reality.

Combining VR and live performance, Interlooped takes the viewers on a journey towards the limitless digital reproduction of themselves. This 7-minute experience created by artist Maria Guta in collaboration with tech company Imverse is simple but uncanny, playing more with perception than storytelling. Participants put on a VR headset and enter a black space, encountering a virtual rendering of María as well as their own bodies, 3D-captured in real time inside the virtual space. As the music changes, so does Maria, with whom the user interacts mainly by dancing. Soon enough, Maria calls out the participant’s name, suggesting that the artist is not only virtually there but is also being captured in real time and sharing the physical exhibit space. At that point, the time loop starts. By touching a glowing floating ball, viewers go back in time to see multiple copies of themselves from the past few minutes, creating an out-of-body experience that blurs the distinctions between the original bodies and its copies.

To generate this experience, Imverse developed a volumetric capture system based on voxels (volumetric pixels). Cameras with depth sensors are connected to software that translates bodies into voxels, rendering 3D digital replicas in real time. Although the voxel resolution is not completely realistic, the low-resolution aesthetic was used to bring the interface to the front as an active agent in the construction of the piece’s algorithmically reproduced identities.

Instead of transferring human bodies into a digital space, Magic Leap’s flagship project Mica brings an in-progress AI into the material world through the use of mixed reality (MR), a technology characterized by the placement of holograms that are responsive to their environment. Mica is a blond, short-haired, white, female-gendered digital hologram powered by the recently launched Magic Leap MR headset. What is impressive about this headset is the quality of the holograms and the built-in sensors to capture the user’s body and surrounding spatial information. This device includes computer vision that not only tracks the elements in the space but also the user’s eye movements, gestures, and speech.

The setting of Mica at Sundance conjured cyborg feminism and surrealism. Inspired by Marina Abramovic’s The Artist is Present, the AI hologram was set in a room, seated at a table waiting for the viewer to put on the headset and start the interaction. For now, Mica can only communicate through nonverbal actions, but her gestures are pretty impressive (although a bit exaggerated). In contrast to an AI assistant like Siri or Alexa, Mica does not follow orders and even gives the participant directions, such as putting a white canvas on the wall. When complete, a holographic representation of René Magritte’s painting The Treachery of Images (Ceci N’Est Pas Un Pipe) appears. Mica stands up and takes the pipe away from it, a sign that we are entering a post-representational sphere where reality is continuously shaped by digital non-human interactions.

There was something at odds about Mica’s cyborg feminist narrative beyond the fact that she reproduced the standard contemporary cyborg figure, recognizable from characters in Ghost in the Shell and Ex Machina. (This in itself raises critical questions such as: What should AI look like? Should it be human-like? Why is it regularly depicted as white and female?) But what was even more troubling (and opaque) was that at the backend of the experience a data collection system was operating to feed this work-in-progress AI with information from the participants’ bodies.

“Magic Leap will collect certain data about you, including the size and position of your pupils and iris… We will extract geometric representations of your face and body movements from the audio and visual recordings… Magic Leap and its service providers will only use the information to develop and improve Magic Leap’s products and services,” read part of the release form that participants had to sign. That makes you wonder if the user’s data is the actual product of this experience (as it is with Facebook and other platforms). In this case, it is not so much about the ghosts in the machine. Rather, it’s about how in this age of biometrics, big data, and surveillance, we should be concerned with our bodies entering the machines. How we are being digitized, datafied, commodified?

Both Interlooped and Mica raise questions about how our bodies are entering and exiting virtual spaces. New Frontier is a good space to have critical conversations on these developments as it brings together people with different backgrounds shaping not only the future of storytelling but of our lives. As Kamal Sinclair, director of New Frontier Lab Programs (and a contributing editor for Immerse), said at one of the panel discussions, “We have the opportunity to change the course of the technology now because it does not have a defined language yet.” But we need to be seriously thinking about the sociopolitical and ethical consequences of the making of a new reality.

Immerse is an initiative of the MIT Open DocLab and The Fledgling Fund, and it receives funding from Just Films | Ford Foundation and the MacArthur Foundation. IFP is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing media projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.