Into the Sensorium

An Immerse response

What’s the relationship between a media experience, the feelings it evokes, and the ways in which it might prompt action? These are the questions at the heart of this issue’s feature piece by Diana Barrett, the founder and president of the Fledgling Fund.

These are the same types of questions I’ve posed to VR makers aiming to transcend the audio and visual by tapping into other senses: touch, smell, proprioception — even the numinous sense that all living beings are connected.

The seat of the senses has been termed the “sensorium.” This includes not only sense organs such as your ears, eyes and skin, but the ability of your mind to integrate and interpret a wealth of inputs. Media theorist Marshall McLuhan posited that particular types of media distort the balance of users’ sensoria, amplifying certain senses (as this type you’re reading emphasizes the eye) and in turn changing habits of mind for entire cultures.

Do new media forms that target the full sensorium rather than one sense have the potential to rewire our relationships to one another, to social issues — and perhaps even to nature itself? Should we be moving away from a concept of media formats based on platforms (i.e., multimedia, transmedia, augmented reality) and instead conceptualize projects in terms of how many senses they engage?

Here are three conversations with makers seeking answers. Please note: these exchanges have been edited for clarity and brevity.

Egg Farm 360

I interviewed Casey Brown, the co-founder of technology and content company Condition One, in January at Sundance. Outside, a blizzard made for hard, slow and sloppy going. Inside of the Condition One installation space, the vibe was contemplative. Meditation cushions arranged in circles were interspersed with tree trunks laden with VR gear. As I sat down crosslegged, my body remembered many other attempts to meditate, mostly unsuccessful. The feel of my soggy snow boots clashed with the 360 video of an arid canyon where I was enjoined to open my heart and cultivate compassion with others practicing yoga on a ledge.

As it turns out, the meditation was designed to prepare me for a very different experience—a harrowing and unauthorized rescue mission into a factory farm where chickens are crammed into windowless corridors stacked with crates that restrict their movement.

Q: I saw Factory Farm, an earlier piece of yours, last year. Your latest piece seems to take the experience one step further. Factory Farm warned participants that it would be a difficult experience but this one makes a more conscious effort to make audiences more receptive to difficult footage and then to bring them back out of the scene and calm them down. How did you come up with that approach?

Casey: In Factory Farm, you go in with an undercover investigator to a pig slaughterhouse in Mexico. It was pretty graphic. We didn’t want to censor that — it’s the reality of where our food comes from. But we found that when people came out of the headset they were really traumatized. They looked like ghosts, some people were crying. They felt kind of lost.

So, we wanted to refine our approach for the followup. It starts with a short compassionate meditation, in a beautiful forest in Utah, with yogis by your side, and a voice prompts you to breathe and find your center. This way, you go into these intense experiences in a state of calm and then you experience the intense suffering, and sit with the suffering, but on the tail end, you go into another meditation to help bring you back down and process what you saw. It’s no less traumatic but at least it gives you some tools for bearing witness to suffering.

Q: This one was also more hopeful…

Casey: Right. At the end, you see one individual chicken who had only known intense confinement coming out and for the first time touching grass.

Q: You have powerful partnerships, such as those with Al Gore and the grassroots group Direct Action Everywhere. How do you find your partners?

Casey: When it comes to original content, we feel the most important stories to be told are those of activists. We want to bring this technology to those on the frontline of the struggle. So we have partnered with Direct Action Everywhere, whose members go into factory farms and witness the suffering of animals, ultimately exposing the truth behind our food system.

Q: Neither one of these pieces has asks inside of them. Do you have any asks along with the campaign?

Casey: We identify the organizations we work with. We are mostly interested in how the headsets can induce compassion.

Tree

The first step in experiencing the Tree installation involves placing your fingers into soil to plant a seed—which, for me, triggers another sense memory. With my feet rooted in New York at April’s Tribeca Immersive exhibit, I soon found my head transported to Peru, and my arms transformed into branches. Afterwards, I asked the project’s c0-director Winslow Porter to explain how—and why.

Q: Could you please describe your project?

Winslow: Tree is an immersive journey with a kapok tree in the Peruvian rainforest. The kapok tree is called an “emergent” tree because it rises above all of the other trees in the forest. Tree takes you through the life of a tree, a few hundred years. After witnessing the majesty of nature, the piercing of the soil, growing with the sun, all sorts of animal interactions, you witness the fate of far too many rainforest trees.

Q: In addition to VR, Tree incorporates other technologies…

Winslow: Virtual reality at the end of the day is the ability to take somebody to a different space, a different reality. It’s been happening since we’ve been telling stories as humans. So in addition to appropriating your eyes and ears, we want to be able to take advantage of other senses. We have haptics, conveying the sense of touch in the vibrations in the floor, to simulate tree growth. We have backpack and touch controllers, as well as different elements for heat and wind, and we also try to take advantage of the olfactory, by using a scent track. Even though participants know they’re not in the rain forest, part of their brain will believe it. That’s what makes VR an amazing platform for taking people to different states of being.

Q: What do you hope people will feel when they do this?

Winslow: We don’t want people to feel helpless, so we partnered with the Rainforest Alliance for a call to action at the end. People can go to our website or the Rainforest Alliance website and find out more about the problem and donate. The situation is very complex and has to do with infrastructure, hierarchies, and finances. If palm oil, livestock, and gold mining went away entirely some of these local economies would be destroyed. So it’s a question of dissecting the problem and taking a comprehensive approach. It involves understanding where things are sourced from and making sure that the people who source things are responsible.

Blackout

A few installations down at Tribeca Immersive, I hovered outside of a replica of of a New York subway car, hoping to experience the participatory VR documentary Blackout. While I never made it into this popular installation, waiting gave me a chance to speak with producer Mei-Ling Wong of production studio Scatter about the project’s technology and goals.

Q: Tell me about this project?

Mei-Ling: Blackout is an immersive documentary set inside the New York City subway. It’s a generative storytelling system. Every time you get on the train you meet a new group of passengers.

We have about 30 passengers that we have interviewed, kind of like a This American Life broadcast in VR. The story system culls from different stories, with different characters, different themes, and different points of view. Every time you get on the train it generates a different story.

Q: The project uses volumetric filmmaking. Could you explain that a little?

Mei-Ling: Volumetric filmmaking is a way of using a depth sensor to capture a 3D kind of mesh, and pairing that with a traditional film camera. …Basically, you’re trying to represent real people in real environments.

Q: So it’s like a moving camera with real people in virtual environments, moving in real time.

Mei-Ling: Yes. And it gets exported as a video file that you can put inside a video game engine. That way, you can walk around the people and experience them from all sides.

Q: Can you tell me about the narrative?

Mei-Ling: We explore what it means to be an American in the current political environment. We have all kinds of people, including undocumented people from different communities, a Trump supporter…. The New York City subway is a beautiful place to explore these stories, because it’s where you can encounter a very broad range of people in a densely populated area.

This is just the start of the production. We’ve built the train and will continue to interview people during the festival. We’re setting up a capture station on the 7th floor and will be continuing to interview more people. It’s a living, evolving documentary.

Works in progress

Each of these installations appeals to multiple senses, both with digital interfaces designed to give users a sense of immersion and bodily control, and with physical props that trigger familiar associations and postures. Technologies that may seem exotic this year—custom olfactory triggers, volumetric scans of real people—are sure to appear in more sophisticated forms that become familiar over time.

Like Diana, I’m not sure if such technologies are ready to carry the weight of a mature full-length narrative. There are still too many distractions both inside and outside to fully suspend disbelief—awareness of my own unmoving body amidst the lithe yogis, a fleeting sense that I should be able to smell the vile corridor where chickens are jammed together, the knowledge that my arms move much more actively than a tree’s branches could, a nervous moment of realizing I have left my bag unprotected in a crowded exhibit hall.

That said, the most moving—and disturbing—experience I tried at Tribeca Immersive pulled me so far in that I didn’t want to spoil it by interviewing the creator.

With Draw Me Close, Canadian playwright and director Jordan Tannahill takes you into a childhood memory through a deceptively simple black-and-white animated VR environment paired with a real set that you briefly inhabit with a live actress. Hand controllers allow you to simulate the act of drawing—your digital scrawlings appear in vivid color as you kneel and run what feels like a marker across a real piece of paper on the floor.

The potency of Draw Me Close rests in resonant, tactile gestures: walking around in stocking feet, being tucked in to a real bed up to your chin, an actual hug from the actress as the virtual mom simultaneously moves in for a squeeze. It’s a melancholy piece, reflecting one of Tannahill’s own memories of his mother as he learns that she’s been diagnosed with terminal cancer. It also triggers the sense of existential dread that comes with being small and helpless, as his threatening father arrives offscreen to shatter this small window of domestic bliss.

Is a hug a form of media? The piece begs the question. “As a theatre maker, the human body is one of my core tools. It was important to me to craft an experience that was tactile, that invited full awareness of one’s own body and the body of another. I wanted to invoke the idea that this childhood memory has come to life so vividly it can be touched and interacted with,” Tannahill told Kevin Ohannessian of Upload.

By calling forth my own childhood experiences, both good and bad, Draw Me Close affected me more deeply than any of the other VR project I’ve tried so far. This points to another threshold for both making and understanding immersive media—not just appealing to audience members’ full sensoria, but conveying what neuroscientist Christof Koch calls “qualia”—what it feels like for an individual to subjectively experience any given moment.

Transmitting subjective experiences wholesale has long been a goal and fascination for artists—from the reminiscent taste of Marcel Proust’s madeleines, to the use of the once groundbreaking and now-ubiquitous TV voiceover, to such baroque contemporary explorations of interiority as Being John Malkovich or the Wachhowskis’ epic Netflix series Sense8.

As Diana observes in her piece, “we’ve all been immersed, to various degrees, in experiences that transport us to another way of being and feeling. Sometimes this immersion can be decidedly low tech: a book, music or a traditional film. However, putting on a headset can take this feeling, this sense of departure and re-entering, to the next degree.”

By adding in other sensory clues and recall cues, Draw Me Close took the experience yet one degree further, mingling Tannahill’s simulated memory with my own in unprecedented ways.

Is this ok?

I emerged from Draw Me Close in tears, and was comforted with another hug by the actress who played my character’s mother, this time with headsets off. “That was beautiful,” I told her.

But was it? Bemused by the layers upon layers of reality and unreality to which I’d subjected myself, I meandered out of the dark, crowded exhibition hall and into the sunny spring day, wondering at my own motivations.

Perhaps the makers from Condition One could be onto something, providing a liminal experience that eases users into and out of a more disturbing one. Explaining the experience of Draw Me Close later to friends and colleagues, I was hard pressed to say how the experience struck me in the end. Was it cathartic? Voyeuristic? Traumatic? Just plain TMI?

There are no easy answers—just as we can’t say for sure how emerging forms of sensorium-focused media will move us to action. But here at Immerse, we’ll keep asking the questions.

Immerse is an initiative of Tribeca Film Institute, MIT Open DocLab and The Fledgling Fund. Learn more about our vision for the project here.