The rise of haptic storytelling
After the explosion, I hear voices calling for help.
I feel my way forward, hand over hand, touching a pile of tires, a rusty barrel, a doorframe, kicking the rubble that demarcates the barricade behind which I’ve watched a peaceful street scene descend into panic.
Stepping through the doorframe, I reel back, throat in mouth, as I find myself edging along a jagged precipice. Intellectually, I know I’m on solid ground, inside a building, no danger of falling. Still, I sidle along with the rough wall as my guide, heading for the voices: a man and a girl. I reach through what looks like a gap in the rubble, encountering a pipe the man begs me to shift—and then a warm, real human arm.
It’s too much. I rip off the VR headset and briefly find myself surrounded by concerned faces. They put the headset back on and lead me out of the experience, to a hallway bench, where I dab tears and catch my ragged breath.
The HERO installation is designed to expose untested civilians to how they might respond during a wartime crisis. Will they step up to help, freeze, hang back? At Sundance’s New Frontier exhibition, participants relinquish their nametags at the door, exchanging them for a role card that they feel best expresses their sense of self. This approach mirrors one that director Frank Oz and magician Derek DelGaudio utilized for Off Broadway show In & Of Itself. My card says, “I am a partner.” As I leave, my name badge is returned to me with a small strip of cloth resembling camouflage tied to it — both a tactile reminder and a symbol to others who have gone through it.
Still a bit shaken, I sit down with Vassiliki Khonsari, the co-founder of iNK Stories, which produced the experience in partnership with Starbreeze Studios. The cards serve as a priming device, she explains, to help shape participants’ mindsets. For example, my impulse was to help—but people who choose the “I am an observer” card might hang back. This helps to make each person’s experience a bit different.
“Once people walk into the HERO space we wanted them to strip themselves of all of the identifiers they’re associated with,” Khonsari says. “We want that to set the tone for how you enter the experience and become receptive to this moment in time. And after you have the experience it offers you the opportunity to reflect on yourself.”
In addition to the StarVR headset, which offers a wide field of view, HERO is enhanced by a haptic backpack that transmits vibrations from the explosion, real-feeling props that mirror the objects shown in the street scene, and ambient sounds rather than those piped through earphones. The combination of analog and digital inputs is unnervingly convincing—the producers call it “Vérite VR.”
“We were fortunate to push on technology, on artistry, on theatrics,” she says. “We wanted to see what VR is capable of, what mixed reality is capable of. We’re bringing together technologies that aren’t used together.”
This variety and level of sensory input is unusual and expensive — but suggests how powerful the sense of touch can be in making an immersive experience seem real.
What another demo in the same exhibition lacked in tactile immediacy, it made up for in gee whiz factor.
Billed as a “cinematic VR platform,” the Voyager chair moves a full 360 degrees and pitches 35 degrees both forward and backward. It also offers haptic feedback and spatial audio.
I strap in to experience Space Explorers, directed by Felix & Paul Studios. The chair allows the viewer to be embedded in a variety of scenes where astronauts are training for the harsh conditions of space travel: side-by-side with the driver of a rover through a snowy tundra, in the cockpit of a tiny plane, floating in a swimming pool next to trainees manipulating massive equipment in conditions that mirror zero-G.
Beautifully filmed, and full of aspirational interviews about humanity’s future in space, the piece is tweaked for maximum awe. I find that the chair requires some suspension of disbelief, which I momentarily achieve during the experience of underwater weightlessness.
A bit dazed from all this—plus an earlier room-size VR installation that had me dancing with a mix of virtual giants, lilliputians, and actual humans represented by digital avatars—I wander through driving snow to the next exhibit space. The real world feels unreal, I explain to my faraway husband through my earbuds. I feel the chilly flakes landing on my face, the slush beneath my boot treads, but for a moment it seems distant, manufactured.
Snow shaken off, I wander around the next venue and settle down at a table, where I’m prompted to put my hand palm up into a rubber tube connected to a metal box, topped by a screen. Created by a company called HaptX, the device is designed to transmit not just touch, but sensations of hot and cold. Using a wand, I direct a spider to prance around, and then a tiny deer to stomp on my palm. A few moments later, a virtual dragon fries it to a crisp, and I’m tempted to pull away and shake my hand as heat washes over it.
“Touch is a very visceral sense,” says Jake Rubin, HaptX CEO and founder. “So if you’re trying to tell a story being able to touch or drag something can bring more emotional engagement or more realism.”
He explains that the effect is achieved through a “smart textile,” infused with channels and air pockets that change shape to create various sensations. Piping hot or cold fluids through conveys temperature shifts. Like pixels, the points of contact need to be smaller to create a higher-resolution tactile experience.
HaptX is not currently geared towards a consumer market—the company’s customers are industrial, medical, and military. They need high-fidelity simulators for training purposes. Photogrammetry can help in mapping the detailed surfaces of an environment such as a fighter jet cockpit, he says, which in turn can help to convey the feeling of complex physical interactions such as buttons, dials, or switches.
Rubin suggests that theme parks or arcades might be the first places users will experience this type of technology in an entertainment setting, probably within three to five years. “The price should come down with volume,” he says, which will make the technology more available to media makers.
What’s coming next down the pike for haptic experiences? Some we’ve already grown used to—such as Apple’s “taptic engine,” which mimics the feel of buttons, taps, and simulated heartbeats that lovers can share across distance. Apple’s Taptic API now allows developers to create apps with their own haptic features. Here’s an interesting breakdown of the different types of feedback that are available, and an X-ray of the engine in action.
Select arcade games have featured haptic feedback for many years, such as racing simulators. For the masochist, newer haptic game accessories can simulate the pain of a kick in the ribs, a punch in the face or a “yelp-inducing pinch,” reports Zara Stone in Wired.
So-called “teledildonics” have also been a focal point in the development of haptic technology. David Parisi, an associate professor of emerging media at the College of Charleston, offers a capsule history in his new book, Archaeologies of Touch: Interfacing with Haptics from Electricity to Computing.
He details the rise and fall of the RealTouch system, which synched up the action in pornographic films to a USB-enabled sheath. Somewhat surreally, Parisi explains, “adding haptics to the videos required the coders to move scene-by-scene through the library, imagining what the scene would feel like on their own genitals, and then assigning that complex of sensations a string of numerical values to control the operation of the motors, the release of the lubricant, and the temperature of the device’s interior.”
By 2012, the RealTouch allowed for remote manipulation of the device, shifting it from a media-driven experience to one managed by a human operator. This continued what Parisi describes as an ongoing effort to naturalize such devices and sensations. This new configuration aimed to “erase the distance between the desiring male body and the females and males captured in [the Adult Entertainment Broadcast Network’s] vast archive.” While the RealTouch eventually failed due to patent disputes, its evolution suggests the trajectory of touch technology—towards the ever-more authentic and multi-dimensional.
Through the looking glass, what happens when devices designed initially for touch begin telling stories of their own? Parisi notes that more recent improvements in sex toys have been aimed at women. The latest wrinkle is making them more networked.
“White hat hackers have already exposed a number of adult companies — Lovense, WeVibe — as unstable repositories for the surprisingly detailed stores of intimate user data they’ve been collecting, mostly unbeknownst to their customers,” reports Claire Lampen for Gizmodo. On the plus side, after your wired vibrator’s been hacked, perhaps it can order you a pizza to help you eat your feelings.
“2018 is shaping up to be the year of haptic feedback,” reports Kyle Melnick for VR Scout. Microsoft’s new CLAW allows users to grasp, touch, and feel objects by simulating the tension of squeezing something hard or soft and allowing the user to feel its contours and surfaces. Of course, it also allows users to pull a trigger, offering the simulated feel of gun recoil.
But no single gadget will suffice to move the field forward, Parisi suggests. “For a high-fidelity haptics system to enjoy widespread adoption akin to radio or television, it will have to function properly in its stated goal of delivering convincing touch sensations, no easy task in and of itself. But in order to justify the development of a shared cross-platform programming language, it will also have to thrive in the competitive marketplace of consumer electronics. Haptics will need to find a killer app — that one activity that robust, high-fidelity haptics would prove indispensable to.”
Can such devices ever deliver the shock of recognition I experienced during HERO—of undeniable human connection? Even if so, can they effectively advance a storyline? That remains to be felt. But, for now, the boundaries between the real and the simulated just keep on shrinking.
Immerse is an initiative of Tribeca Film Institute, MIT Open DocLab and The Fledgling Fund. Learn more about our vision for the project here.