Nonny de la Peña reflects on her experience as a VR pioneer.
Nonny De La Peña is the founder and CEO of Emblematic Group, a digital media company focused on immersive virtual, mixed, and augmented reality. She is widely credited with helping create the genre of immersive journalism. De la Peña was selected by Wired as a #MakeTechHuman Agent of Change and has been called the “Godmother of Virtual Reality” by Forbes, Engadget, and The Guardian. Additionally, Fast Company listed de la Peña as one of the “13 People Who Made the World More Creative.”
A former correspondent for Newsweek magazine, writer for The New York Times, and other major news organizations, de la Peña’s start in virtual reality began in 2006.
In early January, Nonny de la Peña spoke with Immerse about her latest projects and where she thinks immersive nonfiction media is headed. In May, she added some thoughts that take the pandemic into account. Here are a few highlights from the discussion.
Before I created the virtual Gitmo project with digital artist Peggy Weil, I had directed a documentary film called Unconstitutional. It included a segment on Guantanamo Bay prison and my research made it clear that waterboarding should be called “torture” instead of what was being used by the press as the term du jour, “enhanced interrogation.” I got a lot of grief for using the word “torture,” and I was accused by my colleagues of practicing advocacy journalism. It turned out that reporters were being co-opted to use the term “waterboarding” by the White House, which was a shift away from a nearly century-long practice of calling it torture.
We have to acknowledge our vulnerabilities to manipulation. And that’s really important when we start thinking about the power of virtual and augmented reality — and not just deceptively accurate digital characters. What can we learn from these important historical moments in which journalists are co-opted? And how do we reflect on what will be the technological equivalent in the future? The power of virtual reality demands that we journalists are even more cautious and responsible in sourcing our reporting.
A lot of people talk about subjective experience as being key to understanding the fraught ethical implications potentially present in immersive journalism. But I think that what we’re not contending with is something that has much greater implications: How do we understand best practices with fully embodied content? What does it mean when people are going to be able to step over the bodies which have been captured as accurately volumetrically as video is today? What’s going to be our Vietnam-era moment of what gets “broadcast” to our audiences?
Pioneering Immersive Journalism — the challenges
My ideas aren’t as controversial as they used to be. But when I first started creating pieces, it got pretty rough. My colleagues would say, with their fingers literally wagging in my face something like, “You can’t do that. It will never work.”
It led to real financial despair for my family when the director of the journalism school [at USC] retired and the Dean decided not to renew my fellowship. I was stuck with no job during the great recession and it was almost impossible to find work. …. Credit cards were maxed out, I had borrowed money from family friends. One night after feeding my kids, I had nothing for myself. Who could have thought that I would find myself in that position?
And it got to a point where I found myself saying, “I’ve got to stop. I’ve got to forget about this crazy idea.” But my husband kept saying, “Stick with it, stick with it, stick with it.”
Immersive journalism has now become an acceptable form of storytelling. And that was one of the things that I got pushback about for a pretty long time because 360 video was not something people could easily make. Also, volumetric content was so hard and expensive to produce and many felt that the computer-generated (CG) elements made it a nonstarter, since when I began we couldn’t capture in any photo-real way. Today, photogrammetry and videogrammetry solve that problem. And while it still is more expensive to produce, the tools to create and publish content have become much more accessible.
Across the
Line VR puts the participant in the shoes of a patient confronted with pro-lifers as she enters a
health center for an abortion.Still, from early on until today, I was always trying to work in an embodied form, one that allows the viewer to fully move around inside a story. Moreover, as we began to study the viewers’ reactions, we found that if people could walk around in the scene made with computer-generated imagery versus looking at a 360 video, their experience was profoundly different.
The hardware has also undergone a major advance that has validated my long-held conviction that embodied movement was going to be key to the growth of virtual reality. We saw the Oculus Quest sell out in the 2019 holiday shopping season and they have been sold out every time they go back on the market. This didn’t happen with previous Oculus headsets. Why? Because the low-end headsets only offered 360-video, which wasn’t the kind of experience that was going to lead to a paradigm shift.
The hardware has also undergone a major advance that has validated my long-held conviction that embodied movement was going to be key to the growth of virtual reality.
When the Oculus Rift came out, it just sat you in the chair. But younger audiences are used to moving around in digital space. I didn’t want to make or experience content sitting in a chair. We needed goggles that could let people move through space and that required hundred-thousand-dollar tracking equipment. I was invited to present Project Syria at the World Economic Forum, but I had no goggles and no support or funding for making goggles. We had to figure out a way to 3D–print our own headsets based on the earlier prototypes out of USC’s ICT MxR lab, when Palmer Luckey and other very talented folks were there: Mark Bolas, John Brennan, Bradley Newman, Thai Phan, Evan Suma, etc... I found tracking equipment in a bankruptcy sale and bought it for seven thousand dollars. And then we started 3D–printing our headsets. When the Vive came out, it changed everything for me. I no longer had to figure out how to make my equipment. And now the Quest, with inside out tracking, is absolutely fantastic. No computer necessary!
Building the Reach platform
I had been trying to pitch the idea of REACH.Love [a creation and distribution platform] from the very start, about a year into having established Emblematic.
I tried to raise money for a distribution and creation platform. Slowly but surely… we got another grant from the Knight Foundation. Thank you, Knight Foundation! They’ve been huge supporters and were able to make an alpha version available to the public. But I was unable to get investment to move it to a more self-sustaining platform.
Emblematic
Group created Reach.love in partnership with Mozilla to make volumetric VR production more
accessible for storytellers. The platform allows users to operate a simple drag-and-drop web
interface to place scans of people and objects into high-resolution 3D environments.A vision for volumetric media
Something has dogged me since I first conceived of immersive journalism: How was I going to make my ideas visible to everybody else? Unfortunately, people couldn’t “see” it unless I made it, REACH included.
One thing that I’ve slowly come to accept is that I can see volumetrically in a way most people can’t yet. That gives me an edge as a creative person and as a thinker. So, at this point, I’ve learned how to run a company for better or for worse. Last year was brutal; we lost half the team. Now, fortunately, we have some really interesting projects going and things are turning around. With Covid-19, our skills are even more valued.
Recently, The New York Times did a piece on the New Zealand volcano eruption and they embedded a glTF model of a volcano in the website. The model lets you see the path of lava flow, the trails, and where people were. I could probably find a talk from 2014 where I was saying that this was the future, the way journalism was going to work. And the ability to embed, that’s the thing that REACH.Love does so nicely. I can now teach any news organization how to do that very inexpensively. But it is an Alpha and still needs a bit of investment to make it robust.
Our world has volume. It’s got dimension. And the shift from content being flat to dimension, to me, is a complete no-brainer. What’s important here is how is it going to be used for press, non-gaming content, news, and entertainment?
Our world has volume. It’s got dimension. And the shift from content being flat to dimension, to me, is a complete no-brainer
I have this obsession with the Joan of Arc Project. That was a black-and-white silent film, made in 1928 by Carl Theodor Dreyer, which established the closeup. I have an obsession to remake that volumetrically. I introduced a lot of people to VR and one of them happened to be the film director Alejandro Gonzalez Inarritu. He agreed to come on as an executive producer. But I need to find the bandwidth to go out and raise the money for it.
Future speculation
I think haptics are going to get very interesting. There are a lot of frequencies we can’t hear that make us feel. At some point, you won’t need to wear anything cumbersome. You’ll be able to use sound fields to experience the touch of things.
There are a lot of frequencies we can’t hear that make us feel.
I think we’re already seeing a lot of people putting energy into making glasses and that work may become the AR/VR solution.
I also think that spatial data will integrate into the home. I think that you’ll probably have some holographic light fields, displays in the home that everyone can see at once for shared information.
Sexism in tech
I had a conversation recently with a woman who was fighting hard for, but not getting credit for her work. And we started talking about sexism in our industry. I was angry about how we don’t receive investments. In 2019 only 2.8% of venture money went to women! Forget it if you are a person of color.
The relentless stress of whether or not I will be able to cover payroll for myself and my team means I can’t really push Emblematic or REACH to the next level. The dudes get investment with way less experience and notable successes than I have.
But I have had to let that anger go. If I don’t find a way to be positive, it will suck me down.
I am grateful for what really is a privileged position. Who gets to make huge 6DoF* experiences about Lyme disease? Who gets to try to envision and pioneer incredible pieces intended to help get the world to focus on issues that matter? To create ways for people to deeply connect and care about others’ stories? Who gets to imagine a future where our content is not flat and then lead a team to create a toolset and publishing platform that shows the world a glimpse of that future?
As much as possible, I try to stay grateful. No doubt, there will be days where I will need to grouse, but, especially with Covid-19, embracing the joy of life is ultimately more powerful.
*6dof stands for ‘six degrees of freedom’ and refers to the ways you can move within and interact with your virtual reality environment. There are three translational degrees of freedom, and three rotational degrees of freedom, one of each for each axis — x, y, and z.
Immerse is an initiative of the MIT Open DocLab and The Fledgling Fund, and it receives funding from Just Films | Ford Foundation and the MacArthur Foundation. IFP is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing media projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.