Revealing Reality in Imaginary 3D Worlds

Revealing Reality in Imaginary 3D Worlds — Part 2

This is the second part in a series covering featured highlights of Venice Immersive. You can find the first part here.

A city on water with floating buildings, bridges and gondolas, and canals for thoroughfares, it’s fitting that Venice is home to Venice Immersive, a magnificent display of 3D worlds explored through VR headsets. In VR, you find yourself transported to a colorful array of worlds, from animated Ancient Egyptian tombs to fantasy worlds like Gumball Lounge. By yourself or with others, your job is to explore these worlds and find the stories. Some worlds are completely fantastical while others are simulated real worlds created using data capture techniques such as photogrammetry or lidar scanning. All are created in game engines and use a conglomeration of languages and grammars from video games, immersive theater, documentary, film, dance and/or architecture among others.

At this year’s festival, one could see new languages and grammars bubbling up at the intersection of these fields. Using real time 3D game engines, stories move forward with techniques such as treasure hunts, live actors and virtual scenes mapped onto physical sets that participants wander through; the scene changing depending on where the participant moves their hands and eyes. Documentaries at Venice Immersive ranged from the photo real to real time 3D animations.

I sat down with two artists who created imaginary worlds and characters to engage audiences with pressing real world issues. Participants had to discover the non-fiction content by making their way through the world and learning its language. What follows is part two of a three-part series highlighting each of these artists and the worlds they built.

All Unsaved Progress Will Be Lost (2022) by Melanie Courtinat

Melanie Courtinat, an artistic director for brands by day and an immersive artist by night, is a creative particularly drawn to horror and themes of loss. She shares, “Every artist has their obsessions. I really think that melancholy [and] loneliness are themes that speak to me.”

With works such as I Never Promised You a Garden (2017) and Trouble (2022), Courtinat creates speculative and virtual worlds using VR/AR and game engines and likes to experiment with and push the boundaries of these technologies’ expressive capabilities.

One of her experiments, All Unsaved Progress Will Be Lost, premiered at Venice Immersive in September 2022. She created the virtual reality project in a few weeks by herself with no funds. The project tells the story of an unknown narrator who refuses to evacuate their hometown after an environmental catastrophe. The story unfolds through text that appears around you as the experience propels you forward through a landscape that is at once beautiful and stark. Haunting music plays in the background, and the world that she creates is a ghost town. The few animals you see are silent. Based on a true event, all of the text is taken from an archive of interviews with people who experienced an environmental disaster in their village and presented as one narrator’s story. In the spirit of allowing audience members to experience the story with their own eyes, the artist asked that the focus of the archive not be revealed in this article.

“The story is based on real-life events, but I imagined this character. There’s no information about their gender in the story, but for me it’s an old woman. She’s made of thousands of pieces of text, of different testimonies… I didn’t want it to have actual characters. I think it was interesting for me, for the spectator, to be able to project… In the beginning, the narrator is talking about the threat of poison in the air and water that humans cannot see, and some people interpret this as a monster or an alien, and I really like them to keep guessing and keep confronting themselves with their own personal fears. And then, in the end, finding out that it actually happened.”

As you move through, your eyes follow written quotes that appear sentence by sentence, originating from but not narrated by the woman of the village. The words hover in the foggy sky over the grass, as do a few flowers and animals below you. In front of you, a big cement structure looms. The stark landscape and simple texts makes the experience feel eerie and ominous. Behind you, you see the trail of texts that you read, hanging on in the heavy air like the player character hangs on to her home knowing that all will be eventually lost — her home, her village, her life. Although the world Courtinat created is minimalist, it’s evocative.

“It wasn’t a project created because of the story. The story came second. It first came with the sentiment, the emotion.”

The emotion is one of fear and loss. I imagined what it would be like if someone knocked on my door and told me to leave immediately — how I would feel abandoning my home. Or how I would feel if I stayed knowing that toxins surrounded me and eventually would kill everything in their wake. I felt for this woman who refused to go, the pain of loss too great to bear, the anger keeping her there.

During the pandemic, Courtinat started to read voraciously and eventually stumbled upon a book of interviews buried in her book shelf that her mother gave her when she was a teenager, about a catastrophic event that destroyed people’s lives. It deeply disturbed and stuck with her years later, as she started crafting a VR experience based on the book. As an avid gamer, she approached the adaptation as a game designer rather than a documentarian.

“I don’t think in terms of frames, like, you have a little character here and then something big on the left. I don’t know much about it. My personal background is in video games, so I’ve always created worlds thinking in 360 degrees. So when you use a game engine to create, it’s like doing little models. You’re going to do the terrain first, make it uneven and then you’re going to paint your rock or grass textures. And then you’re going to put a little tree here and place everything you want. Then you’re going to light it and perhaps add fog, I personally like it heavy. And only in the end, . . . I will go to first person mode and wander in this world I just created and find the perfect point of view. And I will place the camera here. And that’s the way of creating that I really like. It’s super fun, to be honest.

The title for this piece additionally comes from an in-game statement that appeared when she started playing games in her youth. She elaborates further:

The title All Unsaved Progress Will Be Lost is like a sentence you could read when you were playing video games, like in my first RPGs (Role Playing Games) … At this time, the gameplay mechanism of saving the game wasn’t as immediate as nowadays. You really had to go to some specific place, manually save it, and then you had this kind of message, “Are you sure you want to quit? All unsaved progress would be lost.” It’s such a weird sentence if you think about it. It’s so bizarrely grave and intense in a misplaced moment. You’re just having fun thinking about nothing else and just enjoying the game. And so, reading this sentence would strike me and make me feel a very unique, very peculiar emotion of losing something safe. And to be honest, this project was created because I was wanting to give the spectator this exact feeling of a “lost heaven.” Paradis perdu we say in French.

Since first introduced to creating in VR, Courtinat has been hooked, with All Unsaved Progress Will Be Lost being one, but certainly not the last, of her triumphs in emergent tech. Reflecting on her own creative process in this medium, she loves the feeling of being in another world and the “intuitive” feeling of using game engines to tell stories. She thought, “Okay, this is what I want to do.” And ever since, she has been creating virtual worlds and augmented experiences that haunt, provoke and draw attention to what can be lost.

For more news, discourse, and resources on immersive and emerging forms of nonfiction media, sign up for our monthly newsletter.

Immerse is an initiative of the MIT Open DocLab and Dot Connector Studio, and receives funding from Just Films | Ford Foundation, the MacArthur Foundation, and the National Endowment for the Arts. The Gotham Film & Media Institute is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing emerging nonfiction projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.