Hacking SparkAR and Lens Studio as AR tools that go beyond the typical selfie filter
This article has been researched and developed at the MIT Open DocLab with Alex Herrero, a fellow from the MIT Summer Research Program.
Throughout the past decade, extensive scholarship, documentaries, and experimental pieces in multiplayer worlds such as Second Life interrogated the fluid boundaries between virtual and real worlds. Since games like PokemonGO revolutionized location-based AR in 2016 and VRChat sparked a major VR-maker community in 2018, popular awareness of virtual worlds have grown alongside the powerful social communities co-created by its users.
With major social media giants scaling XR, we are now approaching a new era of augmented social space. Facebook is in its beta stage of Facebook Horizon, a massive-multiplayer VR world, while Snapchat and Instagram are offering new platforms that make Augmented Reality filters and machine-learning features increasingly accessible. On one end of the spectrum, this reads as a nightmare for user agency, with our personal data and online social spaces increasingly co-opted as a marketplace for advertisers. On the other end, social XR bypasses some issues of accessibility that have marked the XR industry. How should nonfiction storytellers reconceptualize these changes to best use platforms for their own creative ends?
One potential way forward is within AR, where Lens Studio and SparkAR have been embraced by the creative community since becoming publicly available last year. AR allows people with no coding background to create filters and share them on Instagram and Snapchat for anyone to use, becoming a household tool overnight. While many of the use cases remain surface-level interventions, such as hamster faces, cosmetic filters, and branding schemes, there’s also unprecedented user-generated creativity.
Since 2010, artists have used AR as a political intervention, like in AR Occupy Wall Street, as well as critical “art invasions” at institutions such as MoMA. In tandem, we’ve seen a growing tradition of making public AR art more accessible, such as Nancy Baker Cahill’s work with 4th wall and the growth of AR installations with unique sponsorships by both private and public institutions. With Lens Studio and Spark AR, AR creation and distribution merge in social media, requiring creators to recalibrate the landscape altogether.
Here are some examples, strategies, and ways forward for thinking about Social AR for documentary storytelling:
Telling stories with the users
In social AR, the story is not about the specific filter but the user-generated content it provokes. For instance, an Instagram filter that simulates an aesthetic of analog film layers the emotion of nostalgia to visual culture to affect the lens from which we see things, perhaps creating “virtual worlds” of its own. AR takes things further by allowing interaction not only with the image, but more crucially with the user’s movements and their environment.
OldSkool
Instagram Lens (photo by Bram
van de Ven)For political advocacy campaigns, AR creators can utilize the virality of interactivity as a tool to engage communities at a mass scale. In 2017, the UN and creative agency VML released a campaign for World Humanitarian Day, in which SparkAR was used for individuals to read aloud the stories of those affected by conflict. More recently, many activists took to AR to push forward the Black Lives Matter movement, creating filters that overlay streets and gamify donations.
Such techniques, of course, run the risk of relying on proprietary software to affect change and in many ways use the same tactics as branded content. Coupled with other avenues for deeper, sustained engagement, however, such campaigns are a good starting point to mobilize direct action. This new form of user-generated content will be important to consider for future forms of nonfiction storytelling.
Hacking the app for unconventional uses
Homeless
Realities by JOVRNALISMThe documentary tradition is marked by creators “hijacking” conventional tools for innovative purposes. If creators aren’t too keen on orchestrating mass media campaigns, other strategic uses of social AR can offer opportunities for meaningful storytelling. In Homeless Realities, users use Snapchat to overlay objects like a 3D-captured tent onto their own physical environment, while hearing the stories of homeless individuals. To do this, the creators needed to adapt the size of their 3D assets to fall within Snapchat’s specs but in return were able to share the project without requiring viewers to download a custom app.
Similarly, Grey Matter AR describes itself as a documentary “hiding” inside of Snapchat. This experience shares stories of 10 senior citizens in Canada. The collection of 50 15-second scenes brings younger generations who use Snapchat in dialogue with their elders. Such projects gesture towards the possibility of subvert a culture of flower crowns filters. As artist Karen Vanderborght expresses in this CBC blog, “That’s my artist practice, basically, to use tools and then tell stories with them, even if they’re not necessarily developed specifically for that. It’s a way of dropping in what I hope is meaningful content in a very superficial sphere.”
Another notable example is how National Geographic used SparkAR to project climate change data while visualizing the globe in 2070. Part of their April Earth Day issue, this project is an intervention in the social media space while also connecting users to a digital issue of the magazine.
Build community with XR creators and enthusiasts, generating interest in new skills
For many who aren’t already seeped in the XR industry, online creative communities have been essential to maintaining an artistic practice. For instance, the SparkAR Community Group has over 75,000 users who frequently share artworks and tips with one another. Some creators report this community as a useful entry-point into AR and a major reason to pursue the medium. Moreover, SparkAR and Lens Studio offer creators access to statistics for their AR projects, such as the amount of views, shares, and the number of pictures and videos taken with the effect. When creators see their filters reaching millions of users and the resultant creativity, this activity can in turn inspire new works.
Looking forward to independent and democratized distribution
SparkAR and Lens Studio require an approval process from Facebook and Snapchat’s content moderators, running risks of moderation biases that limit creators’ agency. Recently, SparkAR publishing has been delayed because COVID-19 halted the ability to make site-specific content approvals on Facebook’s campuses. Moreover, while these platforms make statistics available to the creators, the implications of how Facebook collects big data in their AR tools are up for question.
While riding the hype of social AR, it’s important to recognize that browser-based AR can obliterate the need for app-based AR, replacing SparkAR and Lens Studio. Current developments such as AR.js, an open-source framework, and 8th wall, an emerging software, gesture towards this future but still are in nascent stages. WebAR will allow even more creative liberty through an open-source infrastructure, and allow people to share content as long as they can enable their web browser to access their device’s camera.
Moreover, experienced developers will find limitations in social AR platforms for making more complex graphics and interactions. In this case, AR Foundation is a tool for creating AR in Unity without facing the usual problem with cross-platform integration. Similar tools also have robust creative communities that are useful for developing creators’ coding skill sets and finding collaborators.
As more people are opting out of social media in search of alternate modes of social connectivity, tools such as AR Foundation and web AR will be useful for building digital infrastructures that are more independent from big tech corporations. Doing so offers more data privacy and creative freedom in the construction of new, virtual worlds.
Immerse is an initiative of the MIT Open DocLab and The Fledgling Fund, and it receives funding from Just Films | Ford Foundation and the MacArthur Foundation. IFP is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing media projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.