Part 2: Exploring AI at IDFA’s Doclab
Most AI systems are invisible to us. To the extent that they do whatever they are supposed to, we simply and unselfconsciously embrace them — tools like any other, except for their invisibility. But sometimes, because of curiosity, malfunction, or fear, we stop and reflect on their structure, their processes, their implications. The artists at the heart of this research study have staked out a third position, embodying aspects of AI in their projects and going beyond both “engaged” and “critical” stances. This position offers a powerful vantage point from which to explore AI.
As noted in Part 1, we partnered with IDFA to examine five projects selected from their 2018 Doclab:
- A Dinner With Frankenstein AI (Lance Weiler, Nick Fortugno, Rachel Ginsberg);
- Eat | Tech | Kitchen (Klasien van de Zandschulp and Emilie Baltz);
- The Social Sorting Experiment (Steye Hallema and The Smartphone Orchestra);
- I’ve Always Been Jealous of Other People’s Families (Shirin Anlen);
- Algorithmic Perfumery (Frederik Duerinck).
At the outset, it is important to note that despite centuries of anticipation and effort, these are still early days in AI’s development, particularly in the arts. So, too, this research into a handful of documentary makers, projects, and their audiences, which seeks to establish frameworks for analysis and sites for further inquiry.
Although all of these projects share an interest in exploring AI, few of them actually deploy robust AI systems. However, as rudimentary as the AI techniques might be, all five projects explore the space where digital and analog collide; all put their users into scenarios where they can respond to AI instructions (or not); and all creatively exploit AI as a Rorschach test to evoke (and occasionally confront) users’ ideas and associations.
Critically interrogating AI
Weiler, Fortugno, and Ginsberg’s A Dinner With Frankenstein AI delivered exactly what the project title promised: a sit-down dinner with other guests from the festival and “AI,” a lurking presence whose only trace was a multi-directional microphone centrally positioned on the dinner table, with occasional questions and conversation starters whispered into guests’ ears pieces. The system’s questions explicitly encouraged discussion of the participants’ assumptions and fears regarding AI, but also raised more general questions of whether or not the system was truly responsive to the table’s conversation, and if so, what it was responding to? Were its interventions triggered by word recognition? By syntax and inflection? Or were they merely randomized all-purpose questions? Or might they have been “faked,” with humans behind the curtain listening and feeding prompts to the diners in order to provoke discussion? The discussions that ensued entertained all of these options and more, with a randomized group of people, yielding a portrait of contemporary thinking and fears about AI.
The
Social Sorting Experiment (Steye Hallema and The Smartphone Orchestra)Steye Hallema and The Smartphone Orchestra’s The Social Sorting Experiment enacted on a human scale the basic workings of ranking algorithms so essential to social media and networked preference systems. Physically arranged on a large grid, the user/performers responded to a series of questions, moving to one of three quadrants around them depending upon their answer, in the process slowly winnowing down the number of participants — and finding correlations. By enacting the simple choices that play out on the level of individual algorithms, complexity emerged — as did a sharpened sense of the sometimes spurious correlations that such sorting processes enable. From asking participants to render judgments about the shape of a neighbors’ ears to making conclusions about the emotional compatibility between two performers, Social Sorting confronted users with the systematic workings of algorithmic correlations and ranking systems in real time and physical space.
Klasien van de Zandschulp and Emilie Baltz’s Eat | Tech | Kitchen pushed beyond the algorithm to its supporting metaphor — the recipe, in this case, with a nod to Filippo Marinetti’s 1932 La Cucina Futurista, a manifesto for the Futurists in the form of a cookbook. “A restaurant for digital art” led by an AI Bot-Chef, the project playfully engaged users to reflect on their rituals of technology consumption. Drawing on the Futurist cookbook, unexpected combinations and remixes helped to provoke, confront, and cajole deeper reflection about the technological and algorithmic systems that sustain our daily lives.
I’ve
Always Been Jealous of Other People’s Families (Shirin Anlen)Shirin Anlen’s I’ve Always Been Jealous of Other People’s Families shifted from the kitchen to the dinner table, and from algorithm as recipe to AI, and in particular generative adversarial networks (GAN). Anlen’s project interrogated machine learning by probing the spaces between “real” families (the participants around the table) and different machine learning networks trained on data sets of images of “perfect family dinners.” Distortions and gaps, “error and loss,” between reality and data ensued as the project (like several others, in an initial phase of development) set the framework for asking whether artificially intelligent systems can experience mental disorders — a question Anlen will pursue as her larger Marrow project continues. Anlen, an MIT Open Doc Lab fellow, has written eloquently on her process here in Immerse.
Finally, Frederik Duerinck’s Algorithmic Perfumery delivered on the promise of personalization by concocting a made-to-measure perfume based on users’ social media feeds, answers to a battery of questions, and aroma preferences. The algorithms worked their magic in the form of an impressive computer-driven chemical apparatus that ultimately yielded a small flask of perfume that the user then rated. In fact, this user feedback is the heart of the system, helping to train the algorithms and refine the process, even as the user is given a peek into the special blend of magic and science that lurks in today’s AI systems.
Context … starting points
The AI-related projects in the 2018 IDFA Doclab shared specific contextual concerns that inflected their approaches:
Public anxiety: One of the most outspoken was the concern that popular cultural depictions of AI had distorted the public’s sense of developments and needed to be countered. Emily Baltz spoke about Eat | Tech | Kitchen’s goal of “confronting public anxieties and anticipations regarding A.I” Hallema described The Social Sorting Experiment’s position as being “up against impressions of the self-learning AI nightmare,” even though his project is critical of big tech’s uses of AI. And Weiler said that A Dinner With Frankenstein “aims to upend the dystopian view of artificial intelligence as well as provoke and broaden the conversation around the trajectory of this rapidly emerging technology.”
Subversion: A second contextual concern targeted social conformity and imitation. Duerinck of the Algorithmic Perfumery said, “A lot of things are basically following what’s successful. What comes out is tweaked to what the marketing department came up with when they did their research on what has high potential, or when they tried to clone something that’s doing very well. If we look at it from an AI and data science perspective, we see a lot of unexplored territories to be explored.” The arts explore, providing the field with a radical “research & development” unit.
Disruption of the status quo is another related response in the arts, and it abounds in these projects. Van de Zandschulp described her and Baltz’s project’s response to the habits that settle into our expectations of technology, saying of Eat | Tech | Kitchen that it, “reflects on the absurdity (and reality) of how we consume technology” and “uses absurdity and the unexpected to subvert knowledge and gain fresh insight.” She also noted that Google Home was just introduced in the Netherlands as of July 2018, making their intervention in the form of a “bot-chef” both pointed and timely.
Critique: Third and finally, concerns regarding AI’s problematic use and even nature provided a context that several artists mentioned. Hallema’s project responds in part to how tech companies use (or as Hallema puts it, “steal”) our data. As users “enact” Social Sorting’s algorithm, they gain insight into the value of simple data traces. Shirin Anlen, by contrast, calls out the fundamental nature of systems trained on human behaviors: “We are attaching ourselves to an idea of ourselves, and now we are training life-changing technology to see us through those fake lenses. When the connection between ourselves and the idea of ourselves breaks down the paranoia starts.” How to make this process both visible and legible? How to encourage audiences to see critically? Each of the projects follow their own path, reflecting their understanding of AI and the public.
Algorithmic Perfumery (Frederik Duerinck)Dissecting the phenom without baffling the audience
One of the MIT research project’s initial questions regarded how the artists conceived of and worked with AI — as a tool, creative partner, discursive object, etc. In fact, the majority of works engaged the idea of AI more fully than actual AI technology.
Social Sorting had its audience perform simple ordering algorithms, in the process generating complex conclusions. Frankenstein may or may not have actually deployed AI in interacting with its diners, using the ensuing uncertainty to provoke a conversation about participants’ hopes, fears, and basic understandings of AI. And Eat explored the metaphoric relationship between algorithms and recipes, with one component of the project, the “bot-chef,” scraping conversations and building recipes. Two projects, Perfumery and Jealous, actually used advanced AI systems, in the former case, learning from the audience and testing their responses to algorithmically-concocted fragrances, thus refining the AI; and in the latter, confronting the audience with their relationship to training sets drawn from ideal families and probing the differences as a fertile and powerful insight.
Using an elegantly simple mechanic, Anlen’s Jealous existed in two states. A small brochure explained that the project is a prologue to the larger world of Marrow, which considers mental disorder in AI systems and laid out the work of the Generative Adversarial Network that drives the experience. In one state, the user has the experience armed with this information; in the other, the experience is raw and uninformed until after the fact. Based on audience interviews, we learned that the informed audience had considerably more complete and satisfying experiences; but the uninformed audience left more inquisitive, unsettled, and sometimes confused about the take-away.
In an era of widespread AI-illiteracy, this finding poses a challenge for artists: the illustration of a known concept satisfies but doesn’t provoke, and an ungrounded experience provokes but also confuses. How can we provoke with clarity?
Embodiment
Anlen said of her project, “We were extremely excited to learn how physical interactions support deep and meaningful learning processes.” In fact, embodiment and the blending of digital and analog elements was a key factor in all of the projects.
The physicality of Social Sorting’s reliance on telephone interfaces and physical enactments of a sorting algorithm gave a tactile dimension to the transformation of simple data into social conclusions. And at the same time, it exposed the superficiality of those conclusions by contrasting them with the real sociality that emerged from rubbing shoulders and chatting with one’s neighbors in shared endeavor.
Three projects (Jealous, Eat, and Frankenstein) explored embodiment through food. Jealous did it through the ritual of a family meal, complete with formatted conversation, social protocols, and idealized roles. And Eat and Frankenstein went on to involve the literal consumption of food, drawing on dining rituals. Eat literalized the algorithm as a recipe, even as it offered users an opportunity to “experience the mysterious algorithmic rituals under the hood of social media.” Embodiment in these cases is playful, haptic, and social, throwing the weight of reflection and analysis to the assembled group and the conversations that ensue.
The Algorithmic Perfumery also managed to operate on several levels, playfully exploring personalization algorithms (users’ social media feeds, answers to questions, and olfactory preferences blended into an algorithmic puree) to concoct a fragrance. Duerinck’s AI software and compounding hardware machine are similar to commercial models, but his data mix sets him apart. Personalization as “recipe,” of course, only works in the context of a huge data set (crowdsourcing) and testing — one of the subtle aspects of the project, in which the user is repositioned as data-source. As with the other projects, the bridging of analog and digital serves as a site of attraction for users (Food! Perfume!), but at the same time renders visible the reordering of the real world on the basis of mere scraps of information.
Envisioning the user
Projects are designed for audiences, but in several cases, there was an explicit agenda. In Duerinck’s case, the participating public in fact provided a testing set. Hits and misses on the correlation of a given user and scent enabled the AI to learn. Our role as “AI-trainers” is veiled, yet obvious as the project drives home its point. Social Sorting embraced its participants in the spirit of co-creation. Hallema said that he was interested in creating “stories with the public, not for the public” and in building “a story simulator to give the public the main role.” Anlen spoke of using AI as a “prompt to help us see ourselves.” And Weiler, discussing Frankenstein’s testing protocols, said that his team tried to figure out “how to create more agency for the audience.” Baltz and Van de Zandschulp used the unexpected to confront audiences with the “absurdity (and reality) of how we consume technology,”pursuing “AI as prompt to help us detect ourselves.”
All of the projects except Perfumery (and it too, by implication) were socially situated. They explored typically personalized and private uses of AI openly and with groups of people. By revealing the operations of their AI systems (or pointing to how they might have worked had they been real), and by doing so through social enactments (dinners, performance, production), they enabled embodied insights into AI, moving beyond the familiar critique-or-embrace binary.
Looking ahead
All of the projects studied exist in a state of iteration. Their public appearances are opportunities for testing and subsequent tweaking and refinement. In some cases, the project we saw is the project that continues to be refined (Perfumery); in others, the project is but a piece of a larger nested complex of projects (Jealous and its relationship to Marrow).
That process of iteration is also a condition of the MIT Open Documentary Lab’s research, which is using these findings to help frame an ongoing interrogation of AI as an emergent factor in documentary. As stated in Part 1 of this report, if AI-based experiences represent a major shift from the more familiar subject-object dyad of documentary storytelling, our success in making a case for their aesthetic and documentary potentials will turn on careful analysis and understanding. Stay tuned as the work continues!
This is one in a series of articles for the International Documentary Festival Amsterdam’s Doclab’s Immersive Research and Development Program, launched in 2018 in partnership with, among other organizations, MIT’s Open Documentary Lab (ODL). For the kickoff period, ODL’s research focused on AI-based documentary experiences. This ongoing work builds in part on a field study that MIT’s Co-Creation Studio has undertaken on co-creation with, among other things, algorithms. As research results accrue over the next few years, we hope that the project will yield AI-specific criteria for curators, funders, and ultimately the public, as they seek to assess proposals and documentaries.
Immerse is an initiative of the MIT Open DocLab and The Fledgling Fund, and it receives funding from Just Films | Ford Foundation and the MacArthur Foundation. IFP is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing media projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.