How are Museums Experimenting with Immersive Technology?

How Are Museums Experimenting with Immersive Technology?

A follow-up with Knight Foundation grantees

Last summer, in conjunction with a Knight Foundation call for ideas, Immerse ran a series of articles on how cultural institutions are engaging audiences through emerging media in creative ways. In the end, the foundation awarded nearly $750,000 total to five museums across the country. The winning projects range from a virtual reality experience that connects audiences with women of the African diaspora, to the creation of new open-source software that transforms data into interactive experiences. A common thread among the projects is leveraging immersive technologies such as VR and AR to strengthen audience engagement.

I spoke with leaders of four of the five projects several weeks before the peak of the COVID-19 pandemic in the United States—including those from the American Museum of Natural History in New York City, the Museum of Art and Design at Miami Dade College, the Japanese American Museum of San Jose, and the Colored Girls Museum in Philadelphia. While all of the institutions are temporarily closed, they are continuing to move forward with their Knight projects on a delayed timeline.

American Museum of Natural History

I chatted with Michael Walker, Manager of Media Relations at the American Museum of Natural History (AMNH) in NYC, about AMNH’s Knight-funded project over email.

Can you tell me about the museum’s project?

The Knight Foundation work is really about telling stories about the museum itself. And, for that, data visualization and interactive storytelling allow for exploration of the museum’s history in a compelling new way. The idea is to provide a very broad view of an institution’s history. Of course, we are focusing on AMNH, but the software we’re developing will be open source and we hope it will be adopted by other institutions. This can be achieved by using metadata from our scientific departments and library archives to visualize whole collections over time and space, illustrating when and from where objects have been acquired. That information can be contextualized in other historical narratives as well — what else was going on in the world or the country at that time? Then, we can add layers of more highly resolved information about individual objects or smaller collections to tell stories about how and why they were collected. We can use photogrammetry or CT-scan data to enable visitors to “handle” objects and to study them in detail.

We are considering using parts of our cultural anthropology collections and related exhibition halls as a focus of our prototype.

Why is it important for museums and cultural institutions to explore emergent technology?

There are two reasons. One is simply that the public is interested in it. They encounter it in other settings and enjoy novel experiences. We can leverage that interest to engage visitors with our content in exciting, new ways. It’s also a natural expression of an institution’s continued relevance to culture and society.

The second reason is that digital technology is an essential part of modern scientific investigation. For instance, scientists use imaging technologies such as CT scanners to study biological specimens. These scans result in extremely detailed digital models that allow for close study of morphology without destroying specimens. These data are so important to their work that when I asked one of our scientists if we could refer to them as “digital artifacts,” he told me that they actually call them “digital specimens,” because they are as primary to their research as the physical specimens from which they are derived.

In addition to data collection through observation (scanners, microscopes, telescopes, etc.), there are computer models and simulations that are essential to how researchers study processes in the natural world, from the evolution of the entire universe to climate change. And so, as a natural history museum, we need to engage visitors in these digital products in order to communicate how and what we know about the natural world. We do this on digital platforms. Emerging technologies that support more resolution and more interaction allow for direct communication of these large data sets. The same way AMNH created the modern diorama 100 years ago, we are thinking about how to exhibit digital representations of nature.

What do you think the future of science visualization is?

It’s the future of science itself! The data will get larger and larger, and so the visualizations will get more and more highly resolved, more and more naturalistic, and so more compelling to the general public. The graphics cards are getting faster so we’ll be able to create more immersive interactive experiences in which visitors can experience and explore our best understanding of how nature evolves and behaves. For science, visualization is an essential part of the process of exploration and interpretation. For the public, it’s a gateway to those discoveries.

Museum of Art and Design at Miami Dade College

Sophie Landres, Ph.D, is the Curator of Public Programs at the Museum of Art and Design (MOAD) at Miami Dade College, and curator of their Knight project and upcoming installation, Through Forensic Architecture: True to Scale.

Can you tell me about MOAD’s Knight project?

Forensic Architecture: True to Scale is an invitation for interdisciplinary partnerships committed to learning about — and through — the technologies that mediate our realities. Amidst what is increasingly referred to as our “post-truth” media environment, our programming draws from experts in the fields of architecture, journalism, data analyzation, platform development, and human rights to teach crucial skills for verifying evidence of state violence.

As part of this exhibition, True to Scale debuts a new forensic architecture investigation, co-produced by MOAD. The work chronicles forensic architecture’s first use of virtual reality to assist, record, and represent witness testimony. Beyond supporting this specific project, the show serves as a generative prototype for future engagements with augmented reality. It is an opportunity for MOAD to develop the infrastructure needed to continue utilizing the didactic and aesthetic potential of immersive technology.

Why is it important for museums and cultural institutions to explore emergent technology?

Just as immersive technologies can play a prominent role in exhibiting information in an aesthetic, precise, and accessible manner, arts organizations can take the lead adapting technologies for the collective good. In adopting immersive technologies, museums and cultural institutions can explore these innovations from a critical lens: How should these technologies be used? How do you draw the line between spectacle and education? To what extent can these technologies harm or help the public good? Empowering museums with immersive tools for storytelling allows institutions to be thoughtful in their approach.

Why are grants like this so important?

From acquiring the technology, to training staff on its use, and creating specialized content for these specific platforms, immersive technology requires considerable investment—and arts institutions are notoriously underfunded. If immersive technology is going to emerge as a new tool among cultural institutions, organizations like the Knight Foundation will be pivotal as financial and educational partners. For example, MOAD’s leadership team was flown to Microsoft’s Seattle headquarters to learn more about their AR and VR tools and applications. The experience was invaluable because it prepared our team to resolve key questions and issues we might face when implementing the technology on our own.

Japanese American Museum of San Jose

Tom Izu, the executive director of the California History Center Foundation and Audrey Edna Butcher Civil Liberties Education Initiative and is leading Hidden Histories of San Jose Japantown at the Japanese American Museum of San Jose (JAMsj). This augmented reality project will enhance the museum’s connection to the surrounding community.

Could you tell me a bit about the museum?

JAMsj was founded about 32 years ago in the historical center of the San Jose area’s Japanese American community. The museum’s official mission is to collect, preserve, and share Japanese-American history, culture, and art with a focus on the Greater Bay Area. Its current strategic plan includes ideas of how to expand its scope to place Japanese-American history in a broader, multicultural narrative, and to use new technology to reach a younger demographic of potential museum members. It is a small organization with only one full-time director and is supported by a large number of community volunteers. It is the only museum in Japantown.

Can you tell me about JAMsj’s Knight-funded project?

JAMsj has been searching for a way to expand its audience and volunteer base beyond the local Japanese-American community and to find new ways to reach a younger demographic. We believe an AR art project can do this by:

Going beyond our museum walls to capture the imagination of the growing number of visitors to Japantown’s ethnic-related cultural and art activities;
using AR and displays in our gallery to encourage our regular visitors to explore the surrounding neighborhood to see new potential in what their local museum can do for the entire community, and
using AR art as a way to reach out to the two communities left out of our standard narrative and who share the history of our neighborhood: the Chinese and Filipino communities, both significant and growing regional populations.

Your project statement says that you’ll enhance the museum’s connections with the surrounding community by leveraging augmented reality and art. How do you plan to do this?

JAMsj’s grant team will organize, train, and support local artists, tech people, and college students to create nine new art installations using augmented reality technology that expose a layer of hidden history and little-known stories about the three communities and the common ground they share.

The project will bring together representatives from JAMsj, the Chinese Historical and Cultural Project (CHCP), the Filipino American National Historical Society of Santa Clara Valley (FANHS), and their supporters to inspire the artists and designers of the project. To make future joint projects possible, we will deepen the ties between the three communities the organizations represent.

JAMsj will engage students from San Jose State University, Santa Clara University, and De Anza Community College by further developing existing ties with digital design and Asian-American Studies faculty from those institutions. These youth, who may be art-, technology-, or history-oriented, will be teamed up with ethnic artists from their local communities and trained to utilize AR technology.

Why is this project important?

There is a sense of urgency to this work. While Japantown is experiencing an upsurge in interest from the general public, it remains quite vulnerable to the growing pressures of gentrification overtaking Downtown San Jose. This makes it all the more crucial to broaden our support and seek unity and common cause with the Chinese and Filipino Americans.

Why use AR specifically?

AR allows us to use the physical space of the surrounding community as an exhibit space and makes it highly accessible to the public since they can use their own cell phones or hand-held devices rather than using special equipment. They can view the AR art pieces at any time.

What do you think is the future of emergent technology in museums?

As emergent technology becomes more accessible to the public and affordable for institutions, it will expand the ability of smaller museums to “grow” exhibit and activities spaces by using VR/AR. Besides creating new virtual space within the museum, it can make the physical neighborhood area surrounding the facility become exhibit space.

For museums to stay relevant to the public, they will need to position themselves to become the public space for learning how to use this new technology for social engagement and education.

Colored Girls Museum

Vashti DuBois is the founder of The Colored Girls Museum (TCGM) in Philadelphia, a multidisciplinary-memoir museum focused on “the ordinary colored girl.” This four-year-old institution is established and supported by a diverse collective of community members including artists, neighbors, friends, educators, and students.

Could you tell me about The Colored Girls Museum’s Knight-funded project?

TCGM and The Center for Digital Humanities at the University of Arizona (CDH) have teamed up to create a walk-through virtual reality museum. TCGM is an emergent art community where each exhibition is co-designed by two or more artists. Blended authorship and co-creation is foundational to our curatorial practice. We aim to extend this model to VR, where our community of Black female creatives will collaborate to imagine how their work will show up in virtual space.

The First Time Ever I Saw Your Face is an exhibition that features portraits of young women of the African diaspora aged 7–21. For this exhibition, portrait artists and subjects are asked to document their creation process/experience through journaling. We will work with this dynamic group of artists and models to build a world around the artworks and process journals. For instance, portraits may come to life and talk to the audience. By holding space for a communal process, the art will speak to the models and artists lived experience, vision and memory.

Currently, there are over 10,000 original artifacts located at TCGM, in Philadelphia, PA. Normally, these artifacts are only available for visitors to the museum to view but rarely touch. The stories each of these objects hold is only accessible if museum staff or the artist is physically present to tell it. The introduction of AI/VR technology will provide both physical visitors to the museum, and virtual visitors, enhanced access to the narratives which inspired the creation of each artwork, thus enabling a profound and rich learning experience around our collection, our artists, the local community and African diasporic culture at large.

Why explore VR and AR specifically?

As a new museum intent on serving a historically overlooked audience, we recognize the enduring power a VR web museum will have in making our real-life space accessible to wide audiences online whose lives, stories, and images are traditionally excluded from American institutions.

We intend to uplift hidden histories and lived experiences through technology. Our strategy includes a pre-production phase where we can ask ourselves and our community two fundamental questions: What should a (virtual) museum do? How can we use technology to build equitable communities which center historically marginalized voices?

Museums and cultural institutions are being called upon to play a much more active role in communities — to reach beyond the comfort of display cases to relate to the concerns of ordinary citizens, diverse communities, and to create varied opportunities for education and engagement. New technologies remove many of the barriers that keep some people out of this institution.

Why is it important for museums and cultural institutions to explore emerging technology?

TCGM creates for our audiences an immersive environment in which “Black girlhood” is the primary experience. VR technology allows us to extend that. VR can position the patron as an “an actor,” not just a bystander. The patron can walk in a Black girl’s shoes. AR allows us to imagine multiple ways that guests might interact/engage with our exhibits and objects. It allows us to overlay information in different ways — provide sensory entry points, for example, and make digital experiences adaptive and interactive.

Why are grants like this so important?

TCGM is a small museum and we like that. Our size allows us to be responsive to the communities we serve. What can a museum do ? A museum can be an advocate, it can be a safe space for people and ideas.

A grant from a funder like the Knight Foundation for immersive technology investigation and experimentation is a game-changer because it allows us to have a bigger footprint and broader impact without having to become a “bigger” institution with all the challenges that come with that. Additionally, this kind of grantmaking enables us to attract critical partners like Bryan Carter and his team at Arizona University. Our partnership activates new learning and brings folks into this new technology conversation who typically might not have a seat at the table.

We work with artists. We work with Black girls and women. We work as a collective. Therefore we know that what we create together in the field will be fundamentally different — and new. Without opportunities like this, TCGM might still be “imagining” what we could do with technology if we had the resources — who we could reach. Now we have an opportunity to reach those folks. We have an opportunity to do and not dream. The technology is expensive — it changes quickly and it is so important to be able to get your foot in the door. Small institutions and grassroots projects need this more than anyone. It gives us a chance to really focus on our missions connecting and serving people.

Immerse is an initiative of the MIT Open DocLab and The Fledgling Fund, and it receives funding from Just Films | Ford Foundation and the MacArthur Foundation. IFP is our fiscal sponsor. Learn more here. We are committed to exploring and showcasing media projects that push the boundaries of media and tackle issues of social justice — and rely on friends like you to sustain ourselves and grow. Join us by making a gift today.