Immersive and interactive Augmented Reality museum experience at Carnegie Museum of Art

Artium is an interactive AR museum experience designed to make the Carnegie Museum of Art (CMOA) more approachable, informative, and memorable for visitors.

‍My work centered on designing user experiences that enhance interactive storytelling and foster meaningful connections between visitors and artworks. I conducted user research to understand visitor engagement patterns, developed interaction models to refine AR-driven experiences, and designed intuitive interfaces that encourage exploration and discovery. Through visual and experiential design, I contributed to creating an accessible and immersive system that enriches learning, fosters curiosity, and transforms visitors into lifelong collectors.

SKILLS

User Research
Concept Development
AR Interaction Design
3D Component Development
Video editing

MY ROLE

UX Designer

TEAM

Iris Cai
Kristen Cai
Andrew Sim
Shafiqah Zulfikar

TIMELINE

October 2022 - December 2022

Overview of Artium Experience

Artium Experience Structure

Artium aims to foster a deeper sense of understanding and connection for those who visit CMOA by putting art back into its original context. In order to achieve this goal, Artium experience is divided into three main parts: Unlock, Reveal, and Collect.

Artium Features

01 | Unlock the experience

Learn how to be part of Artium

Learn basic instruction about how to use Hololens technology along with gesture instructions that are needed during the museum experience

Customize the experience

Museum preference setting

Personalize your museum experience by indicating how you want your museum experience to be like

Reveal story behind artwork

Understand context of artwork

Learn about the artwork in depth by engaging with interactions that are designed specifically for understanding the story behind the artwork

Develop your own collection

Collect items for memory

Collect digital souvenir from the artwork
after unlocking all of its interactions

Build personal collection

Curate collected items

Reflecting upon museum experience at CMOA, curate items that you collected into the virtual museum

Concept Video

When we first started to select a museum for emerging technology implementation, our team was determined to choose a museum that has opportunities for multi-sensory experience. Our team chose Carnegie Museum of Arts as our client not only because of our group’s shared appreciation for art but also because of the museum’s role of building a bridge to connect people to art, ideas, and one another.

User Interview

In order to align our proposal to the museum's goal and vision as well as visitors' expectations to the museum, our team conducted interviews and field investigation. Through interviews with one of the professionals from CMOA and ten CMOA visitors, we were able to identify current challenges that the museum was facing as well as insights that gave our team directions on how to approach this challenge.

Research Findings

After compiling and organizing conversations that we had with each interviewees, our team organized our insights in three different categories: Quotes from the interview, Insights/Values in the quotes, Opportunities that we were able to discover from looking at the insights/Values.

This progress helped our team to identify the directions that we are aiming to go for when redesigning museum experience for CMOA. Reflecting upon what we have researched and analyzed from our investigation and interview, our team divided our focus into four directions:

Based on these insights, we developed the experience values and principles that drove our concept. With the insights and design principles as our guide to this project, our team was able to get a better understanding of how to approach the challenge in a way that helps the museum to achieve its visions and goals.

To explore potential solutions to the main problem statement, our team focused on making Artium's main concept to emphasize approachable storytelling as well as multi-sensory and two-directional experiences to transform the museum as a learning resource, and turn curious visitors into collectors. With this concept in mind, we explored through different types of emerging technologies while synthesizing our initial findings into visuals.

Technology Exploration

In order to visualize how our concepts and initial discoveries will be transitioned in action, our team tested out different types of emerging technologies. We focused on exploring with Oculus and Hololens2 to observe their potentials of changing the museum experience to become more immersive and engaging while providing dynamic activity in diverse format. At the end of the exploration, our team decided to focus on using Hololens2 in physical museum experience, while implementing opportunity to use Oculus during post-museum experience for the following reasons:

Before moving into the next steps with redesigning CMOA’s museum experience, we needed to learn how the visitors will be engaging with the artworks; we went in depth with learning about the hand gestures that are used in Augmented Reality to understand how we could provide more dynamic and robust museum experience in diverse formats.

Gesture explorations studying different kinds of interactions

Framework Development

Before crafting scenarios for Artium's experience, we needed to understand how the data of the contents are being retrieved, then displayed on AR surface. In order to see how the system works, we created a system diagram that shows overview of system's flow in how it shows interactive contents through HoloLens2.

System diagram reflecting overall system flow

Compiling what has been previously researched, investigated then discovered, we ideated our concept and apply initial design opportunities into a storyboard. Through building storyboards, we were not only able to construct a basic structure of the overall museum experience, but also receive feedback on parts of the experience that need to be refined.

Brainstorming overall experience flow and making iterations

Overview of Artium's storyboard

Transitioning what we have initially structured on storyboard, we went in depth with refining the ideal user journey of Artium by creating a journey map. We divided this museum experience into four different categories: Discovery, Onboarding, Exploration, and Post-museum experience.

Journey map showing ideal journey of Artium experience

In order to define the user journey in depth, we implemented and organized hand gestures that are being used in this user journey map. This same idea applies with any kind of user journey; depending on what art pieces the visitors interact with, the hand gestures used by the visitors will differ.

Gestures implemented on each steps of the journey

After conceptual development of Artium, our next step was to make the overall experience feel holistic and comprehensive. Going in depth with visual system of Artium, the saturated color palette focuses on highlighting key areas that are significant factors of Artium experience while the style of typography balances intensity of the color palette in a way that balances overall visual of Artium's brand.

Procedure of building Artium's brand narrative

Visual Guideline

Artium's visual system

Our key focus of using the brand narrative is to reflect the process of unfolding hidden layers of artwork's hidden story; we integrated this brand story and identity not only into interactable interfaces and contents, but also into visuals used for promoting Artium experience.

Breakdown of design components

Integrating design components into Artium experience

Prototype Integration

After building foundation of brand's identity, experimenting with digital tools and prototyping virtual objects was a crucial step to visualize conceptual Artium experience into reality. During this step, we emphasized on importance of creating 3D components because we needed a component that not only represents the collectables from art pieces, but also leads the visitors to participate in this interactive art museum experience. In order to understand how Artium's visual components will be interacted in both physical and virtual environment, we did physical computing in various prototyping softwares to see interactive patterns that could be implemented as part of Artium experience.

Developing 3D Models

Testing in various prototyping software

Transferring 3D Model into Unity

As our next step, we transferred 3D Models into AR system in a way that supports the process of helping the visitors to connect with the art pieces beyond current art museum experience. With this process, we were able to convey different methods and formats of content to show how these features make the overall museum experience more engaging and interactive.

3D model into HoloLens2

3D model into mobile AR app

Post-museum experience

What did I learn?

Starting from the process of building a narrative of refined museum experience and matching the client’s needs and values, we challenged ourselves to prototype and execute the concept and idea into reality. Although the process of prototyping through physical computing was difficult, testing and iterating with emerging technologies was crucial in order to understand how to make the static art museum experience more dynamic and entertaining. Having the opportunity to experiment with Augmented Reality was exciting and I was able to learn two of the key aspects that are important during implementation of Augmented Reality: consideration of usage in physical space and interactive components that support the overall experience to become engaging.

What is our next step?

As for our next steps, it is critical to not only attract the visitors to come back to the museum to engage with other interactive components of the art pieces, but also attract their friends and families to be part of this experience by sharing the virtual space they have created with digital souvenirs from art pieces. With this refined museum experience, we hope for the audience to not only be encouraged and motivated to see the artworks in person, but also have the opportunity to implement their own interpretation and make connections to the artwork through physical presence of the artwork.