The challenge
The solution
Car buying and conversation
The human side of car buying
Evaluating target audience
Opportunity spaces evaluation
The hand's on experience
The conversational dynamics
Kicking the tires
Improving the experience
Testing in the wild
Moving forward

The before and after

I was challenged to explore if and where a conversational agent could improve the user experience of the end-to-end car buying process. I assumed that this approach, starting solution first, would mitigate our ability to create a truly user centered solution. What the project has shown me is that there is significant value in starting with a hypothesis about the value of emerging technology but, it is even more important to spend the time to understand and validate whether or not the technology truly provides real user and business value through rapid idea testing and iteration.

Organization

CarMax
Jan 2021 - Present

My hats

Project management
Conversational design
Development

Tools

Figma
Miro
ReactJS
Voiceflow
Confluence

Project overview

The challenge

How might a conversational agent create value for both the customer and CarMax in the end-to-end used car buying experience?

The solution

I designed a conversational agent to help customers learn contextually relevant information about a car and validate their decision to buy it while exploring and test driving it in person, completely autonomously.

Check out the solution in action

How the agent answers customer questions

Interaction model for agent reactivity

How the agent proactively engages customers

Interaction model for agent proactivity

How I tested the agent with real customers

I designed a rig to put in customer's test drive cars to detect contextual data and send multi-modal information

I developed an intelligent Wizard of Oz research interface in React to simulate the agent

Behind the scenes at the simulation command center

Learn more about our testing with real customers

I also developed a future plan for integrating a conversational agent across different contexts and modalities into CarMax's entire end-to-end service system.

Explore how the agent exists across the end-to-end car buying process

Solution

01. Wizard with intelligence

The final design shows a step by step wizard that takes users through the most important information needed to provide an accurate analysis. With the amount of data needed and the potential for users to have missing information, the tool uses intelligent dynamic defaults to estimate numbers where users may have gaps.

02. building trust with transparency

The focus here is to help build trust with the user by showing them that there are significant algorithmic processes being undertaken to provide accurate analyses and show what those actual processes are.

03. actionable insights

The final screen of the tool displays the results of the analysis showing the most important data points with opportunities to get more detailed analyses delivered electronically by providing personal contact information. Users can also manipulate important real-time variables that do not require a full analysis re-run.

04. Flexibility to run multiple analyses

Using the left collapsible menu, users can see their current inputs and easily modify one or more and run a new analysis without having to go back through the entire wizard.

01. wizard with intelligence

The final design shows a step by step wizard that takes users through the most important information needed to provide an accurate analysis. With the amount of data needed and the potential for users to have missing information, the tool uses intelligent dynamic defaults to estimate numbers where users may have gaps.

02. building trust with transparency

The focus here is to help build trust with the user by showing them that there are significant algorithmic processes being undertaken to provide accurate analyses and show what those actual processes are.

03. actionable insights

The final screen of the tool displays the results of the analysis showing the most important data points with opportunities to get more detailed analyses delivered electronically by providing personal contact information. Users can also manipulate important real-time variables that do not require a full analysis re-run.

04. Flexibility to run multiple analyses

Using the left collapsible menu, users can see their current inputs and easily modify one or more and run a new analysis without having to go back through the entire wizard.

Process

Getting familiar with used car buying and conversation

How does the current car buying experience exist today?

It's complicated.

The used car buying experience is non-linear. People enter at different points with different levels of knowledge and take different paths.

It varies.

Every customer is unique and so are their needs and motivations when looking for the “right” car.

It's competitive.

The car buying industry is filled with companies taking different approaches to facilitating this complex process from fully online experiences to online/in-store combinations.

How can we successfully apply a conversational agent to a service that extends across contexts and modalities?

5
Expert
Interviews

What I explored

I decided to engage experts in the field of conversation design and conversational agents to help us understand more about different kinds of conversation methods, the effectiveness of agent embodiments, when to use verbal and text-based interaction, and factors that affect engagement with the agent and a person’s willingness to disclose personal information.

Key insights

01. An agent's capabilities should match the user's perception of what it can do

02. Discovering how to best interact with a conversational agent can be difficult

03. Conversational agents are great at providing information but, struggle when it comes to contextualizing it

04. Conversational agents have difficulty replicating people's ability to handle complex questions and show empathy

Understanding the human side of car buying

What specific problems are present in the current CarMax experience for everyone involved?

I knew that in order to create a solution that could be successfully integrated into this complex service, I needed to get a more nuanced understanding about the problems faced by the different actors on both the consumer and business sides of the experience.

18
Sales Consultant and Chat Member Interviews

Our learning objectives

I interviewed CarMax sales consultants and chat team members to understand more about how to enable a good interaction with a customer, areas where they need more help meeting customer's expectations, and what parts of the process might be the most fruitful for a conversation agent.

10
Directed Storytelling Interviews

I had CarMax customers who had visited a lot and test drove a car within the past 5-days walk me through their experience on the lot to understand their behaviors and motivations, identify any influential social factors, and uncover any unmet needs from the online and in-store experience.

5
Contextual Inquiries

I observed people currently in the market for a car conduct online research to understand how they navigated online research, what they were looking for, and areas where they had difficulty finding what they needed.

Conducting virtual interviews with sales consultants

Key insights

"I felt that the car could tell a story about myself"

People make a decision on the kind of car that is right for them based on their needs but, make a final decision on what to buy based on their personal preferences.

"I knew what I wanted but, I needed to go in and take a look"

The hands on experience validates people's research and assures them of their decision to purchase a vehicle.

"I didn't want a rep there to steer the conversation"

People want the flexibility to shop for cars without social pressure but, also want contextualized information that a sales consultant can provide.

"I trusted a sales person when they understood my needs"

When visiting CarMax, people expect to have a knowledgable guide who meets them where they are in their own car buying process.

Levels of proactivity

I helped us define the level of proactivity by recommending we vary it based on the estimated confidence the voice assistant had in the actions it was taking (a lower level of proactivity is assumed to use less personal data and deliver less overall value). In all scenarios, the assistant engages the participants.

Low proactivity

In a low proactive scenario, we had the agent ask the participant what they wanted to listen to and confirm all actions before taking them. The assistant would not make any music recommendations.

Medium proactivity

In a medium proactive scenario, we had the agent recommend songs, playlists, volume levels and, changing speakers (based on prior knowledge and contextual factors) to the participant but, still confirm all actions before taking them.

High proactivity

In a high proactive scenario, we had the agent just communicate that they are changing the music, volume level, or speaker (based on prior knowledge and contextual factors) without confirming with the participant. We also had the agent explicitly explain their actions.

Evaluating our target audience

While every car buyer is a customer, not every customer is our target user

From the insights we gained, I identified two kinds of customers that we wanted to narrow in on with our conversational agent.

The first persona is "The In-Store Explorer", who doesn't always have the time to conduct enough research themselves and use experts and the "hands-on" experience with the car to narrow in on the right one.

The second persona is "The Validation Seeker", who conducts significant research to find the car that is the right fit for them and is willing to take the time required to do that. They are confident in their research and look to use the in-store experience to validate it.

Ideating and evaluating opportunity spaces

I reframed the major pain points we identified in our user research into opportunity areas to explore with conversational technology. We began this process by defining opportunities for innovation based on their feasibility for a conversational agent, significance of customer needs, and value to the business.

Key opportunity areas for a conversational agent

Bridge the information gap between the on-line and  on-lot experience

Facilitate an informative solo test drive

Strengthen communication between sales consultants and customers

Guide the customer through the CarMax process and build assurance

11
Speed dates

Methodology

We created 6 storyboards exploring each of the opportunity areas to quickly present to CarMax customers to validate that the problems we identified were indeed a signifiant pain point and understand people's reactions to having that problem be solved through the use of a conversational agent.

Conversational agent principles

Our speed-dates both validated most of the pain points we had identified and also guided us towards defining 3 principles for defining our conversational agent.

Autonomy

Allow customers to design and control their own car buying experience

Accuracy

Be a reliable source of accurate and relevant information

Assurance

Use information about the customer and the vehicle to validate their decisions

Focusing on the hands-on experience

The CA principles helped us to narrow on designing a solution that facilitates a more informative autonomous hands-on/test drive experience

Using our validation research we gathered as a team to evaluate what opportunity would not only solve a high-impact problem and deliver significant business value but, also push the boundaries of conversational technology. We landed on designing a conversational agent to facilitate an autonomous hands-on/test drive experience to provide customers with a better way to experience a car on their own terms. The hands-on experience involves any kind of in-person interaction with the vehicle.

Creating a competitive advantage

Additionally, a key insight that our research informed us of was that for both the customer and business, the in-store experience provides significant value and will continue to do so in the future. I hypothesized that focusing on a solution that takes advantage of this experience would differentiate CarMax in the evolving competitive automotive landscape and give them a unique advantage over their online-only counterparts.

Defining the conversational dynamics

First things first, how should the agent interact with customers on the test drive?

Core to understanding how to design the best autonomous and informational test drive experience was first determining the interaction dynamic between the customer and agent. We wanted to know if people preferred interacting with an agent that just answered their questions (reactive) or one that would also actively engage them with contextually relevant information and questions on the drive (proactive). We also wanted to know how much data people were comfortable sharing with the agent.

The Interactive Test Drive, v1
Participant interacting with our agent on the test drive

We had real CarMax customers (virtual) and other car buyers interact with a conversational agent on a test drive experience. Participants were allowed to ask the agent questions about the car's features, history, and the CarMax process. They were also engaged by the agent during the drive with contextually relevant information about the car based on their reason for buying and features of interest.

Building the test drive experience

Data usage

I started by defining what data the agent would need to be proactive, how that data would be captured, and mapped the actions that could be taken.

Defining the agent features and rules

I created a list of agent capabilities, rules, and a system diagram for how it should interact with the participant. The agent features were based on our hypotheses on what kind of information people would most likely want the agent to provide.

The in-car screen

We created a display to be used during the drive to help people discover how to interact with the agent and provide feedback on the agent's current state.

The rig

We used a microphone and two cameras to capture real time input data, monitor the drivers, and simulate the assistant.

Key usability and UX findings

People found value in the agent proactively engaging them with contextually relevant information

People generally felt that the dynamic between them and the agent was appropriate and felt in control even with the agent's proactive interventions

People want to ask very detailed questions about the car's features and operation and expect the agent to know that information

People want to know about the operation of the car and the features that exist before they begin the test drive

People want to be reminded to experience the features important to them and discover other features they may be interested in

People were fine with an always-on assistant if they could clearly understand what data was being used, how it was being used, and the value provided

Kicking the tires

What if we separated the informational and emotional experiences?

One of the biggest revelations of our initial prototype was that people wanted to start learning about the the car before they even hopped on the test drive. Learning about the features of the car and how to interact with the agent on the test drive was more difficult than we anticipated. People preferred to use the test drive time to experience what they had learned about the car in context and focus on determining whether or not it felt like the right car.

Kicking The Tires, v1
Participant interacting with our agent while walking around the vehicle

I simulated a conversational agent guided tour of a car with information about both its history and features contextualized to be relevant to the participant's car buying needs. We used the same participants from the test drive study to see how this added experience could have improved their previous one.

Key usability and UX findings

Engaging with the agent prior to the test drive helped people become more comfortable while interacting with it on the test drive

People wanted to be guided by the conversational agent while understanding the features and operation of the car then have control over the interaction with the agent while on the test drive

People want to see visuals of where the feature is located when the agent introduces it

People need to feel like they have the time and space to explore the features as the agent guides them

Improving the experience

How did I iterate on the experience?

Scoping down on the agents proactive engagements

I decided to focus the agent's proactivity on providing contextualized information about the features and history rather than also responding to real time conversational cues and information about the larger CarMax process. We made this decision because people felt that that information was more relevant to the current experience and preferred.

Defining an agent personality

I defined how the agent should interact with users based on it's perceived role and the values it should embody in order to be effective

Testing in the wild with a novel prototyping tool

I designed a setup to simulate a conversational agent that we deployed with CarMax customers in their real test drive cars

The intelligent Wizard of Oz interface

I built a testing tool to help us improve our ability to simulate more realistic conversational agent experiences by helping us deliver responses faster and more accurately.

The testing tool I created pulls API data for any car a customer wants to test drive, populates the interface with categorized responses for the specific vehicle, generates suggested responses based on detected utterances, and logs a transcript of the interaction.

The full "hands on" experience

Out team simulated an autonomous walk around the car and test drive with a conversational agent with seven customers to evaluate our full experience. I used the intelligent Wizard of Oz research interface and designed a phone rig for participants to interact with the agent and capture contextual data to inform proactive interactions.

Key usability and UX findings

One of the six cars we sold with our conversational agent

The customer's perception of the conversational agent's personality improved their enjoyment of what they characterized as a usually stressful experience.

Customer's found value in the information the agent was providing and did not have many concerns with the real-time data collection.

My hypothesis of creating a guided experience while initially exploring the car did not prove to be effective as customers wanted to explore the car on their own terms

Sales consultants felt better about their ability to guide customers using the data they received from the interaction between the agent and customer

Moving forward

Having validated our design concept, the CarMax team was ready to begin testing our new experience at higher levels of fidelity. In order to support their continuing development, I created a phased implementation plan for integrating this new experience into their existing in-store experience.

Conclusion

Reflecting on the project

This was my first full end to end product design project starting with research all the way through to the implementation of the beta version of the product for both consumers to interact with and to be presented to potential customers for licensing. It was an incredible experience and challenge learning how to create a truly value-generating product with a great user experience under a variety of time, technical, and user constraints. I was unfortunately unable to continue the project beyond the beta release, however, I helped to define the scope and direction for future iterations of the product. These next steps included conducting extensive user testing, dynamically changing the required inputs based on a user's specific fleet management role and desired analyses, and the creation of a dashboard for clients such as electric utilities who want to license the tool to collect and manage user data.

Back to projects