2021 - 2024
Premium Conversation Intelligence
CallRail • Senior Product/UX Designer • Marketing, Analytics, AICallRail is an industry leading call tracking and analytics platform with the mission of helping their customers market with confidence. Their robust marketing, analytics, and reporting software allows businesses to quickly set up and manage marketing campaigns using one or more phone numbers. Building on this foundation, Conversation Intelligence bundles call transcripts with powerful automation tools, such as key term highlights, auto-tagging, and automation rules.
Launched in June 2024, Premium Conversation Intelligence (PCI) leverages these capabilities to go one step further. Using machine learning to analyze hundreds of thousand of minutes of customer calls, the suite includes call summaries, sentiment analysis, and aggregate insights reporting among its AI-enabled tools.
01 . Project overview
Problem space
The core problem is quite simple: as CallRail customers use our marketing software to attract business, call volume goes up. As the volume of calls and potential leads increases, the ability to effectively address and follow-up with callers decreases. This reduces the effectiveness of marketing campaigns, as they produce more leads than a small business with limited resources can realistically address.
“Can AI deliver the insights needed to drive customer revenue?”
Our team had a hunch that AI could indeed be very helpful in our customers day-to-day business operations, but we needed more than that to build a truly compelling and high-impact product. We needed to know: Could AI be used to drive customer revenue by surfacing more high quality leads? Could AI drive revenue and ROI not only for our customers, but for CallRail itself?
My role
Product strategy
Defined product vision and articulated scope, including phased product launch and feature rollout.
End-to-end design
Led product discovery and exploration, designed wireframes and prototypes in Figma, worked with engineering on implementation.
User research
Identified customer cohorts, worked with UXR to set up and conduct customer interviews, distilled and communicated key insights.
Follow up and iteration
Continued to develop Premium Conversation Intelligence through usability testing and further design iteration.
Key pain points
Call volume too high to manually review calls for high-quality leads
Time spent reviewing calls reduces time spent on core business
Too few signals indicating lead quality
No aggregate insights
Key results
60%
19%
31%
Reduction in time spent reviewing transcripts
Increase in MRR for PCI subscribers
Increase in MRR for Conversation Intelligence
02 . Research
Product discovery
Designing a high-quality and high-impact solution required that I better understand who we were designing for and the exact problem we were solving. Using Looker I segmented our customer base by product plans, Conversation Intelligence (CI) subscription status, call volume, and total call transcription minutes per day. We focused primarily on small and medium sized businesses with high call volume and limited staff.
120 mins transcribed
Customers needed to have at least two hours of calls transcribed per day.
Fewer than 25 employees
Small businesses with fewer than 25 full-time employees were prioritized.
Segment demographics • 24 total interviews
6
5
4
3
6
Home services
Real estate
Legal
Medical
Marketing agencies
Understanding customer needs
FullStory delivered crucial insights into customer behavior. Specifically, I wanted to know how long customers were spending in each part of the app reviewing calls. On average, businesses were spending 91 minutes a day reviewing calls for signs of a lead. The call log, interaction timeline, and call detail were the most used areas for call review. This was a good start, but I wanted to dig deeper to understand:
What signals matter most to you when considering a lead for follow-up or as a qualified lead?
After 24 interviews, we came away from the discussions with a better understanding of their needs and the following key insights:
91 mins
30%
73%
Customers spent an average of one-and-a-half hours per day reviewing calls for signs of high-quality leads and following-up.
Approximately 30% of our interviewees were already using AI on a regular basis to assist with daily business operations
The majority of customers stated that reviewing calls was the second-most time consuming activity aside from maintaining business operations.
Milestone 01 • Key insights
Nice to Haves
Easy way to understand the subject and overall content of a call.
Easy way to understand if it’s worth following up with a lead.
Easy way to know if the caller had a good experience with the agent.
Easy way to know if a sale was made.
Easy way to know if an appointment was booked.
Easy way to know what questions were asked during a call.
Be able to summarize multiple calls.
Automatically generate an email or message for follow-up.
Must Haves
If we do a great job summarizing a call, the business owner won’t have to listen to the entire call to know what was said.
If we do a great job surfacing indicators of lead quality, the business owner will be better able to prioritize leads that are more likely to convert.
If we do a great job surfacing customer sentiment, the business owner will have a better understanding of the agent’s performance during the call.
MVP Goal definition
Are you currently using AI in any form to help you with daily operations of your business? If so, how? If not, why?
Prompt crafting
Finally - it was time to put our prompt-writing skills to the test! At this point, we had a much better understanding of what our customers needed: summarization of calls and clear signals of lead quality. We partnered with AssemblyAI and used their LLM to draft and test various prompts.
Using Postman and Assembly’s API documentation (as well as a healthy amount of assistance from our engineering partners), the content strategist and I were able to quickly test and iterate prompts for each of the insights we wanted to surface. At the end of the process, we found that the best prompts were relatively short and clearly articulated both the call context and the expected output format. This minimized hallucinations and delivered the consistent results needed to serve as the foundation of our new product.
👉🏾 For an in-depth look at writing LLM prompts, see my CallRail Labs case study.
Wireframe explorations
Our initial data from FullStory had already given us our starting points: the call log, interaction timeline, and call detail were most likely to yield the highest impact results for our customers. If our ultimate goal was to reduce or remove the need for manual call review, then we needed to target the areas where users spent significant time engaged in that workflow.
Considerations and constraints
Since the “jobs to be done” in each view were different, my goal was to not impede upon the primary function of the view as it existed. The information needed to harmoniously integrate into areas that were already quite visually busy, but also be easy to spot and understand.
03 . Exploration
Tight product scope and the integral nature of these views to customer workflows limited how drastic the design changes could be.
AI experiences needed to feel unified and cohesive across three very different workflows.
-
The call log is a live overview of campaign performance, as well as a log of calls and interactions. Main job to be done: quick review and classification of calls and other interactions for follow-up.
-
The interaction timeline - accessed the via call log - traces the trail of lead interactions from first call through lead qualification, and into conversion.
-
The call detail’s focus on an individual call gave us more flexibility in deciding how much data to display and how it should function and look.
Atomic design principles
Working in tandem with the Design Systems Manager, we were able to bring the product vision to life. Our primary objective was to create a modular system of components that were accessible, expressive, and felt like an extension of the brand palette.
Starting with the atoms we worked out textual, iconographic, and other visual treatments for the sentiment analysis data.
Combining atoms to create rating and progress bars. Colors were chosen that would bring the information to life, help elevate the overall design of the product, and be reflective of the brand.
Premium Conversation Intelligence launched June 2024 with call summaries and sentiment analysis. To maintain speed to market, we kept the project scope tight and saved questions asked for a future iteration.
v 1.0 • Product launch
04 . Product strategy
Driving ROI for small businesses
Our goal was to drastically reduce customer reliance on listening to calls for signs of a high quality lead. We knew from our research that this would allow our clients more time to focus on the core business or grow their client base.
Sentiment column in the call log allows users to filter calls by sentiment, addressing high-quality leads first. Call summary is the default view in the slide-up panel.
Sentiment and call summarization were also added to the timeline, providing valuable insight into the history of a lead's interactions.
From our early research, we knew that the call detail would be an excellent place to drive value for customers. Designing a scalable experience and UI was key.
We followed-up with our initial research groups after launching. Based on their feedback, I focused on improving the visibility and hierarchy of the sentiment analysis insights. We also launched two new AI features - aggregate insights reporting and scheduled emails.
v 2.0 • Feedback and iteration
Customers liked the sentiment rating, but they struggled to understand WHY a call was rated as "positive" or "negative". With Sentiment Spotlight and the new data visualization, we highlighted the phrases that define the overall sentiment rating and clearly illustrated the flow of the call.
We also made subtle visual enhancements to other aspects of the product, with a more consistent content hierarchy across views and more legible graphs.
Sentiment reporting allows customers to drill down into campaign and agent performance by sentiment.
Weekly emails provide the benefit of Premium Conversation Intelligence without needing to log in to the app.
04 . Summary
So…can AI actually drive revenue?
If the great promise of AI is to be a productivity enhancing tool, then the AI-enabled features of Premium Conversation Intelligence delivered. Small businesses with limited staff were able to more effectively address a greater volume of calls, following-up with more high quality leads and converting more customers.
50%
Decrease in time spent qualifying leads.
As calls come in, leads must be sorted by quality. Those with the greatest likelihood of conversion are prioritized as “Qualified Leads”.
31%
Increase in MRR for Conversation Intelligence.
Reduction in churn and greater trial conversions.
60%
Decrease in time spent reviewing calls.
Increased marketing spend results in greater call volume. Manually reviewing calls is time-consuming and often not feasible.
19%
Increase in MRR for Premium Conversation Intelligence subscribers.
Surface high-value leads, trigger follow-up based on keywords that matter to the business.
Milestone 03 • Key results
It’s not just about spending less time reviewing calls. It’s about helping our customers get more from their marketing efforts so that they can do more business. CallRail customers were able to increase revenue without hiring additional staff because they could rely on Premium Conversation Intelligence to surface high quality leads and insights.
AI can be a key driver of growth and revenue with a clear and realistic use case. The tech is still relatively young and requires an understanding of its limitations, as well as its capabilities. AI won’t solve every problem, but it certainly helped our customers focus on their core business and enabled them to market with confidence.
Design drives revenue via research and strategic execution. All too often design is treated as a service within organizations, limiting the ability of designers to deliver high-quality, high-impact work. Involving design from the beginning allowed CallRail to develop a cohesive product strategy and user experience for Premium Conversation Intelligence.
What did we learn? 🧐