2022 - 2024 • CallRailDriving revenue with Premium Conversation Intelligence
Artificial Intelligence / B2B SaaS / Growth
CallRail is an industry-leading call tracking and analytics platform that helps businesses monitor marketing campaign performance across phone numbers.
Project duration
~18 months
Role
Product design, Research
Team
Product manager, Lead Designer, Content strategist, Engineers (x5)
Tools
Figma/FigJam, Fullstory, Looker, Postman
The Problem
As customers scaled their marketing efforts, the increasing call volume overwhelmed their ability to identify quality leads and created a growth ceiling.
Solution
To understand the call content and assess lead quality, customers relied on manually reviewing calls by either listening to the audio or reading the transcript. Leveraging artificial intelligence to analyze customer conversations at scale allowed me to surface a call summary and sentiment analysis for a call or lead.
Aggregate reporting further reduced reliance on manual call review by summarizing a week of calls and provided greater visibility into trends and outliers over time. Scheduled emails met the uer wherever they were, extending the value of the product beyond the application.
Business Impact
-60%
Reduction in call review time
Accurate call summarization meant customers could quickly understand what a call was about without reading an entire transcript or listening to a call.
+10%
Increase in lead-to-close rate
Sentiment analysis provided a clear signal regrading the quality of a call and/or lead, aiding in the prioritization of high-quality leads for follow-up.
+19%
Increase in MRR for subscribers
Aggregate insights reporting surfaced qualified leads and agent performance, allowing customers to focus on business operations instead of reviewing calls.
Research & Discovery
How might we reduce customer reliance on manual call review and surface crucial caller and conversation insights?
Market context and opportunity
The conversation analytics market was projected to reach $1.8B by 2023, with most solutions targeting enterprise call centers. Our research identified a significant gap for SMBs who needed enterprise-level insights without enterprise complexity or cost. This aligned perfectly with CallRail's strategy to increase ARPU through premium features while addressing competitive pressure from emerging conversational intelligence tools.
Understanding customer behavior
I analyzed 150+ user interactions with call data using FullStory and conducted interviews with 24 small businesses of with fewer than 25 FTE employees. After distilling quantitative analysis data and qualitative experiences from customers, we came away with valuable information and several key insights, including:
91 mins - Customers spent an average of one-and-a-half hours per day reviewing calls for signs of high-quality leads.
73% - The majority of customers stated that reviewing calls was the second-most time consuming activity after maintaining business operations.
Segment demographics • 24 total interviews
6
5
4
3
6
Home services
Real estate
Legal
Medical
Marketing agencies
120 mins transcribed
Customers needed to have at least two hours of calls transcribed per day.
Fewer than 25 employees
Small businesses with fewer than 25 full-time employees were prioritized.
Heuristic evaluation
Call Log & Slide-up Panel
Combining Jakob Nielsen’s heuristics for usability and visual design with customer feedback and behavior analysis gave us the framework necessary to conduct a thorough usability audit.
The data table in the Call Log presented an opportunity to provide simple signals regarding call outcome and lead quality. Since it was already home to the call transcript, the slide-up panel provided an area we could utilize to surface summaries.
Design Consideration: The Call Log is a foundational piece of the product architecture and experience, so major overhauls to the experience or interface were off the table.
Pain points
Manual call review meant clicking each call and either listening to the audio or reading the transcript.
Too few signals indicating lead quality, meaning potential missed conversions.
Opportunities
Accurately summarize individual calls, removing the need to listen to or read transcript for the entire call.
Surface more insights indicating lead quality, customer interactions and call outcomes.
Call Detail
The focused nature of the Call Detail gave us the opportunity to explore how we might surface more in-depth insights about a call and/or caller.
Design consideration: Use the audio waveform and transcript to build a scalable foundation for future in-depth call analysis.
Pain points
No aggregate insights across conversations, limiting the value customers could derive from transcripts.
Manual call review mostly required customers to spend time in the CallRail app. However, many small business owners were constantly on the go or analyzed with call data through aggregate reports.
Opportunities
Summarize multiple calls, surfacing trends and outliers across data.
Meet the customer where they are and extend value beyond the product.
Customer personas
Motivation: Cares about growing their customer base, as well as being responsive to current customers. Reluctant to hire additional staff due to increased operational overhead.
Behavior: Engages with CallRail frequently, using the Call Log to check campaign health, review calls, and prioritize leads for follow-up.
Pain Points: Lacks time to listen to each call individually. Worried that they may miss calls from potential new leads.
Design considerations: Needs easy access to operational insights to streamline process and develop targeted training for employees.
Small Business Owner
Sole proprietor, marketing is secondary to running the business.
Motivation: Cares about campaign impact. Works primarily with spreadsheets to generate reports for clients.
Behavior: Rarely interacts with the CallRail application, preferring instead email summaries, reporting, and raw data exported as a CSV file or via API.
Pain Points: Frustrated by lack of clear signals indicating campaign or channel effectiveness, limiting their ability to drive lead generation for current clients. Wants aggregate insights to better identify trends and outliers in campaign performance.
Design considerations: Needs analysis of changes in call volume or sentiment before and after marketing campaigns.
Marketer
Freelancer or a part of an agency. Marketing is their business.
Motivation: Cares about customer retention and providing a high quality support experience.
Behaviors: Engages with CallRail daily. Appreciates clear messaging on the outcome of a call.
Pain Points: Frustrated by lack of clear signals indicating caller sentiment. Wants to quickly understand call outcomes and if there is further follow-up required.
Design considerations: Needs information on common issues and customer satisfaction levels.
Customer Service Agent
Employed by SMB, customer satisfaction is the primary concern.
Prompt crafting
Using Postman and Assembly’s API documentation (as well as a healthy amount of assistance from our engineering partners), the content strategist and I were able to quickly test and iterate prompts for each of the insights we wanted to surface. We found that the best prompts were relatively short and clearly articulated both the call context and the expected output format. This minimized hallucinations and delivered the consistent results needed to serve as the foundation of our new product.
Design for AI-Enabled Humans
As I explored visualizing the LLM output, I prioritized translating the data into a clear and concise format. This served our goal of making the insights easy to understand, speeding up call review.
Further Design explorations
Working from feedback generated by the usability studies, our primary goal was to facilitate quick call and lead assessment.
Confidence ratings were removed from sentiment, as customers found it confusing and decreased trust in the sentiment rating.
For summaries, we removed “The Gist” and headline, as they often duplicated content found in the summary without providing additional context or value.
Product Definition
After several rounds of customer interviews, design exploration, and usability testing we were able to prioritize features that would deliver value without expanding scope.
Phase 1:
Call summaries, basic sentiment analysis visualization
Phase 2:
Sentiment analysis timeline and highlighting, customer/agent sentiment breakdown, aggregate insights
Phase 3:
Questions asked, appointment booked, sale made, call coaching
To validate our hypothesis, I led usability testing using prototypes to better understand if our proposed solutions were truly helping our customers understand call content and the likelihood of a lead to convert.
Figure: Prioritization framework
-
Technical
The engineering team identified processing limitations that would impact real-time sentiment analysis. Together, we decided to focus on post-call analysis first, with questions asked and real-time features being explored for a future iteration.
Design
Additionally, the integral nature of these views to our customers Jobs To Be Done limited how drastic the design changes could be. Redesigns of the interface or experience were off the table.
-
This research phase involved close collaboration with:
Design Lead & Product Management: Define feature scope and alignment with overall AI strategy.
Engineering & ML: Engineering helped evaluate processing requirements for potential real-time sentiment analysis. AssemblyAI refined sentiment detection algorithms based on our visualization needs.
Sales & Customer Support: Validating value proposition with existing customers.
Marketing: Aligning with go-to-market messaging strategy via Pendo and other channels, pricing strategy.
Explorations
Can AI deliver the insights needed to automate call review and drive customer revenue?
Design Goal: Quick Lead Assessment
Adding sentiment to the data table unlocked advanced filtering options, facilitating lead qualification and follow-up.
Combining advanced sentiment filtering and accurate call summarization gave our customers the tools they needed to more efficiently address increasing call volume.
Design Goal: Accelerate Call Review
Call summaries were added to the slide-up panel, replacing the transcript as the default view. This minimized friction and allowed easy access to accurate call summaries. To further accelerate call review and provide quick access to in-depth conversation analytics, a link to the call detail was added.
Design Goal: In-Depth Conversation Analysis
The singular focus of the Call Detail gave us the opportunity to expand the sentiment analysis visualization. Taking advantage of Atomic Design principles allowed me to create a scalable foundation for visualization.
Solutions
Quick wins were prioritized for launch. Call summaries and sentiment analysis filters automated much of the call review workflow.
Initial Impact
17%
Conversion from trial in launch quarter
Incorporating sentiment analysis into the call log gave customers the ability to filter by call quality, providing more advanced filtering options and surfacing a crucial signal about the potential likelihood of a lead to convert.
-50%
Decrease in time spent qualifying leads.
The combination of call summaries and sentiment analysis drastically simplified the call review workflow by facilitating the quick assessment of call content and lead quality, without the time-consuming task of listening to a full call.
Refined Content Hierarchy Supports Quick Review
Based on customer feedback, we refined the visual design of the slide up panel and redesigned the sentiment indicator to increase visibility. We also added sentiment ratings per speaker. This allowed us to better support our goal of quick assessment.
Customer Journey and Call Outcome
Expanded sentiment analysis visualization gave customers deeper insight into call outcomes. The sentiment timeline highlights key moments in the conversation, and is linked to the transcript. Clicking a key moment in the timeline spotlights that section of the transcript, providing crucial context.
Meet the Customer Where They Are
Aggregate insights reporting accurately summarizes a week of calls, surfacing commonly asked questions and highlighting trends and outliers in campaign performance. Automated email summaries extend value beyond the product, supporting customers who heavily utilize reports and data export.
Product Evolution
Building on customer feedback gathered after launch, we introduced expanded sentiment visualization and aggregate insights reports.
Business Impact
Customers were able to more effectively address a greater volume of calls, following-up with more high quality leads while increasing sales and appointments booked.
+10%
Increase in lead-to-close rate.
Caller sentiment proved to be a crucial tool when prioritizing leads for follow-up. The addition of sentiment data to the call log allowed for advanced filtering, surfacing more high quality leads.
+31%
Increase in MRR for core Call Tracking product.
Insights surfaced with the help of machine learning drove revenue for CallRail’s customers, offering a compelling reason to upgrade from trial and reducing churn overall.
-60%
Decrease in time spent reviewing calls.
Applying machine learning to the call review workflow was a game-changer for our customers. By accurately summarizing calls and surfacing caller sentiment, we provided the information needed to significantly reduce the amount of time spent reviewing calls.
+19%
Increase in MRR for Premium Conversation Intelligence subscribers.
Small business owners could maintain focus on the core business without worrying about missing potential leads. Marketers and customer service agents could more accurately assess campaign performance and customer satisfaction levels.
It’s not just about spending less time reviewing calls. It’s about helping our customers get more from their marketing efforts so that they can do more business. CallRail customers were able to increase revenue without hiring additional staff because they could rely on Premium Conversation Intelligence to accurately summarize calls and surface high quality leads and insights.
AI can be a key driver of growth and revenue with a clear and realistic use case. The tech is still relatively young and requires an understanding of its limitations, as well as its capabilities. Artificial Intelligence won’t solve every problem, but it certainly helped our customers focus on their core business by automating repetitive and time-consuming tasks.
Design for AI-enabled humans. Artificial Intelligence doesn’t add value alone. It’s how we design for machine learning enabled features and the humans that use them that makes the difference. Understanding how user mental models change - and the resulting evolution of UX patterns - will determine if customers actually realize value.
What did I learn? 🧐