Find out how Ada and other leading conversational AI platforms compare on key AI performance metrics.
Enterprises looking to gain an edge in a crowded market and keep up with ever-rising consumer expectations should invest in a conversational AI platform that can instantly understand who the customers are and what they need.
When powered by advanced Natural Language Understanding (NLU) technology, conversational AI is the key to unlocking standout interactions. It makes it as easy as possible for customers to get what they need through automated brand inte ractions and self-service, driving customer loyalty, retention, and revenue.
With so many AI-powered solutions now on the market, knowing what to look for can be a challenge—even for leading global brands. To help lift the veil on conversational AI and NLU for enterprises seeking to transform brand interactions and stay competitive, we ran a benchmarking study comparing how well popular commercial solutions learn and perform.
Before we start, let’s align on some definitions
Technologies that enable interactions between humans and machines, imitating human intelligence and performing tasks previously done by people—faster and at greater scale. Ada’s AI solution automates answers and actions traditionally performed by sales reps or support agents.
A set of algorithms and statistical techniques that enable programs to implicitly learn from patterns in data sets, rather than being explicitly programmed to perform tasks (e.g. rule-based systems). For example, Ada’s platform uses customer inquiry data to learn what content should be served to a customer based on their question.
Natural Learning Understanding (NLU)
A branch of AI that uses algorithmic models to interpret user inputs in the form of text, understand the meaning, and represent it in a way that can be used for different tasks. For example, Ada uses NLU for text classification to discern what a customer is trying to ask or do.
NLU benchmarking study: running the numbers
Our goal at Ada is to help our clients succeed, so...
To benchmark current NLU capabilities, we identified six popular conversational AI platforms and ran an experiment using a multilingual data-set, extracted from Ada's own corpus of utterances, across all platforms in a consistent manner. To ensure accurate and fair results, we made sure Ada's own model wasn't previously trained on these utterances so the results wouldn't be skewed in our favour.
A natural language expression that the user inputs and automation solution must interpret to figure out their intent.
A label or categorization that captures the general meaning of an utterance. In other words, what the customer is trying to ask or do.
How many times the model is correct over the total number of questions asked, shown as the percentage of correctly predicted items. Accuracy is all about being precise.
This is a weighted average between precision (across all scenarios where a certain intent was predicted, how many times did this intent actually apply to the scenario) and recall (across all scenarios where a certain intent should have been detected, how many times was it properly detected). F1 takes intent coverage into account in addition to accuracy.
What we looked at
Based on the needs of our own enterprise customers, we evaluated each platform on four key areas.
English-only intent classification
We tested each platform for accuracy, recall, and F1 score when using NLU to classify consumer intents from English-only text inputs. Scores show what percentage of items were correctly predicted.
Multilingual intent classification
Enterprise companies operating globally require a conversational AI that can understand multiple languages, not just English. To measure this criteria, we tested each solution on intent classification in Spanish and French. Ada was trained with English-only data and inputs/outputs were translated automatically while the other solutions had to be trained individually in the target languages.
Training size performance
The volume of training data required to reach the best recognition rate directly impacts speed to value. We measured each solution's F1 score after training with different data sets ranging in size from two to 10 questions.
This refers to the conversational AI's ability to understand the true meaning behind a customer's inputs when multiple meanings are possible, which directly impacts the customer experience. Also vital to disambiguation is what next steps the conversational AI takes when it's not able to understand the correct meaning.
Conversational AI Flow
For this study, we asked each conversational AI, including Ada's, the same set of test questions and analyzed the results. We wanted to know:
- How precise is the model?
- How accurate is the model?
- How well is the conversational AI able to distinguish a specific intent when multiple answers are possible?
In each of these key areas, our results show that Ada's conversational AI performs better than other platforms, consistently delivering the most human-like automated experience.
Accuracy is everything
Understanding is at the heart of every brand interaction. If your automation solution doesn't know what a customer needs, it's not going to provide a satisfying experience—and that can cost you business.
Results: Ada is the industry leader in both F1 score and accuracy.
Accuracy and F1 score help to make brand interactions more satisfying by preventing customer frustration and accelerating resolution, and Ada’s conversational AI outperformed big-name vendors on both metrics. What these scores mean in practice is that Ada does a better job of understanding your customers, making it quick and simple to get what they need, and keeping them coming back.
Multilingual abilities—speak your customers’ language
With a multilingual platform, brands can expand their reach without opening contact centers across the globe or hiring an army of new agents in every new region. However, having to train a dedicated conversational AI for every new language you want to support is a major barrier to adoption and deployment.
The value an automation solution can provide grows exponentially when it can be trained in one language and deployed in multiple other languages. This is one area where Ada delivers significant time and cost savings.
Results: Ada learns in one language
Ada’s machine translation baseline is either better than or on par with competing NLU solutions that were exclusively trained in the local language, and has the potential to get even better with multilingual training.
While the other solutions in our benchmark study had to be trained in the target language before they could identify intents, we tested Ada on its ability to answer the same set of test questions in Spanish and French after being trained on the data set in English only. Ada’s performance was comparable with the dedicated linguistic models.
This is significant because being able to train a bot in English-only and communicate with customers in multiple languages saves considerable time, effort, and cost.
Training size equals speed to value
When setting up a new conversational AI, the amount of data required to train the AI model directly impacts time to value and ROI. Just like with humans, training AI involves teaching it to correctly interpret the information it receives.
In the case of an automation platform, you start by giving the conversational AI a set of data and then asking it to make decisions based on that dataset. Once it’s completed that basic training, you validate the results by testing its performance with a new set of data.
The less training data an AI model requires to reach proficiency with your chosen utterances and intents, the faster you will start to realize true business value by meeting customer needs.
Ada optimizes performance in this area by using a core model trained across all brands on customer support use cases.
Results: Small training set, lots of learning.
For this experiment, we asked each automation platform the same 10 training questions for each intent and measured the F1 score of its responses. While the other platforms performed decently with minimal training, Ada demonstrated a high level of predictability and understanding.
This is because we use a dedicated global model that was trained with a lot of data specific to CX. What it means for the business is that with Ada, the desired level of accuracy and understanding for each interaction requires less effort to reach than with competitors. And that translates into quicker deployment, faster time to value,
and better performance.
Disambiguation: intent is key
Brands love NLU technology because it allows customers to speak the way they do normally, which makes interactions feel natural and human. But the everyday language customers use is sometimes confusing for the AI model, because the same words can have different meanings depending on the speaker and context.
For conversational AI to work well, it needs to be able to discern the true intent behind a customer's words. And if it runs into trouble—if it can’t figure out what a customer really means—it needs to be able to fail gracefully.
The skill that conversational AI tools use to distinguish a specific intent when multiple meanings are possible is called disambiguation. In today’s market, this is table stakes. What differentiates a conversational AI platform in this area is its level of sophistication and how it deals with failure.
Results: Ada gets what your customers need, no matter what
Picking up the correct intent when user inputs are ambiguous is vital to helping customers reach resolution quickly and easily. That's why we automatically train Ada to know when one answer is sufficient, when two or more answers are relevant, and when it doesn't know what to answer.
Depending on the brand's goals and strategy, Ada can escalate by searching a knowledge base, asking for clarification, or handing off to a support agent directly. This helps customers get results as quickly as possible in any situation.
What makes Ada different?
Ada's conversational AI is empathetic, contextual, and easy to train. Our AI model learns faster and with less effort from your builders, which accelerates time to true business value.
Our proprietary NLU engine understands customers instantly, detecting intent and context even when the conversation includes things like jargon, typos, and incomplete sentences. This makes conversations feel more human while also driving higher resolution rates, increasing containment, and boosting CSAT.
Ada combines the benefits of machine translation with a language-agnostic model, allowing brands to capitalize on pre-existing content while also being able to fine tune locally across many languages. Our language-agnostic model improves multilingual understanding and intent recognition, improving the experience for customers around the world—no matter where they are or what language they speak.
With Ada, brands have the flexibility to train the model directly in local languages or build in one language and deploy in over 40+ others, right out of the box. That means you can expand across borders with greater personalization and customization, without increased cost. Accelerate time to value, boost containment, and expand your reach without hiring new agents.
Time and cost savings
Our language agnostic model reduces the time to train the conversational AI in each new language, lowering effort and expense. And the small training set size Ada learns from means it requires less training to perform well. Save time, lower maintenance costs, and free up your team to focus on optimization—which leads to better performance, more satisfying outcomes, and happier customers.
Ada's industry-leading NLU differentiates brand interactions
Ada's best-in-class conversational capabilities provide the nearest-to-human automated experience in any language. This allows businesses in any industry and of any size to use automation for so much more than answering FAQs. With Ada, you can build the automation-first strategy you need—quickly, easily, and with maximum accuracy. Serve global markets and make customers around the world feel valued. Integrate seamlessly with your existing CRM or other platforms to personalize interactions at scale and handle complex needs with ease. That's the power of our industry-leading conversational AI. With Ada, your brand talks.