Technical Report
NLU Benchmarking Study
Find out how Ada and other leading conversational AI platforms compare on key AI performance metrics.
What we looked at
Based on the needs of our own enterprise customers, we evaluated each platform on four key areas.
English-only intent classification
We tested each platform for accuracy, recall, and F1 score when using NLU to classify consumer intents from English-only text inputs. Scores show what percentage of items were correctly predicted.
Multilingual intent classification
Enterprise companies operating globally require a conversational AI that can understand multiple languages, not just English. To measure this criteria, we tested each solution on intent classification in Spanish and French. Ada was trained with English-only data and inputs/outputs were translated automatically while the other solutions had to be trained individually in the target languages.
Training size performance
The volume of training data required to reach the best recognition rate directly impacts speed to value. We measured each solution's F1 score after training with different data sets ranging in size from two to 10 questions.
Disambiguation support
This refers to the conversational AI's ability to understand the true meaning behind a customer's inputs when multiple meanings are possible, which directly impacts the customer experience. Also vital to disambiguation is what next steps the conversational AI takes when it's not able to understand the correct meaning.
Conversational AI Flow
Experimental setup
For this study, we asked each conversational AI, including Ada's, the same set of test questions and analyzed the results. We wanted to know:
- How precise is the model?
- How accurate is the model?
- How well is the conversational AI able to distinguish a specific intent when multiple answers are possible?
In each of these key areas, our results show that Ada's conversational AI performs better than other platforms, consistently delivering the most human-like automated experience.
Accuracy is everything
Understanding is at the heart of every brand interaction. If your automation solution doesn't know what a customer needs, it's not going to provide a satisfying experience—and that can cost you business.
Results: Ada is the industry leader in both F1 score and accuracy.
Accuracy and F1 score help to make brand interactions more satisfying by preventing customer frustration and accelerating resolution, and Ada’s conversational AI outperformed big-name vendors on both metrics. What these scores mean in practice is that Ada does a better job of understanding your customers, making it quick and simple to get what they need, and keeping them coming back.
Multilingual abilities—speak your customers’ language
With a multilingual platform, brands can expand their reach without opening contact centers across the globe or hiring an army of new agents in every new region. However, having to train a dedicated conversational AI for every new language you want to support is a major barrier to adoption and deployment.
The value an automation solution can provide grows exponentially when it can be trained in one language and deployed in multiple other languages. This is one area where Ada delivers significant time and cost savings.
Results: Ada learns in one language
Ada’s machine translation baseline is either better than or on par with competing NLU solutions that were exclusively trained in the local language, and has the potential to get even better with multilingual training.
While the other solutions in our benchmark study had to be trained in the target language before they could identify intents, we tested Ada on its ability to answer the same set of test questions in Spanish and French after being trained on the data set in English only. Ada’s performance was comparable with the dedicated linguistic models.
This is significant because being able to train a bot in English-only and communicate with customers in multiple languages saves considerable time, effort, and cost.
Training size equals speed to value
When setting up a new conversational AI, the amount of data required to train the AI model directly impacts time to value and ROI. Just like with humans, training AI involves teaching it to correctly interpret the information it receives.
In the case of an automation platform, you start by giving the conversational AI a set of data and then asking it to make decisions based on that dataset. Once it’s completed that basic training, you validate the results by testing its performance with a new set of data.
The less training data an AI model requires to reach proficiency with your chosen utterances and intents, the faster you will start to realize true business value by meeting customer needs.
Ada optimizes performance in this area by using a core model trained across all brands on customer support use cases.
Results: Small training set, lots of learning.
For this experiment, we asked each automation platform the same 10 training questions for each intent and measured the F1 score of its responses. While the other platforms performed decently with minimal training, Ada demonstrated a high level of predictability and understanding.
This is because we use a dedicated global model that was trained with a lot of data specific to CX. What it means for the business is that with Ada, the desired level of accuracy and understanding for each interaction requires less effort to reach than with competitors. And that translates into quicker deployment, faster time to value,
and better performance.
Disambiguation: intent is key
Brands love NLU technology because it allows customers to speak the way they do normally, which makes interactions feel natural and human. But the everyday language customers use is sometimes confusing for the AI model, because the same words can have different meanings depending on the speaker and context.
For conversational AI to work well, it needs to be able to discern the true intent behind a customer's words. And if it runs into trouble—if it can’t figure out what a customer really means—it needs to be able to fail gracefully.
The skill that conversational AI tools use to distinguish a specific intent when multiple meanings are possible is called disambiguation. In today’s market, this is table stakes. What differentiates a conversational AI platform in this area is its level of sophistication and how it deals with failure.
Results: Ada gets what your customers need, no matter what
Picking up the correct intent when user inputs are ambiguous is vital to helping customers reach resolution quickly and easily. That's why we automatically train Ada to know when one answer is sufficient, when two or more answers are relevant, and when it doesn't know what to answer.
Depending on the brand's goals and strategy, Ada can escalate by searching a knowledge base, asking for clarification, or handing off to a support agent directly. This helps customers get results as quickly as possible in any situation.
What makes Ada different?
Ada's conversational AI is empathetic, contextual, and easy to train. Our AI model learns faster and with less effort from your builders, which accelerates time to true business value.
Near-human understanding
Our proprietary NLU engine understands customers instantly, detecting intent and context even when the conversation includes things like jargon, typos, and incomplete sentences. This makes conversations feel more human while also driving higher resolution rates, increasing containment, and boosting CSAT.
Multilingual training
Ada combines the benefits of machine translation with a language-agnostic model, allowing brands to capitalize on pre-existing content while also being able to fine tune locally across many languages. Our language-agnostic model improves multilingual understanding and intent recognition, improving the experience for customers around the world—no matter where they are or what language they speak.
With Ada, brands have the flexibility to train the model directly in local languages or build in one language and deploy in over 40+ others, right out of the box. That means you can expand across borders with greater personalization and customization, without increased cost. Accelerate time to value, boost containment, and expand your reach without hiring new agents.
Time and cost savings
Our language agnostic model reduces the time to train the conversational AI in each new language, lowering effort and expense. And the small training set size Ada learns from means it requires less training to perform well. Save time, lower maintenance costs, and free up your team to focus on optimization—which leads to better performance, more satisfying outcomes, and happier customers.
Ada's industry-leading NLU differentiates brand interactions
Ada's best-in-class conversational capabilities provide the nearest-to-human automated experience in any language. This allows businesses in any industry and of any size to use automation for so much more than answering FAQs. With Ada, you can build the automation-first strategy you need—quickly, easily, and with maximum accuracy. Serve global markets and make customers around the world feel valued. Integrate seamlessly with your existing CRM or other platforms to personalize interactions at scale and handle complex needs with ease. That's the power of our industry-leading conversational AI. With Ada, your brand talks.
Automation That Makes Things Happen
Give customers what they want before they ask. Make proactive suggestions. Take real action. Go beyond automated FAQs. With Ada, automation does more.
