Today, every company is working on incorporating AI into its ethos. We made this prediction a few years ago, and sure enough, we were right. 

For customer service, this means transitioning from an agent-first to an AI-first model, where the nucleus of the service organization is the AI and leaders build around it. Your customers are expecting this too. Zendesk’s research shows that 73% of consumers expect more interactions with AI in their daily life, and 74% believe AI will improve customer service efficiency.

In a few years, AI and automation will supplant human agents almost entirely, and human customer service agents will instead be elevated to a more strategic and important role in the company. This foundational shift is naturally accompanied by a shift in how we measure success. Agent-first metrics will continue to be relevant — albeit adapted to a new paradigm — but a new metric will become the North Star for customer support: Automated Resolutions (AR).

Automated Resolutions are conversations between a customer and a company that are safe, accurate, relevant, and don't involve a human.

As you embark on becoming an AI-first customer service organization, it’s crucial to ensure that you are onboarding solutions from AI-native companies; companies who are not just selling AI software, but employing it themselves internally in day-to-day processes across the entire organization. This is the kind of dedication that ensures the product is truly purpose built for an AI-first world.

To demonstrate this concept, I’m going to dive into how we built Ada’s AI to optimize for AR. When a customer makes an inquiry and the AI generates an answer, Ada runs the generation through 3 filters to ensure that every conversation is safe, accurate, and relevant. Let’s dig into them.

Safety filter

The first step is identifying whether the generated answer is safe. Safe means that the AI interacts with the customer in a respectful manner and avoids engaging in topics that cause danger or harm.

Ada currently incorporates OpenAI’s safety tools to achieve this.

Accuracy filter

The next step is ensuring that the generated answer is accurate. Accurate means that the AI provides correct, up-to-date information with respect to the company’s knowledge and policies.

When thinking about generative AI for customer service, you have to consider the knowledge sources that the AI is using to generate answers. When AI is powered by publicly available LLMs, it distills information from all over the internet.

While this is more or less okay for general use, it becomes much riskier for customer support, especially if it gives customers answers that are offensive or factually incorrect. Depending on the severity of the misinformation, your company can become liable for legal or financial reparations. 

For Ada’s generative capabilities, we’ve limited the knowledge source that the AI distills information from to only the company’s first-party support documentation. By training the AI to use only the company’s knowledge base, we can ensure that the generated answer is grounded on the company's official documents and therefore mitigate a lot of risk.

When the AI generates an answer, Ada runs an Elasticsearch index to search for and retrieve the most relevant documents from the company’s knowledge base that potentially contain the information needed. Then the LLM gets to work. 

Among other methods, Ada uses a BERT-style natural language inference model that’s trained on pairs of sentences that have a target binary flag, where sentence B is logically entailed from sentence A. Ada runs the model on each of the documents and the generated answer in order to determine the probability that the generation is logically entailed from any of these documents. 

Relevance filter

It’s not enough for a generation to just be safe and accurate, it also needs to be relevant. Relevant means that the AI effectively understands the customer’s inquiry, and provides directly related information or assistance. Without this, the AI might be only generating truthful facts that are unrelated to the customer’s inquiry, and doesn't really resolve anything.

The relevance filter uses a question-answer, BERT-style model that’s trained on Ada’s data, which means the training data set is specific to and optimized for the customer service use case. 

The data consists of:

• Half a million positive pairs of training questions and answers, where the question is matched with the right answer
• One million negative pairs of training questions and answers, where the question is matched with the wrong answer

The model was fine tuned to consider the generated answer in relation to the original question and determine a probability regarding how relevant it is.

Successful generation

Companies can set probability threshold values to indicate how confident the AI needs to be before serving the generated answer back to the customer, and can set separate thresholds for the accuracy and relevance filters. The higher the value, the higher the confidence. This comes in handy for sensitive questions with lower risk tolerances, relating to banking or health for example.

The generated answer that meets the threshold values on both filters is the one that gets served back to customers.

But keep in mind that higher value thresholds could potentially mean that the AI is less likely to be confident enough to serve back an answer, and may ask the customer to rephrase the question or hand them off to an agent. So for lower risk questions, it’s better to have more moderate threshold values.

Either way, you can rest assured that the generated answer will be safe, accurate, and relevant, and it will bring you one step closer to realizing your AR potential.