In a perfect world, AI would start off working well… perfectly! Sadly though, that’s a rare case.
What usually happens is our bots responses aren’t at the nuanced level a person can provide, yet. So we need to refine, or optimize, its responses so it gets there. A lot of that comes down to the knowledge that we’re providing it.
That’s exactly where AI insights and reporting comes in.
What is AI reporting, and what insights can I get from it?
When it comes to AI, reporting usually involves finding out how well the AI is performing according to what you’re asking it to do. Good AI platforms give you graphs and figures that you can use to investigate any underperformance, and come up with solutions.
Usually, you examine several data points such as:
- Number of replies
- Deflection rate
- Gaps in knowledge/training
- ROI
From here you can produce some AI insights. These are essentially descriptions or assessments that the AI makes for you. For example, a description of exactly why the AI could not generate a great response for a question based on the Confluence Pages you provided. Or, specific examples of what replies were considered deflected and which ones were not.
You should have a clear understanding of your problem, which you can use to come up with possible solutions. In this quick guide, we’ll explain how our AI insights can help fill in knowledge gaps in your Confluence pages.
AI reports for your Confluence AI
In the Reports tab of the dashboard, you can find data about your AI. The two that will help us examine our Confluence pages and any gaps are:
- Gaps in training: identify the percentage of replies where the AI did not have all the necessary information from your Confluence – so you can feed it more information.
- Deflection rate: observe the percentage of replies that did not need any human escalation.
1. Gaps in training – when AI can’t answer a question
Your AI is trained with Sources that you provide it. A Source can be anything from a Google Doc, to a PDF to a Help Center. If you’re here, you likely trained it on your Confluence knowledge.
Just like a person, an AI bot needs to know how to answer a question when a customer asks it.
The gaps in training report shows the percentage of replies where it can’t answer a question, due to it missing a Source, or from that Source (like your Confluence) not being comprehensive enough to cover the question that was asked.
With eesel AI, you simply click on a graph and it takes you to your bot’s history. You can then hover over the ‘gap found’ icon to find a specific question that the AI couldn’t answer.
From here you can see a description of the missing information, and upload a new Confluence page that plugs the gap. Problem solved!
If you’d like to keep a copy of these AI insights, simply export your chats and eesel AI will give you a handy spreadsheet.
2. Deflection rate – when AI can answer a question without human intervention.
Deflections are a good thing. It means your AI has automatically solved the problem of whoever was speaking with it, successfully ‘deflecting’ the issue from coming to your team.
However, sometimes the bot can’t deflect a question. This could be because the question is too complex for the AI to respond to, or requires human intervention to be actioned or answered accurately.
Deflections are different from ‘Gaps in training’. If a bot isn’t deflecting, your Confluence sources need adjusting. Whereas if there’s a gap in knowledge, your bot is probably missing the info all together.
To improve your deflection rate, check your relevant Confluence pages that you’ve imported, and make sure that they’re clear and actually solve the problem your customers are having.
You can find your deflection rate by heading to the reports tab in your dashboard. Here you can see the percentage of replies that have been successfully deflected by the AI.
If you click on the graph, it will take you to the History tab where you can get more details. You can hover over the yes or no deflection icon for a question to see a description of why a question was or was not deflected by the AI.
To keep a copy of these AI insights, export your chats and examine the spreadsheet. You’ll learn whether prompt changes or Source changes within Confluence need to happen.
And now you’re primed in AI insights with Confluence and eesel AI.
There’s more that you can glean from the data AI provides. But this primer has hopefully given you a good understanding of the basics. Pay attention to the charts and figures, and you’ll be optimising your AI knowledge base in no time at all.
If you need more detailed information, please reach out to it@eesel.app.
If you’re someone who learns by doing, you can also sign up to a free 7-day trial for eesel AI here. You might have a little trouble getting enough data to optimize your chatbot in 7 days, but it will give you a good functional understanding of how the app works, and what you can customize once the data starts coming in.
