Understanding Zendesk AI Reporting and Analytics

AI in customer support relies on continuous training, evaluation, and improvement. However, your ability to do so means that having access to essential reporting information and analytics is fundamental. Zendesk has limited analytics capabilities for its AI, making it difficult to judge what works and what doesn’t work.

We’re going to cover what kind of analytics Zendesk has available, as well as how to use them and any common problems users experience.

Keep in mind that some of these analytics will apply to Zendesk Advanced AI features, which is a paid add-on, rather than the basic Zendesk AI or Generative AI. You can find more information about what the difference is here, and what the pricing for each looks like here.

What types of analytics are available?

Some analytics are quite simple and straightforward. Others are more involved, specifically analytics attached to the Advanced AI add-on. There are extra filtering features involved in some tools, but it’s up to the user to be able to utilize these datasets appropriately and recognize any potential confounding variables that impact results.

1. Generative AI agent tools dataset and prebuilt dashboard: This is Zendesk’s first AI-related dataset, offering comprehensive insights into the usage of generative AI agent ticket tools, including summarize, expand, and tone shift features. It comes with a prebuilt dashboard for easy visualization and analysis.
2. Intelligent triage predictions and confidence reports: These reports focus on the predictions made by the intelligent triage feature, including intent, language, and sentiment analysis, along with their respective confidence levels.
3. Agent engagement analytics: These analytics specifically track how agents are utilizing the AI tools, providing detailed information on usage frequency and patterns across the support team.
4. Ticket metrics related to AI tool usage: These metrics allow for comparison between tickets where AI tools were used and those where they weren’t, helping to quantify the impact of AI assistance on key performance indicators.

What information do they provide?

Generative AI agent tools dataset and prebuilt dashboard:

Metrics:

• Ticket count with AI tools usage: This metric quantifies how many tickets have been processed using any of the AI tools, giving a clear picture of AI tool adoption.

First reply time spent: This is the median time between ticket submission and first reply, allowing teams to assess if AI tools are speeding up initial responses.

Full resolution time: This measures the median time from ticket creation to full resolution, helping to evaluate if AI tools are reducing overall ticket handling time.

AI tool usage over time: This tracks the frequency of AI tool usage across different time periods, allowing for trend analysis.

Attributes:

• Agent name: Identifies which agents are using the AI tools.
• Ticket ID: Allows for drill-down into specific tickets for detailed analysis.
• Interaction ID: Enables tracking of individual interactions where AI tools were used.

Insights provided:

• Detailed view of how agents are utilizing AI tools across the support operation.
• Impact assessment of AI tools on critical metrics like resolution time, CSAT, and requester wait time.
• Ability to compare performance between AI-assisted and non-AI-assisted tickets.

Intelligent triage predictions and confidence reports

Shows:

• Intent predictions: What the system believes is the primary purpose or goal of the customer’s inquiry.
• Language predictions: The detected language of the customer’s message.
• Sentiment predictions: The perceived emotional tone of the customer’s message.
• Confidence levels: How certain the system is about each of these predictions.

This allows support teams to:

• Assess the accuracy and effectiveness of the intelligent triage feature.
• Identify areas where the triage system may need improvement or additional training.
• Understand how well the system is interpreting customer communications.

Agent engagement analytics

Provides data on:

• Frequency of AI tool usage per agent: How often each agent is leveraging AI assistance.
• Total number of times AI tools were used by each agent: Quantifies the overall adoption of AI tools by individual agents.
• Distribution of AI tool usage across the agent team: Helps identify top adopters and those who may need additional training or encouragement.

This information helps managers:

• Identify which agents are most effectively utilizing AI tools.
• Spot agents who may need additional support or training in using AI tools.
• Understand overall team adoption of AI technologies.

Ticket metrics

Compares metrics for tickets with and without AI tool usage:

• First reply time: Assesses if AI tools are helping agents respond more quickly to initial inquiries.
• Full resolution time: Evaluate if AI tools are contributing to faster overall ticket resolution.

This comparison allows support teams to:

• Quantify the impact of AI tools on ticket handling efficiency.
• Identify which types of tickets benefit most from AI assistance.
• Make data-driven decisions about expanding or refining AI tool usage.

Where to find the reports and how to use them?

Generative AI Agent Tools dashboard

• In Explore, click the Dashboards icon (represented by a visual icon in the interface).
• Navigate to the Dashboards library page.
• Select Zendesk AI > Generative AI Agent Tools.
• Click on the desired tab (e.g., Agent engagement or Ticket metrics).

Custom intelligent triage predictions report

• In Explore, click the reports icon and select “New report”.
• Choose Support > Tickets dataset as the data source.
• Add relevant metrics such as Ticket ID, Intent confidence, Language, and Sentiment.
• Apply necessary filters, such as excluding NULL values for Intent and setting an appropriate date range (e.g., This week).
• Save the report for future access and analysis.

Optional filter for intelligent triage:

• Open any report in the same dataset.
• Create a Standard calculated attribute via the Calculations menu.
• Name the filter appropriately and enter the required formula (specific formula not provided in the given information).
• Use this newly created attribute to filter reports, showing only tickets that have been enriched by intelligent triage predictions.

What to do with the information?

Having lots of fancy charts and numbers won’t do much on their own. But it’s important to regularly check and compile this information for several reasons. The most important of which is to make sure your AI usage is both helpful and cost-effective, as it can become very expensive when not monitored.

Measure AI tool adoption

• Use agent engagement analytics to identify high and low adopters of AI tools.
• Set benchmarks for AI tool usage and track progress over time.
• Recognize and reward agents who effectively incorporate AI tools into their workflow.

Assess AI tool impact

• Compare resolution times and CSAT scores for tickets with and without AI tool usage.
• Quantify the time savings and customer satisfaction improvements attributable to AI assistance.
• Use this data to justify further investment in AI technologies or to refine existing tools.

Optimize workflows

• Identify which AI tools (e.g., summarize, expand, tone shift) are most effective in reducing resolution times or improving CSAT.
• Develop best practices based on the usage patterns of top-performing agents. Observe and ask how they best utilize AI in their workflow and help other agents do the same.
• Encourage the use of high-impact AI tools across the entire agent team. Some might be hesitant to start using AI if they’re unfamiliar with it.

Evaluate intelligent triage accuracy

• Analyze confidence levels for intent, language, and sentiment predictions.
• Identify patterns in low-confidence predictions to improve the triage system.
• Use this information to refine the intelligent triage system or provide additional training data where needed.

Track performance over time

• Monitor AI tool usage trends and their impact on key metrics over weeks or months.
• Identify any seasonal patterns or trends in AI tool effectiveness.
• Use this longitudinal data to make informed decisions about expanding or modifying AI tool implementation.

Justify AI investment

• Compile data showing improvements in efficiency and customer satisfaction due to AI tool usage – or dissatisfaction.
• Calculate whether AI has produced cost-saving or productivity improvements.

Enhance training programs or look for alternatives

• Use insights from top AI tool users to develop training materials for other agents.
• Identify common scenarios where AI tools are most effective and incorporate these into agent onboarding and ongoing education.
• Decide whether the metrics and analysis are enough for your team’s needs, and consider alternatives.

By thoroughly analyzing these reports and acting on the insights gained, support teams can make data-driven decisions to improve their use of AI tools, enhance agent performance, and ultimately provide better, more efficient customer service.

At eesel AI, we want you to have full control over assessing and adjusting how AI works for your company. We know that being able to accurately evaluate how AI impacts your customer satisfaction, employee productivity, and your bottom line, is essential. We provide more value, more customization, and more reporting features, at a fraction of the price.

We believe it’s worth making informed decisions, so hopefully our free trial and our discussion about Zendesk’s features helps you implement AI in the way that works best for you.