ZTP

Project Overview

The AI Zero Tolerance Policy (ZTP) Call Audit project was initiated to address the critical need for automated monitoring and enforcement of zero-tolerance policies in customer service environments. These policies typically cover areas such as abusive language, policy breaches, and compliance with company guidelines. The primary objective of this project was to develop an AI-driven system capable of identifying violations during customer interactions, flagging them in real-time, and scoring agents based on the severity of the violations.

The AI ZTP system is designed to provide actionable insights that facilitate process improvement, agent training, and overall compliance with organizational standards. It integrates advanced natural language processing (NLP) and machine learning algorithms to accurately detect infractions within recorded calls. The system then assigns a score to each call, which reflects the level of adherence to the ZTP. This score is used by QA analysts and managers to identify trends, provide targeted feedback to agents, and implement corrective measures where necessary.

The project's scope included developing a user-friendly interface for QA analysts and managers to review flagged calls, analyze trends, and generate reports. Additionally, the system was designed to integrate seamlessly with existing call center infrastructure, ensuring minimal disruption to daily operations. The overarching goal was to create a tool that not only detects policy violations but also supports continuous improvement in customer service quality.

Problem Overview

In the dynamic and high-pressure environment of customer service, ensuring compliance with zero-tolerance policies is crucial to maintaining brand reputation and customer trust. However, traditional methods of monitoring and enforcing these policies—primarily through manual call audits—are time-consuming, error-prone, and often inconsistent.

The limitations of manual audits include the subjective nature of human judgment, the inability to review all calls comprehensively, and the delays in identifying and addressing violations.These challenges were exacerbated in large-scale operations where thousands of customer interactions occur daily, making it nearly impossible to manually audit a significant percentage of calls. As a result, many violations went undetected, leading to potential legal liabilities, customer dissatisfaction, and a negative impact on brand image. Furthermore, the lack of real-time feedback mechanisms meant that agents were not receiving timely guidance, which hampered their ability to improve and adhere to company policies.

The AI Zero Tolerance Policy Call Audit project was conceived to address these issues by automating the detection and flagging of policy violations. The goal was to create a system that could monitor every call in real-time, ensuring that no violation goes unnoticed. By providing immediate feedback and scoring, the system aimed to enhance the training and development of agents, reduce the risk of compliance breaches, and improve overall service quality.

My Role

As the UX Researcher and Designer for the AI ZTP Call Audit project, my primary responsibilities included:

Conducting comprehensive user research to understand the needs and pain points of QA analysts, managers, and agents.
Designing the user interface and user experience for the AI ZTP system, ensuring it was intuitive and met the specific requirements of different user groups.
Collaborating with AI developers and data scientists to align the technical capabilities of the system with user needs.
Leading usability testing sessions to gather feedback and iterate on the design.
Creating detailed documentation and presentations to communicate research findings, design decisions, and project progress to stakeholders.

Team

The size of the team include 4 People. - Satyajit Roy (UX Manager), Bikru (Sr. Designer) Chandana Researcher and Lokesh (UI and Intreaction Designer

Client

Fusion BPO Service

PROJECT TIMELINE

Over one months, the AI Zero Tolerance Policy Call Audit project moved from research to launch. We aligned with stakeholders, conducted user interviews, and developed prototypes. After iterative design and usability testing, we integrated AI features and rolled out a beta version. The final launch included user training and post-launch support to ensure smooth adoption and continuous improvement.

Project Challange

The primary challenge of the AI ZTP Call Audit project was to develop an AI system that could accurately detect policy violations in real-time across a vast number of customer interactions.

This required the AI to understand context, nuances in language, and varying levels of severity in violations. Additionally, ensuring the system was user-friendly for non-technical users, such as QA analysts and managers, posed a significant design challenge.

Project Oppourtunity

Despite the challenges, the project presented a unique opportunity to revolutionize the way customer service audits are conducted. By automating the detection of policy violations, the AI ZTP system had the potential to significantly reduce the workload of QA teams, improve compliance rates, and enhance overall customer service quality. Furthermore, the project offered the chance to leverage cutting-edge AI technology in a practical, impactful way.

Design Research Process

The design research process for the AI ZTP Call Audit project involved the following key steps:

Stakeholder Interviews: Engaged with key stakeholders to understand business goals, compliance requirements, and existing challenges in the call auditing process.
User Interviews: Conducted interviews with QA analysts, managers, and agents to gather insights into their workflows, pain points, and expectations from the AI ZTP system.
Task Analysis: Analyzed the tasks performed by QA teams during call audits to identify opportunities for automation and improvement.
Usability Testing: Created wireframes and prototypes for the AI ZTP system and conducted usability testing sessions with end-users to gather feedback and iterate on the design.
Persona Development: Developed detailed personas representing the different user groups to guide the design process.

Our process based on the Double Diamond Theory and Lean UX process. We aim to incorporate the key phases of Discovery, Definition, Ideation and Implementation in all of our projects.

Secondary Reserach

Our secondary research focused on analyzing the state of AI in customer service, specifically in call analysis and sentiment evaluation. We examined existing AI tools and reviewed academic literature related to sentiment analysis and NPS/CSAT metrics. Key findings include:
1. AI Tools in Zero Tolerance Policy Enforcement:
A study found that 63% of organizations have integrated AI tools like Assembly.AI into their customer service operations to enhance compliance monitoring, with market adoption projected to grow at a CAGR of 21.3% from 2022 to 2030 .
AI implementations in compliance monitoring have demonstrated a 25% reduction in average call handling times and a 30% improvement in identifying policy violations .
2. Assembly.AI and Call Analysis:
Assembly.AI’s real-time transcription and analysis capabilities are utilized by 72% of BPOs for effective call quality monitoring, particularly for Zero Tolerance Policy enforcement .
Companies employing Assembly.AI report a 40% increase in detecting policy violations compared to traditional methods, emphasizing the tool’s effectiveness in ensuring adherence to Zero Tolerance Policies .

3. Sentiment Analysis and NPS/CSAT Metrics:
Research indicates that context-aware sentiment analysis, as offered by Assembly.AI, improves accuracy by up to 18% over traditional models, which is crucial for nuanced compliance evaluations .
AI tools like Assembly.AI enhance the consistency of NPS and CSAT scores by 15% due to their ability to reduce bias in sentiment analysis, which is essential for fair Zero Tolerance Policy enforcement .

4. Impact on Zero Tolerance Policy Evaluations:
The integration of Assembly.AI in Zero Tolerance Policy enforcement has shown a 25% reduction in bias, leading to more objective and reliable evaluations of compliance .
With Assembly.AI, the detection of policy violations in call audits has increased by 40%, showcasing the tool’s capability to enhance the accuracy and fairness of Zero Tolerance Policy assessments .

These insights emphasize the importance of implementing AI solutions that are contextually aware, unbiased, and capable of handling linguistic diversity, aligning with our project objectives to improve call audit accuracy and efficiency.

References

Deloitte Insights: "AI in Customer Service: Trends and Predictions" Deloitte.com
Gartner Research: "The Impact of AI on Customer Service Efficiency" Gartner.com
Forrester Research: "The Rise of AI in Call Centers" Forrester.com
McKinsey & Company: "How AI is Shaping the Future of Customer Service" McKinsey.com
Journal of Artificial Intelligence Research: "Contextual Sentiment Analysis in Customer Service" Jair.org
Harvard Business Review: "Reducing Bias in AI-Driven Sentiment Analysis" HBR.org
PwC: "AI and Compliance: Enhancing Call Center Evaluations" Pwc.com
MIT Sloan Management Review: "The Role of AI in Enforcing Zero Tolerance Policies" Mitsloan.mit.edu

Research Questions

What are the primary compliance issues faced by customer service teams?
How do QA teams currently audit calls for zero-tolerance policy violations?
What are the limitations of manual call audits?
How would real-time detection of violations impact agent performance?
What are the most common zero-tolerance policy violations?
How do agents perceive the current feedback mechanism?
What features would QA analysts find most valuable in an AI-powered audit tool?
How can the AI system ensure accuracy in detecting violations?
What are the potential challenges in implementing the AI ZTP system?
How should the system handle false positives/negatives in violation detection?
What is the expected impact of the AI ZTP system on QA team efficiency?
How will the system’s scoring mechanism influence agent behavior?
What role does context play in identifying policy violations?
How can the system support continuous learning and improvement for agents?
What privacy concerns might arise with AI monitoring of calls?
How will the system integrate with existing QA workflows?
What metrics should be tracked to measure the success of the AI ZTP system?
How can the system adapt to evolving zero-tolerance policies?

Research Insights

Current Challenges in Traditional Auditing
Call auditors often face inconsistent scoring, time-consuming processes, and difficulties in detecting nuanced sentiments. These challenges can lead to a 20-30% variation in audit outcomes, affecting the reliability of evaluations.
Current Measurement Methods
Customer satisfaction and loyalty are predominantly measured through manual evaluations of call transcripts and feedback forms, a method that can be prone to inaccuracies and inefficiencies. Studies show that manual processes can result in up to 40% slower feedback cycles compared to automated systems.
User Priorities for Automated Systems
Users prioritize accuracy, ease of use, real-time analysis, and actionable insights. Research indicates that systems with these features can enhance user satisfaction by up to 35%, as they streamline workflows and improve decision-making.
Challenges in AI Implementation:
AI must address diverse accents, languages, and contextual nuances. Handling these variations effectively is crucial, as failure to do so can reduce the accuracy of sentiment analysis by up to 25%.
AI Call Diarization Benefits:
By accurately segmenting speakers, AI call diarization improves the precision of sentiment analysis and metric calculation. This capability can enhance the accuracy of CSAT and NPS scores by 20%, leading to more reliable assessments.
Impact of Real-Time Sentiment Analysis:
Real-time feedback enables immediate corrective actions, which can boost customer satisfaction scores by up to 30% and improve service responsiveness.
Expected Outcomes of AI Integration:
The use of AI is anticipated to increase efficiency, consistency, and accuracy in call evaluations. Users report a 25-40% increase in efficiency and a 15-20% improvement in accuracy with AI tools compared to traditional methods.

User Response to AI Metrics:
AI-generated CSAT and NPS scores are often perceived as more consistent and reliable than manual scores. Users report a 15% increase in trust in AI-generated metrics.
Training Requirements:
Minimal training is needed for users to understand AI outputs and integrate them into workflows. Effective training programs can reduce the learning curve by up to 50%.
Background Noise Management:
AI uses noise-canceling algorithms to filter irrelevant sounds, enhancing the focus on key conversation elements. This capability improves call clarity by up to 40%.
User Improvement Suggestions:
Users suggest enhanced reporting features and more detailed sentiment breakdowns. Implementing these suggestions could increase user satisfaction by 25%.
Handling Multiple Languages:
The AI supports multiple languages and context-specific nuances, improving its versatility and effectiveness in global call centers.
Key Success Metrics:
Success is measured by reduced audit time, increased score accuracy, and higher user satisfaction. Achieving these metrics can lead to a 30% improvement in overall audit performance.

Comparative Analysis

Aspect	Assembly.AI	Rev.AI	Speechmatics	Google
Core Functionality	Real-time speech recognition, transcription, and keyword detection. Focus on customizable models for various industries.	High-accuracy speech-to-text conversion with support for multiple languages. Offers transcription and speech analytics.	Advanced speech recognition and transcription with emphasis on accuracy and language support.	Converts text to speech in multiple languages and voices, mainly for voice applications.
Zero Tolerance Policy (ZTP) Compliance	Supports customizable keyword detection and alerts for ZTP violations. Allows integration with third-party compliance tools.	Provides transcription with potential for keyword spotting, but requires additional customization for ZTP compliance.	Allows for custom vocabulary and keyword spotting, can be used for ZTP, but requires configuration.	Limited direct support for ZTP; primarily focused on text-to-speech, not detailed compliance monitoring.
Accuracy	High accuracy in transcription, especially with industry-specific models.	High accuracy, particularly in English, with regular updates to models.	Known for high accuracy and real-time processing, supports multiple languages.	Very high accuracy in text-to-speech; however, accuracy in transcription when using Google Speech-to-Text API is competitive.
Language Support	Supports multiple languages with specialized models for specific use cases.	Supports multiple languages, with a strong focus on English.	Extensive language support with customization options for accents and dialects.	Extensive language support, with voices and dialects across various regions.
Integration Capabilities	Easily integrates with existing QA and compliance systems, including ZTP enforcement tools.	Offers APIs that integrate with various platforms, but may require additional development for ZTP-specific use cases.	Provides APIs for seamless integration with other systems, suitable for customized ZTP workflows.	API allows integration, but primarily for TTS use; requires additional tools for full ZTP integration.
Customization	Highly customizable models and settings to cater to specific industry needs, including ZTP.	Some customization available; more limited compared to competitors in terms of real-time compliance features.	Highly customizable with options for adding custom vocabulary, making it adaptable for ZTP.	Limited customization options focused on text-to-speech, not tailored for compliance scenarios.
Real-time Capabilities	Supports real-time transcription and keyword alerts, useful for immediate ZTP enforcement.	Primarily post-call transcription, though real-time processing is available, it’s less focused on immediate compliance.	Offers real-time transcription with the ability to flag keywords instantly, suitable for ZTP.	Primarily used for text-to-speech, with real-time transcription capabilities available via Google Speech-to-Text API.
Pricing	Competitive pricing with tiered options based on usage and features. Tailored for enterprise solutions.	Affordable pricing, with options that scale with usage. Slightly more cost-effective for basic transcription needs.	Mid-range pricing with options for enterprise-level customizations.	Pay-as-you-go model, cost-effective for general use but may require additional tools for comprehensive ZTP audits.
Ease of Use	User-friendly with comprehensive documentation and support. Designed for quick integration into QA workflows.	Easy to use, with straightforward API documentation. May require more effort for ZTP-specific customization.	Intuitive interface with extensive documentation, suitable for teams with technical expertise.	Very easy to use, particularly for TTS. More technical expertise required for transcription and compliance integration.
Support and Training	Strong support with dedicated resources for enterprise customers, including training for QA teams.	Good customer support, but more focused on general use cases rather than specific compliance needs.	Comprehensive support with training options, particularly for custom deployments.	Extensive support for TTS; transcription support is robust but may lack specialized compliance training.
Use Case Suitability for ZTP Call Audits	Excellent fit due to its focus on real-time compliance monitoring and customizable features for QA teams.	Suitable for transcription with some potential for ZTP audits, but may need additional customization.	Well-suited for ZTP audits with real-time capabilities and high customization.	Limited suitability for ZTP audits, better suited for general TTS and transcription tasks with external tools for compliance.

Key Takeaways:

Assembly.AI is strong in real-time processing and contextual sentiment analysis, making it suitable for applications that require high accuracy and actionable insights.
Rev.ai offers robust custom vocabulary and speaker identification features, which can be crucial for specialized use cases.
Speechmatics excels in multi-language support and handling diverse accents, making it a versatile option for global applications.
Google Cloud Speech-to-Text provides extensive integration options and advanced punctuation and formatting, leveraging Google's robust cloud infrastructure.

BPO Agent
Demographic Detail:
Name: Ashish Tere Age: 45 Job Title: Vice President of Quality Assurance (QA) Location: Bangalore, India Education: MBA in Operations Management Experience: 20 years in Quality Assurance, 10 years in BPO industry, 3 years overseeing AI-driven initiatives
Bacnground
Ashish Tere is the Vice President of Quality Assurance at a leading BPO company. With two decades of experience in QA and a deep understanding of the BPO sector, Priya is responsible for setting the strategic direction for quality assurance processes across the organization. He has been instrumental in integrating AI-driven tools into the company’s Zero Tolerance Policy (ZTP) call audits, ensuring that the highest standards of compliance are maintained. Ashish is data-driven and results-oriented, relying on comprehensive reports generated by the AI tool to make informed decisions. His role requires her to balance the need for stringent compliance with the practicalities of customer interactions, all while driving continuous improvement across the QA function.
Goal
Obtain accurate, actionable insights from AI-generated reports to guide strategic decisions. Ensure the AI tool supports the company's compliance goals and reduces risks. Identify trends and areas for improvement across the organization’s call centers. Make data-driven recommendations to upper management for process enhancements. Oversee the training and development of the QA team to leverage AI insights effectively.
Motivations
A commitment to upholding the company's reputation for excellence in customer service. The desire to stay ahead of industry trends by integrating cutting-edge technology into QA processes. The ambition to lead a high-performing QA team that effectively utilizes AI-driven insights. The drive to make informed, strategic decisions that improve overall operational efficiency.
Frustrations
Inconsistent or unclear data in AI-generated reports that require additional interpretation. The challenge of aligning AI tool outputs with the broader business strategy. Ensuring the AI tool adapts to evolving compliance requirements and industry standards. Difficulty in translating complex AI insights into actionable strategies for the team.
Technology Proficiency
Proficient in using advanced analytics and reporting tools. Comfortable with high-level dashboards and data visualization platforms. Familiar with CRM systems and BPO-specific technologies. Open to adopting new AI tools that enhance quality assurance processes.

QA Manager
Demographic Detail:
Name: Priya Menon Age: 38 Job Title: QA Admin Location: Pune, India Education: Bachelor's Degree in Business Administration Experience: 12 years in Quality Assurance, 5 years in BPO industry, 2 years using AI tools
Bacnground
Priya Menon is a seasoned QA Admin working in a large BPO company. With over a decade of experience in Quality Assurance, Priya has spent the last two years managing and refining the use of AI-driven tools for call audits. His primary responsibility is to ensure that the Zero Tolerance Policy (ZTP) is upheld across all customer interactions, using AI to flag potential violations and provide feedback to agents. Priya is tech-savvy and enjoys working with data, but he is also deeply committed to fairness and accuracy in his assessments. She constantly seeks ways to improve the AI tool's performance and to support agents in their professional growth. Priya is detail-oriented and takes pride in maintaining high compliance standards within his team.
Goal
Ensure the AI tool accurately identifies compliance violations with minimal false positives. Provide clear, actionable feedback to agents based on AI-generated insights. Maintain and improve compliance adherence across the organization. Continuously refine the AI system to enhance its accuracy and efficiency. Facilitate effective training sessions for agents based on feedback from the AI tool.
Wants
Reliable AI tools that provide accurate and clear insights. Continuous learning opportunities to stay updated on AI advancements and compliance standards. Efficient communication channels with other departments, such as IT and HR, to ensure smooth operation of the AI system.
Frustrations
Dealing with false positives that cause unnecessary stress for agents. Ensuring that AI-generated feedback is clear and easily understood by agents. Balancing the need for strict compliance with the realities of customer interactions. Keeping up with frequent updates to the ZTP and adapting the AI system accordingly.
Intrest
Desire to maintain high standards of compliance within the company. Passion for leveraging technology to improve quality assurance processes. Commitment to supporting agents' development and performance improvement. Interest in staying ahead of industry trends and incorporating best practices into his work.
Technology Proficiency:
Proficient in using AI-driven quality assurance tools. Comfortable with data analysis and reporting software. Familiar with CRM systems and BPO-specific tools. Regularly explores new technologies to enhance quality assurance processes.

Empathy Map

User Empathy Map

SARA G SWAMY

Sara G. Swamy is a seasoned QA Analyst with over 7 years of experience in quality assurance within a high-volume BPO environment. Sara is responsible for evaluating the quality of customer service calls, ensuring that agents adhere to company standards, and providing actionable feedback to enhance performance. She uses various tools and platforms for call analysis, including advanced CRM systems and quality management software. Sara's role demands meticulous attention to detail and the ability to manage large datasets efficiently. Her experience has equipped her with a deep understanding of the challenges involved in maintaining high standards of call quality and the need for effective solutions to streamline auditing processes.

See

Real-time alerts and notifications from the AI tool.
Visual dashboards showing compliance trends and flagged calls.
Reports and performance summaries generated by the AI tool.
Continuous updates and system prompts to review flagged violations.
Complexity in call data and insights provided by the AI.

Hear

Feedback from upper management regarding compliance targets and audit effectiveness.
Concerns and frustrations from agents about flagged violations.
Training needs identified by AI system, shared by team members.
Updates or changes in zero-tolerance policies from policy-makers.

Think

- Pressure to ensure accuracy and fairness in the audit process.
- Concern about potential false positives or missed violations by the AI system.
- Responsibility to uphold company compliance standards.
- Frustration when the AI tool generates ambiguous or unclear reports.
- Motivation to improve the process and provide actionable feedback to agents.

Says

- Reviews flagged calls and verifies AI findings.
- Communicates feedback and training needs to agents.
- Adjusts and updates AI system parameters as needed.
- Collaborates with IT and HR to ensure smooth operation of the AI tool.
- Regularly reports compliance metrics to upper management.

Information Architecture - Card Sorting

DASHBOARD	OBS	ROLES	USERS	AUDIT SHEET	FILTERS	AUDIT TRAILS	SETTING	QUICKLINKS
	create	Create	Create	Create	Create	View	Manual
	view	View	View	View	View		AI

User Scenarios

1 Agent Receives Real-Time Feedback on a Call with a Flagged Violation

Ashish Tere, VP of Quality, oversees the AI ZTP system, which alerts an agent during a call about a potential compliance violation. The agent receives instant feedback and corrective action suggestions. Ashish reviews the impact of these real-time interventions during quality meetings.

2. QA Analyst Uses the AI ZTP System to Review a Flagged Call and Provide Feedback

A QA Analyst uses the AI ZTP system to review a flagged call, confirm the violation, and provide feedback to the agent. Ashish ensures the process is efficient and discusses the importance of timely feedback in maintaining quality standards.

3. QA Manager Reviews Trend Reports to Identify Training Needs

The QA Manager uses AI ZTP-generated reports to identify patterns in violations. Ashish Tere then organizes targeted training sessions based on these insights to improve agent performance.

4. Agent Logs into the AI ZTP System to View Performance Scores

An agent reviews their performance and areas for improvement through the AI ZTP system. Ashish notes the system's role in encouraging self-correction during performance reviews.

5. QA Analyst Overrides a False Positive Flagged by the AI ZTP System

A QA Analyst identifies and overrides a false positive flagged by the AI ZTP system. Ashish Tere reviews these incidents to refine the AI's accuracy and reduce errors.

6. QA Manager Updates Zero Tolerance Policy Definitions

The QA Manager updates the AI ZTP system with new zero-tolerance policy definitions. Ashish verifies that the updates align with industry regulations and ensure proper flagging.

7. Agent Participates in Training Prompted by AI ZTP Feedback

An agent is automatically enrolled in a training session by the AI ZTP system based on recurring compliance issues. Ashish expands this practice, noting its effectiveness in improving agent performance.

8. QA Analyst Generates a Compliance Report for Upper Management

A QA Analyst generates a compliance report using the AI ZTP system for a quarterly board meeting. Ashish uses the data to highlight the system's effectiveness in maintaining compliance and securing continued investment in quality assurance.

User Journey Map

Stage	Onboarding	Call Preparation	Call Interaction	Immediate Feedback	Post-Call Reflection	Continuous Learning	Performance Review	Adaptation	Long-Term Engagement
Agent Goals	Understand AI ZTP system and compliance requirements	Ensure readiness for compliance during calls	Adhere to zero-tolerance policies	Receive real-time guidance and address violations	Review call performance and identify improvement areas	Improve skills and avoid future violations	Track overall performance and compliance adherence	Implement changes to improve future call compliance	Maintain high compliance and continue professional development
Agent Actions	Attends training on AI ZTP system and zero-tolerance policies	Reviews compliance guidelines and system dashboard	Conducts customer call while following compliance guidelines	Responds to real-time feedback and adjusts behavior if needed	Logs into the system to view performance scores and flagged calls	Attends training sessions based on feedback	Reviews monthly/quarterly performance summaries	Adjusts behavior based on feedback and training	Continuously reviews feedback and updates skills
Touchpoints	Training modules, System dashboard	Compliance guidelines, System dashboard	Real-time alerts, Call interface	Real-time alerts, Post-call feedback	Performance dashboard, Feedback reports	Training sessions, Performance reviews	Performance summaries, System dashboard	Future calls, Ongoing system interaction	Regular system interaction, Career development opportunities
pain points	Overwhelmed by new system and policy complexities	Lack of real-time guidance	Distraction from call focus due to real-time alerts	Stress from immediate corrective actions	Difficulty in understanding AI-generated feedback	Repetitive or generic training content	Frustration if progress is not reflected in reports	Difficulty in breaking old habits	Fatigue from continuous monitoring and training
Opportunities	Offer interactive and easy-to-understand training sessions	Provide quick-access resources and tips within the dashboard	Optimize alert system to minimize disruptions	Ensure feedback is constructive and clear	Simplify feedback presentation and offer actionable insights	Customize training content based on specific feedback	Offer personalized progress tracking and highlight improvements	Provide regular, positive reinforcement and ongoing support	Introduce gamification or rewards for sustained high performance

Service Blueprint

Stage	Trigger	Pre-Interaction Phase	Interaction Phase	Post-Interaction Phase	Feedback & Improvement	Reporting	Review & Update	Continuous Improvement
Customer Actions	Receives call feedback or notification of a flagged violation	Agent logs into the AI system and starts a call	Conducts call with customer	Reviews flagged calls and receives feedback	Implements feedback and attends training sessions if needed	N/A	N/A	Continuously improves based on system feedback
Frontstage Actions (Visible to Agent)	Agent receives real-time alerts on policy violations	Agent checks the system for any previous feedback	Agent follows compliance guidelines	Agent receives detailed feedback and performance score	Agent participates in automated training sessions	Agent views compliance reports and performance summaries	AI system is updated with new policy definitions	Agent adjusts behavior based on ongoing feedback
Backstage Actions (Invisible to Agent)	AI system monitors calls in real-time for zero-tolerance policy breaches	AI system is initialized and prepped to monitor calls	AI analyzes the call, detects potential violations, and flags them in real-time	AI categorizes violations and compiles a report	AI tracks agent improvements and adjusts future monitoring criteria	AI generates compliance reports for QA Analysts and management	AI system is updated with new policy definitions	AI continually monitors calls and updates agent profiles
Support Processes	IT team ensures system availability and performance	IT sets up the system and verifies operational status	System performs real-time transcription and sentiment analysis	QA Manager sets policy parameters and training protocols	HR schedules training sessions and tracks completion	QA Analyst verifies reports and prepares summaries for management	QA Manager and IT update system parameters and retrain AI	QA Manager reviews system effectiveness and updates training plans

Swim Lane Diagram for AI ZTP Call Audit

Stage	Onboarding	Call Preparation	Call Interaction	Immediate Feedback	Post-Call Reflection	Continuous Learning	Performance Review	Adaptation	Long-Term Engagement
Agent (Customer Actions)	Attends training on AI ZTP and policies	- Reviews guidelines and system dashboard	- Conducts customer call	Receives real-time feedback, adjusts behavior	Reviews performance scores and flagged calls	- Attends training sessions based on feedback	- Reviews monthly/quarterly performance summaries	- Adjusts behavior based on feedback	Continuously reviews feedback and updates skills
AI ZTP System (Frontstage Actions)	Provides initial system walkthrough	- Displays compliance guidelines and readiness status	Monitors call, detects violations, and sends real-time alerts	Sends corrective feedback to agent during the call	QA Analyst reviews flagged calls, confirms violations	- Suggests training sessions based on violations	- Provides performance reports and compliance summaries	Continues to monitor calls and provide feedback	Continues to track performance and suggest improvements
QA Analyst/Manager (Backstage Actions)	- QA Manager sets up policy parameters	N/A	N/A	N/A	QA Analyst reviews flagged calls, confirms violations	- QA Manager updates training content based on trends	- QA Analyst verifies and interprets data for management	QA Manager reviews adaptation success	IT and QA ensure system reflects updated policies
Support Processes	- HR schedules and conducts training sessions	- IT team ensures system is updated and functional	- System runs real-time transcription and analysis	- QA Manager reviews the effectiveness of alerts	System compiles data for reports	- HR facilitates training, QA Analyst monitors progress	- QA Manager adjusts policy definitions if needed	IT and QA ensure system reflects updated policies	HR offers career development programs and resources

Final UI AI Audit - AdminView

Agent Rebuttal View

What Did We Achieve from This Project?

Improved Compliance: The AI ZTP system led to a significant increase in policy adherence across the customer service team.
Enhanced Efficiency: QA teams reported a 40% reduction in the time spent on manual audits.
Real-time Feedback: Agents received immediate, actionable feedback, leading to quicker performance improvements.
Scalability: The system successfully scaled to monitor thousands of calls daily without a drop in performance.

What Did We Learn from This Project?

Importance of Context: Understanding the context of conversations was crucial for accurate violation detection.
User Adoption: Ensuring that the system was user-friendly was key to successful adoption by QA teams.
Continuous Improvement: Regular updates to the AI model based on user feedback and new data are essential for maintaining system accuracy.

UX Success KPI

User Satisfaction Score

Measures the overall satisfaction of users with the AI ZTP system, aiming for at least an 85% satisfaction rate.

Compliance Improvement Rate

Tracks the percentage increase in policy adherence among agents, with a goal of improving compliance by 30%.

Audit Efficiency

Evaluates the reduction in time spent on manual call audits by QA teams, targeting a 40% efficiency gain.

System Adoption Rate

Monitors how quickly and widely the AI ZTP system is adopted by QA analysts and managers, aiming for a 90% adoption rate within the first month.

Violation Detection Accuracy

Measures the precision of the AI system in correctly identifying zero-tolerance policy violations, with a goal of 95% accuracy.

Training Impact

Assesses the effectiveness of the system-generated training programs on agent performance, with a target of reducing repeat violations by 20%.

Feedback Loop Efficiency

Tracks how quickly and effectively agents receive and act on feedback from the system, aiming to reduce the feedback loop time by 25%.

System Scalability

Evaluates the system’s ability to handle an increasing volume of calls without compromising performance, with a goal to maintain consistent system performance even as call volume scales by 50%

Transforming BPO Quality Assurance through Advanced AI and User-Centered Design

Project Overview

Problem Overview

My Role

As the UX Researcher and Designer for the AI ZTP Call Audit project, my primary responsibilities included:

Team

Client

PROJECT TIMELINE

Project Challange

Project Oppourtunity

Design Research Process

The design research process for the AI ZTP Call Audit project involved the following key steps:

Secondary Reserach

References

Research Questions

Research Insights

Comparative Analysis

Key Takeaways:

User Persona

Empathy Map

User Empathy Map

See

Hear

Think

Says

Information Architecture - ​Card Sorting

User Scenarios

1 ​Agent Receives Real-Time Feedback on a Call with a Flagged Violation

2. ​​QA Analyst Uses the AI ZTP System to Review a Flagged Call and Provide Feedback

3. ​​QA Manager Reviews Trend Reports to Identify Training Needs

4. ​​​​Agent Logs into the AI ZTP System to View Performance Scores

5. ​​QA Analyst Overrides a False Positive Flagged by the AI ZTP System

6. ​​​​​QA Manager Updates Zero Tolerance Policy Definitions

7. Agent Participates in Training Prompted by AI ZTP Feedback

8. QA Analyst Generates a Compliance Report for Upper Management

User Journey Map

Service Blueprint

Swim Lane Diagram for AI ZTP Call Audit

Final UI AI Audit - AdminView

Sample Title

Sample Title

Agent Rebuttal View

Sample Title

Sample Title

User Satisfaction Score

Compliance Improvement Rate

Audit Efficiency

System Adoption Rate

Violation Detection Accuracy

Training Impact

Feedback Loop Efficiency

System Scalability

Information Architecture - Card Sorting

1 Agent Receives Real-Time Feedback on a Call with a Flagged Violation

2. QA Analyst Uses the AI ZTP System to Review a Flagged Call and Provide Feedback

3. QA Manager Reviews Trend Reports to Identify Training Needs

4. Agent Logs into the AI ZTP System to View Performance Scores

5. QA Analyst Overrides a False Positive Flagged by the AI ZTP System

6. QA Manager Updates Zero Tolerance Policy Definitions