IVR Caller Experience Metrics. World IA Day 2013 March 2

Transcription

1 IVR Caller Experience Metrics World IA Day 2013 March 2

2 What is Caller Experience? Processes and Design Principles that put the caller at the center of attention, to ensure that an IVR is: Useful Usable Aesthetically Pleasing 2

3 3 Dimensions of Design Design To Fit Needs Determine Caller Needs Engaging Artful Clever Pleasant Make Obvious Reduce Errors Save Time 3

4 Equation for Caller Experience Measure P(Successful Caller Experience) = P(Useful) x P(Usable) x P(Aesthetically Pleasing) Equation is more about people than technology Even if the caller experience is useful, it will not succeed if people cannot use it or do not like it.

5 3 Key Processes for Caller Experience 1 Know the Caller 2 Employ Iterative Design 3 Evaluate Appropriately

6 Caller Needs Analysis 1 General More Specific Very Detailed Design Requirements Demographics - age, education, location Call Environment - home, car, office Concurrent Activities - walking, driving, working Needs get something done Wants quickly and reliably complete Enjoys no what ifs Exact procedures - step-by-step Automation delegation - the caller acts, the system responds How to think - mental models

7 Iterative Design 2 & Analysis 3 Requirements High Level Design Compare to needs analysis, standards, lessons learned, best practices, etc. Detailed Design Heuristic Evaluations Usability Testing Building the functional pieces, placement, order Dialog design matches persona, error recovery, transitions, natural traversals UI peers reviewing the design different peers find different issues Compare caller performance & perceptions/preferences to CE Thresholds

8 Caller Experience Design Principles Consistency Error recovery Cultural norms Obviousness Menu option grouping Caller-controlled pace Flexible order of actions Consider caller s environment Consider caller s other activities Familiar language & terminology Differentiate confusable functions and commands Know human cognitive and auditory limits Allocate tasks that people are better at to people, and tasks machines are better at to machines Fit technology to people, not visa versa Minimize what must be learned Build in performance support including reminders & examples Keep it simple

9 Measuring Caller Experience Best Success Performance (Success on tasks) Caller Experience Score Perceptions/Preferences (Opinion Ratings) Best Rating

10 Measuring Performance Performance Success - metric for how usable a product is. Behavioral Data - measuring speed and accuracy collected by having callers do tasks. Scenario-Based Tasks - information-gathering or problemsolving descriptions are commonly used. Design Performance Data - compared to target thresholds for performance.

11 Performance Examples Task Scenario Examples Task 1: Take money from your checking account and deposit it into your savings account. Task 2: Report a death in the family and identify what needs to be done to collect benefits. Threshold Examples Gold Medal Performance: 90% of participants perform each task error-free. Silver Medal Performance: 80% of participants perform each task error-free.

12 Performance Success (%) Summarizing Performance Performance is summarized for each task. 100 Performance on each task can be compared to acceptance thresholds. 90 Gold Medal (outstanding) Overall performance is measured by averaging across all measured tasks. This number summarizes how usable the product is Silver Medal (acceptable) Task 1 Task 2 Task 3 Overall

13 Measuring Perceptions/Preferences Opinions form the metric for callers Perceptions and Preferences. Questions are asked to determine opinions about usefulness, usability, and aesthetics. A Numerical Scale with anchoring verbal labels is used to make the opinions quantifiable. Deductive interviewing is used to probe callers on opinions and why is that data?

14 Perceptions/Preferences Examples Useful example The system was missing some important features Strongly Disagree Strongly Agree Aesthetics example The system voice was pleasant like an agent that I might speak with Strongly Disagree Strongly Agree Usable example I think most people would learn to use this system quickly Strongly Disagree Strongly Agree Threshold Examples Gold Medal Preference: Average rating is 4 or higher. Silver Medal Preference: Average rating is 3 or higher.

15 Perception/preferences Ratings Summarizing Perceptions/Preferences Perceptions/Preferences are summarized for the three dimensions of design, usefulness, usability and aesthetics. Ratings for each dimension can be compared to acceptance thresholds Gold Medal (outstanding) Silver Medal (acceptable) An overall rating can be obtained by averaging across the three dimensions. This number is the Perceptions/Preferences opinion rating. 2 1 Useful Usable Aesthetics Overall

16 Caller Experience Score Matrix GOLD MEDAL CALLER EXPERIENCE SILVER MEDAL CALLER EXPERIENCE Performance Success (%) (Combined results for all measured tasks) Qualitative Ratings (Combined opinions about usefulness, ease of use and aesthetics) The Caller Experience Score Matrix combines the quantitative data (callers performance) with the subjective data (callers perceptions/preferences) into a single summary statistic about the overall Caller Experience.

17 Caller Experience Score Matrix Examples GOLD MEDAL CALLER EXPERIENCE SILVER MEDAL CALLER EXPERIENCE Performance Success (%) (Combined results for all measured tasks) Life Auto & Home Disability Dental Life Auto & Home The Caller Experience Score Matrix combines the quantitative data (callers performance) with the subjective data (callers perceptions/preferences) into a single summary statistic about the overall Caller Experience. Qualitative Ratings (Combined opinions about usefulness, ease of use and aesthetics) Dental Disability

18 Questions?

19