Usability Evaluation and Analysis Overview for CCSM CDS Interface

Attachment A: Usability Evaluation and Analysis Overview for CCSM CDS Interface

Overview

This document describes a test plan for conducting the usability evaluation and analysis of the CCSM CDS interface. The goals of usability testing are:

• Determine usability problem areas within the user interface.

• Exercise specific functionality in the interface under controlled test conditions with representative users.

• Establish user-performance and user-satisfaction levels of the interface for future usability evaluations.

• Identify navigation errors in the design which interfere with the user’s flow between functions.

• Evaluate how well the interface communicates recommendations made by the CDS to the end user, and how well the user can understand why those recommendations are being made.

Participants in the usability testing will be asked to provide feedback on key usability heuristics, based on their overall experience with the interface. They will also be given specific tasks to complete and their performance completing these tasks will be used to identify any specific usability issues that can be improved through changes to the user interface or user interactions.

The CCSM CDS development team will review the results to determine if changes need to be made to the interface design, functional behavior of the CDS tool or configuration of the overall system.

Test Plan Summary

Users will receive appropriate instruction and training on the features and elements of the CCSM CDS interface before their participation in usability testing. They will be given the opportunity to familiarize themselves with the interface and to have any questions about the CDS tool or interface answered before starting usability testing. Usability evaluation will be performed at each pilot site, and the results will be collected and analyzed by the CCSM CDS development team.

Methodology

Users will be given a series of specific actions to perform with the interface, using a set of synthetically generated test patients, under controlled conditions. Participants will be asked about their experiences performing the given tasks. They will also be given a form to provide their subjective evaluation of the interface, and to make suggestions they feel would provide improved usability or user satisfaction with the CCSM CDS interface.

Usability Metrics

Evaluation guidelines will cover the main aspects of the interface itself:

• Navigation

Can users easily find all functionality? Does the user know how to move between functions and return to desired points?

• Orientation

Is the user aware of where they are in the process of performing a given task? Are the entry and exit points for a task clear? If they are interrupted, can they pick up where they left off?

• User objectives

Does the interface clearly identify which features will assist the user to achieve their goals?

• Layout

Does the order and grouping of items in the interface make sense to the user? Does the visual hierarchy help the users focus on important content? Are related content and functions placed together?

• Visual Treatment

Is the visual styling of content and interface elements clear and readable? Are any items obscured or difficult to interpret?

• Interface Feedback

Does the interface give appropriate feedback to user actions? Is it clear when a function is disabled and why? Is there clear feedback when an action is successful or has a problem?

• Forgiveness

Can the user explore the interface without causing unintentional errors or problems? Is it easy to recover from an erroneous input or action?

• Language and Terms

Does the interface use language and terminology familiar to users? Are any new terms easily understood by context, labels, or headings?

Usability Tasks

The CCSM CDS tool is designed to run automatically, evaluating a patient’s available medical history and present an appropriate care recommendation based on the rules encoding the latest clinical guidelines. Participants will be asked to identify and compare items in a patient’s medical history with the given recommendation.

Sometimes a patient’s medical history is incomplete or has missing data. Participants will be asked to identify items with incomplete results, and to identify and resolve instances of missing data. The participants’ ability to use the interface to complete these tasks will be evaluated, and their suggestions for improvements will be considered.

Reporting Results

After the completion of usability evaluations, the results will be analyzed, and a summary usability report will be provided. Internal team stakeholders will use this to determine if changes or improvements to the user interface are needed. The summary usability report will be available within 14 calendar days of the completion of the usability evaluation.