Project Overview

The health insurance company was about to release an online guided process that allows healthcare practitioners to submit medical authorization requests on behalf of patients. I convinced the project team to incorporate usability testing into the project plan before release, which led to improved ease of use.

In my role as the user experience researcher, I collaborated with team members to create medically-realistic test scenarios, prepared the test scripts and questions, and facilitated the usability tests with end-users. Following the usability study, I explained the 32 discovered usability issues and gave redesign recommendations in a written report. I also facilitated the team discussion of the discovered usability issues and next steps. If the originally proposed solutions were not technical feasible, then we discussed other solutions until each issue could be addressed. The development team has implemented 20 of the recommendations, which has helped healthcare practitioners more accurately and easily submit authorization requests.

Summary of Usability Test Outcomes

• Verified that the start page gave all the necessary information to include in an authorization request
• Verified that users could easily search for members and select the correct individual from the search results
• Provided definitions for terminology that users found confusing
• Revised field labels to help users submit the correct information
• Clarified the process to start the authorization request
• Explained whose contact information should be given on certain steps
• Updated the visual styling of the tabs to show they’re clickable

Please note: Sensitive information about this project has been removed.

Team Members

• Product owner
• Behavioral health network manager
• Behavioral health network coordinator
• Business analyst
• Project manager
• 2 Developers
• User experience researcher (myself, Cynthia)

Process

1. Convince the project team to incorporate usability testing
2. Help team members recruit the appropriate participants
3. Create medically-realistic test scenarios and answer keys
4. Prepare questions and notetaker’s template for usability test
5. Facilitate usability test sessions
6. Write report of usability test findings and recommendations
7. Facilitate team discussion of results and next steps to address usability issues

Convince the Project Team to Incorporate Usability Testing

The project team was planning to release the newly developed authorization request guided process to the healthcare practitioners and had never considered conducting a usability test before.

To persuade the project team to incorporate usability testing, I shared and discussed the potential benefits, including:

• Learn if any of the terminology or phrases are confusing or if they are understandable
• Fewer calls to the provider services help line
• Fewer authorization requests submitted with errors
• More satisfied healthcare practitioners
• Learn if the steps in the authorization tool make sense to the users

Considering the benefits of the usability testing, the project team agreed and were excited to my joining the team to conduct the research study.

Help Team Members Recruit the Appropriate Participants

None of my team members had ever heard of usability tests before so I wanted to ensure they knew what kind of participants would be appropriate to recruit.

Our criteria for our test participants included:

• Healthcare practitioner
• Has submitted authorization requests to this healthcare insurance company before
• Has not seen or used the current online authorization request guided process yet
• Would be willing to do a one-hour remote usability test session

Create Medically-Realistic Test Scenarios and Answer Keys

To help the end-users more realistically interact with the authorization guided process during the usability tests, I collaborated with team members and sought the assistance of individuals with medical knowledge outside of our immediate project team to create and review the test scenarios that would be medically appropriate for each user group. This would help us receive more accurate feedback from the test participants.

To write the medical inpatient test scenarios, we worked with a care manager familiar with hospital admissions. For the medical outpatient test scenarios, a care manager for skilled nursing facilities assisted us. For the behavioral health test scenarios, a care manager specializing in behavioral health situations helped us.

Also, I created an “answer key” for each test scenario with the help of the above individuals to understand when participants were completing the steps correctly during the test sessions. Thus, if I saw they were entering something incorrectly, I would be able to know when to immediately ask follow-up questions to better understand the participant’s thought process.

Prepare Questions and Notetaker’s Template for Usability Test

Key Questions

There were certain features and elements of the guided process that the team specifically wanted to ask the participants about during the usability tests. I prepared unbiased questions to inquire the users’ thoughts of these features. Here are a sample of the prepared questions:

• Does this page contain all the information you would expect to request an authorization?
• Do you agree with the choices that appear? Are there any options you think are missing or should not be here?
• What does “rendering provider” mean to you?
• What does “referring provider” mean to you?
• Whose contact information do you think these telephone and fax number fields are for?
• Does this process of searching for a diagnosis code or description make sense to you? Would you want to change anything about this?
• Overall, how hard or easy was searching and selecting the diagnosis code? (1 = Very hard, 7 = Very easy)
• What would you expect to see on the next screen?

Notetaker’s Template

The notetaker’s template contained an outline of all the test scenarios and questions I planned to ask during the test session. The notetaker during each test session would fill in this document in real-time to more efficiently capture key points of the test session. This saved me time when I was analyzing the usability test results later on so that I could rely less on the video recordings of the sessions.

Facilitate Usability Test Sessions

I facilitated each of the 7 usability tests remotely with online screensharing software. Remote test sessions were more convenient for our participants who would have otherwise needed to travel to our office.

At the beginning of each test session, I explained what the test participant should expect during the session and made sure they felt comfortable to give their honest feedback. Then I gave each person a test scenario that was appropriate for that healthcare practitioner’s field and asked them to think aloud to share their decision-making thoughts as they used the system. Each usability session lasted about an hour. Each participant also received test data  for specific steps so they did not need to enter any real patient’s personal information.

While I was facilitating the study, one person from our project team functioned as the notetaker. As I conversed with the participant and asked follow-up questions, the notetaker would record the participant’s responses and behavior.

Write Report of Usability Test Findings and Recommendations

Through my written usability test report, I enumerated the 32 discovered usability issues and gave suggested solutions with design mockups to give the development team more guidance. The issues were organized by severity level, “High”, “Medium”, and “Low”.

Due to the sensitive information in the written report, this document is not available online, although I discuss parts of it in the following sections. If you wish to learn more about it, please feel free to contact me.

Key Findings of Usability Test

• The introduction page gave all the necessary information to include in an authorization request. Users considered this information sufficient
• Users were able to easily search for members and then select the correct individual from the search results
• Some terminology was confusing or unclear. Even when users selected the correct option for their test scenario, they were not always confident of their selection
• Users misunderstood the meaning of certain fields and unknowingly submitted incorrect information

Sample of Discovered Usability Issues

I’ve listed a select number of usability issues discovered during the study here. The entire list is not available online and parts of the following screenshots have been redacted to protect sensitive information.

Usability Issue: It’s unclear how to begin the authorization request process

To start the process, users need to click the “Begin” button. However, all users who saw this screen tried to click on the top left “Submit a request” tab instead, thinking that was a button instead of the active tab. Two participants also tried clicking on the “My requests” tab to begin, but this leads to a view of previously submitted requests. Despite the initial confusion, all users eventually understood they need to click on the “Begin” button.

Usability Issue: Some of the options in the “Place of Service” dropdown were confusing

Some participants did not fully understand the terminology of certain options in the “Place of Service” dropdown. Even when they selected the correct option for their test scenario, they were not fully confident of their selection.

Usability Issue: The field labels “Start Date” and “End Date” were unclear

One participant did not know if “Start Date” referred to a pre-authorization request or to a regular authorization request. Another participant did not understand if “End Date” referred to the last day of the authorization or the day when the patient no longer needs their services.

Usability Issue: The section labels “Rendering Provider” and “Referring Provider” were unclear

Multiple participants did not understand which provider to put for the “Rendering Provider” and which for the “Referring Provider”. Some expressed confusion at the difference between these two.

Usability Issue: Searching for a provider or facility can be very difficult

Some users were not able to find their own provider or facility they work for in the search results. They did not know that the search terms they entered were too restrictive and that they could have entered fewer search terms or even only part of the facility’s name. Part of this difficulty was due to the search engine’s strict algorithm.

Usability Issue: Users entered the wrong person’s contact information

On the Rendering Provider and Referring Provider steps, users were asked to provide contact information, but over half of the test participants incorrectly entered their own contact information rather than the provider’s contact information. Most of these users did not even know they were submitting the wrong contact information.

Usability Issue: Users did not notice the Provider and Member tabs on the Review step

On the final step, the user can review all the details of their request in the default tab, “Authorization Request” as well as the “Provider” and “Member” tabs. However, some users overlooked the “Provider” and “Member” tabs and did not realize they could click on them. Thus, they did not fully review their request before submitting it.

Facilitate Team Discussion Of Results and Next Steps to Address Usability Issues

After I shared the usability test report with team members and stakeholders, I held a team meeting to go over the results and proposed recommendations to address each usability issue. Each recommendation was discussed with the developers. If the originally proposed solutions were not technical feasible, then we discussed other solutions until each issue could be addressed. Of the 32 discovered usability issues, 20 of them were addressed with development updates in the first release of the authorization request guided process.

Key Improvements to Authorization Request Guided Process

• Clarified terminology and field labels that users found confusing
• Clarified the process to start the authorization request
• Explained whose contact information should be given on certain steps
• Updated the visual styling of the tabs to show they’re clickable

Sample of Resolved Usability Issues

I’ve listed the resolutions the team implemented to address the selected number of usability issues discussed above. The entire list of resolved issues is not available online and parts of the following screenshots have been redacted to protect sensitive information.

Usability Resolution: Clarified how to begin the authorization request process online

To ensure that users could more easily understand how to begin the process, we changed the styling of the tabs so that it became more obvious that the active tab is an active tab instead of a button. We also changed the label of that active tab from “Submit a new request” to “Welcome”, which sounds less like an action. Thus, users would be less likely to get confused and know to click the “Begin” button.

Usability Resolution: Defined the terminology in the “Place of Service” dropdown options

My initial solution was to provide an information icon next to the “Place of service” dropdown that the user could click on to view definitions for all the options. However, even before I had joined this team, my teammates, business stakeholders, and developers already had plans to add a help link to each step of the process. Clicking on this help link would show specific information for the current step.

For the current step shown above, the help link gave definitions for all of the “Place of Service” dropdown options so that users could accurately and confidently select the correct choice.

Usability Resolution: Clarified the “Start Date” and “End Date” field labels

To ensure there would be no confusion around the “Start Date” and “End Date” field labels, we changed them to “Start Date of Service” and “End Date of Service”, respectively.

Usability Resolution: Explained the meanings of “Rendering Provider” and “Referring Provider”

We also provided definitions of the “Rendering Provider” and “Referring Provider” so that users would know the appropriate provider to assign to each role, and the authorization request could be more accurate.

Usability Resolution: Added search tips and examples to help users search for a specific provider or facility

To help users search for the correct provider or facility and receive relevant search results, my first recommendation was to alter the search engine algorithm to be less restrictive. However, due to the project timeline and technical restrictions, this solution was not possible.

Thus, the team decided to proceed with my second recommendation: Give search tips and examples of successful search terms to the user, such as entering only some of the letters of a provider or facility’s name instead of entering the entire name. For example, “If your facility name is ‘Mount Sinai New York’, then try searching for ‘Mou’ in the facility name field.”

Usability Resolution: Explain whose information should be given for the contact information fields

On the Rendering Provider and Referring Provider steps, additional language was added to clarify whose contact information should be given. We changed “Please provide the following contact information” to “Please provide the following contact information for the Referring Provider” or “Please provide the following contact information for the Rendering Provider” for the their respective steps.

Usability Resolution: Updated the visual styling of the Provider and Member tabs on the Review step to show they are clickable

To help users see that the “Provider” and “Member” tabs are interactive, we changed the visual styling so that they look more like clickable tabs. Thus, users would be more likely to verify all of their authorization request information before submitted them. Furthermore, we also added a “Edit this information” link to each section to give users a quick way to revise any details from previous steps.

Lessons Learned

I learned that it can be beneficial to regularly schedule even 15 minutes of debrief time immediately after each usability test. It’s a chance for team members to discuss and compare observations during the study while the research is still fresh on our minds.

I also learned how valuable it was for the product owner to attend and observe so many of the usability test sessions. Her first-hand witness of the participants’ struggles encouraged her to be a strong advocate for the end-users and for many of my proposed usability recommendations during later discussion with business stakeholders. In future usability tests I conduct, I would strongly encourage team leads and business stakeholders to observe these sessions.