Protect Texas Together Usability Study
This usability study was conducted to evaluate University of Texas at Austin’s Protect Texas Together Mobile App and recommend improvements based on usability testing and user feedback
Objectives and Goals
Key Objectives
- Location Privacy – Investigate perspectives on location privacy features and the importance of location privacy
- Features – Gather thoughts about using the app’s features on a daily basis and opportunities to encourage daily engagement
- Contact Tracing – Explore thoughts on contact tracing and gain deeper insights into implementation
- Data & Resources – Learn what value the resources section provides to users and what the users decide are the most useful pieces of information
Methodology
Our Process
Evaluate
Searching for pain
We searched for key points to lock into via a heuristic evaluation on the existing designs
Analyze
Seeing what others do
We performed a comparative analysis of both direct and indirect competitors within the space
Interview
Collecting people’s thoughts
After finding areas to look into, we interviewed a variety of users and tested mockups with them for usability
Synthesize
Creating answers
Based on what we found in our research, we have synthesized our findings and created suggestions
The Heuristic Evaluation
Evaluation scale
For our evaluation, we chose to use a scale from one to five to rate usability issues throughout the application. Through this scale, we were able to provide light to varying severity issues throughout our evaluation.
Heuristics used
We based our heuristic evaluation off of Nielsen Norman Group’s 10 usability heuristics for user interface design and added notes about accessibility to the evaluation.
Heuristic Definitions
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
The system should speak the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
Minimize the user’s memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
Accelerators — unseen by the novice user — may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.
What we found
We found 1 blocker, 15 high severity, 14 medium, and 11 cosmetic problems within the application through our evaluation.
The Competitive Analysis
We compared direct competitor and market adjacent applications to find what the UT HornSense app does well and what it needs to improve upon. We rated changes on a scale of minimal important changes and critical features
What we found
We identified five major features that need to be addressed, but none that block the deployment of the application completely.
The Austrian COVID application does a good job of breaking down what contact tracing is and giving it a good marketing term
The NHS Covid app shows a screen with a summary of how the app works and the user’s role in it and keeps the users informed of all the steps and processes in the app.
The current log book within the application is convoluted and hard to use. We should simplify it and create a way for people to quickly log their locations or tests.
Show users potential exposure risks based on previous locations where the user may have been exposed
The current log book within the application is convoluted and hard to use. We should simplify it and create a way for people to quickly log their locations or tests.
Other Findings
User Interviews
Interview Screening
For our usability testing, we needed to look at people that would have the highest probability of needing to use the Protect Texas Together application.
Who we screened for
| Undergrads | 5 |
| Graduate Students | 5 |
| Covid Aware | 70% |
| Location | Austin, TX |
Who we interviewed
| Undergrads | 5 |
| Graduate Students | 3 |
| Non-UT Grad Students | 5 |
| Total | 13 Participants |
Testing Format
1. Interview Medium
One-on-One recorded interviews held over Zoom as video calls
2. Time
We limited our interview time to roughly 35-45 minute blocks depending on the flow of each one
3. Questionnaires
We asked each participant to complete both a pre and post-test questionnaire
4. Scripting
We used a general script for consistency within the interviews but allowed freedom to deviate if a conversation led to good insights
Detailed Task Findings
We performed card sorting, affinity diagraming, and insight extraction techniques to organize data received from the interviews at a task level
Task 1: Login and First Experience
Task Description
Imagine you received an email from UT about a mobile app to help you stay healthy during COVID. You decide to download it and try it out for the first time.
Findings
Design
The UI and UT branding are well-received by participants Participants did not spend much time reading the “Campus Access Page” due to overly dense text
Initial Survey
The initial survey was easy for participants but the lack of information detailing symptom meanings led to confusion
Room Setup
Overall process was perceived to be overly complex Room names were complicated and difficult to remember The checkout process was not clearly defined as participants questioned whether or not the check-out process is automatic
Task 2: Entering a Room
Task Description
The University mentioned they would like everyone to use the app so they can tell how many people are in a room together in order to adhere to local policies on social distancing. You just arrived in BEN A121-A, how would you use the app to note where you are?
Findings
Overall Process
Some participants found it confusing for room check-in to be a 2-step process and assumed they were checked-in once setting up the room
Room Names
Difficult to add rooms within the app due to complicated naming convention
Usefulness
- Process felt to manual for an everyday process
- Participants noted that voice integration, homescreen widgets, notifications, or school mandates would push them to use the app everyday
Task 3: View Check-In History
Task Description
You forgot what room you were in last Thursday afternoon, how would you find that?
Findings
Check-out Process
No feedback from the application to notify when a user has been checked out of a room
Navigation
- Users found it difficult to understand what screen they were on due to tab navigation
- Navigation from the home screen was not intuitive due placement of the room check-in button and confusion with the “tracking” button
Requested features
- Synchronization with class schedule
- More informative date layout such as “Yesterday”, “This week”, or day of week in date format
Task 4: View Symptom History & Information
Task Description
You feel concerned you may have been showing signs of illness recently. You want to remember how you felt on August 3rd, 5th, and 21st. You also want to know more about what qualifies as tiredness, and what the risk levels you were assigned mean. How would you do this?
Findings
Calendar Function
Design was well received but the functionality wasn’t clear to participants and not well connected to the list view
“Tracking” button labels
Most participants associated “tracking” with contact tracing
Symptom Tracking
Participants had mixed feelings about the usefulness of retroactive symptom tracking
Symptom and Risk Level Information
Contradictory process for viewing symptom and risk level information caused confusion for participants
Task 5: Resources’ relevance and usability
Task Description
You want to keep yourself informed of the latest COVID news and the resources for wellbeing that are available to you. How can you do that from the dashboard?
Findings
- News and resources, particular those that are UT specific, are helpful to users
- Graphs generally felt useful, but interaction with them was difficult for users
- Participants expressed distrust around the news sources, wished they were more transparent with the source of the news
- Users wished to see COVID hotspots and higher risk areas around UT and vicinity in Austin
Task 6: QR Code to upload COVID test results
Task Description
Assume that you have just taken a Covid test and you want to report the results on the app. Navigate to the screen where you would scan your test results for the app. Once there, click on the test shown inside the QR scanner. Read and understand the test results before entering the test.
Findings
- Having a QR code scan button on the home screen would be helpful
- Participants found test history post-scan confusing and questioned its relevance
- Participants wanted to see the results of the tests logged in the app
- Functionality around logging off campus tests seemed confusing for some participants
Task 7: Contact Tracing
Task Description
Assume that you are feeling good today, are experiencing no symptoms, and have not yet filled out a daily survey. On July 26th you ate at Taco Bell with a friend before online class. You just found out that the friend you ate with tested positive for COVID-19. Since this dinner, you’ve been in close contact with your roommate.
How would you log this in the app?
Findings
- Functionality around tracking needs to be simpler. It demands too much cognitive load from users
- Terminology around contact tracking (eg. close contact) needs to be defined clearly
- Main headings need to be reworded to be more descriptive, and subtext needs to be large in size
- Users found the survey to be useful and engaging
- Users question the importance of logging actions they were doing at locations.
Recommendations
Privacy and Location
Problems
- Users found location information too invasive and did not want UT to have this data about them
- Users found it confusing and difficult to input their location in the app
Proposed Solutions
- Use the onboarding process to clearly and concisely explain why UT needs location information and how it will benefit the user
- Simplify the location input screen to either automatically collect the users information or make the location input one-click
User Input Survey
Problems
- Users found this process tedious and would not complete the action without a mandate from the school
- Users found that many questions asked in the survey should be answered by the school rather than the user
Proposed Solutions
- Consider an offline solution done by the school where a student could not get on campus without filling out the survey
- Consider adding an incentive program where students could get discounts on school products by filling out the survey
- Users found the “Resources” page useful. Consider keeping that page locked until students fill out the survey
- Consider advertising and media pages to teach users about the usefulness of surveys before the user starts using the app
- Sync the information available to UT through each UT eID so that the user fills out as little information as possible
Usefulness of Resources
Problems
- Users found the Resources page useful but questioned the validity of the information
- Users want more information regarding Covid statistics
Proposed Solutions
- Clearly explain to the user the sources of the information in Resources
- Give users the ability to drill down on specific case rises based on location or the college within UTUsers found the “Resources” page useful. Consider keeping that page locked until students fill out the survey
- Collaborate with other organizations making similar applications so that you can share the COVID statistics and give users information on other areasSync the information available to UT through each UT eID so that the user fills out as little information as possible
- Allow users to compare their town or school to others doing the same thing