Usability Testing Report for Friends of Richmond Community High School completed in April 2022 by Michelle Saiyed
Executive Summary
This report details a usability study conducted by Michelle Saiyed on the Friends of Richmond Community High School website. This report highlights the noteworthy findings that impact the usability of the website. Recommendations to improve the usability of the website are provided.
The three primary recommendations are to
1) Present information about the organization's mission statement and relationship with RCHS in simple terms that name specific actions that the organization does and provide evidence of the impact of those actions
2) Update Registration Form and Information for Volunteer Opportunities for Each Program
3) Change all buttons to open new pages inside the same tab instead of a new tab or window, and mark them with consistent labels.
Introduction
This report is intended as a guide to aid in the growth and development of Friends of Richmond Community High School. Friends of Richmond Community High School is a nonprofit organization that supports the continued education of Richmond Community High School students. RCHS is a college preparatory alternative high school in Richmond, Virginia. Friends of RCHS support the school through programs, raising funds, and community outreach. Friends of RCHS launched a new website to reach new community members, increase donations, and engage more student volunteers. I focused on the following features to evaluate the site's usability:
Organization of Information
Donation Process
Volunteer Registration Process
I created a usability test for the website. Quantitative and qualitative data from the test pinpointed high and low points in the website's usability. The report includes findings highlighting usability issues with the website and recommendations to improve usability.
Methodology
Five user participants completed the usability test. All five participants signed a consent form (Appendix 2). I observed the participants in a controlled setting as they performed the three tasks on the website. The test was moderated; I instructed participants to vocalize their thought processes as they navigated through each task. The moderator script is available in Appendix 1. Individual screen recordings, audio recordings, and notes for each participant are available here. Each task explores a different area of the website.
Task 1:
You are a parent whose child was recently accepted into Richmond Community High School. You are looking for additional sources to support your child. Find out more about what Friends of Richmond Community High School does.
Task 2:
You are a donor looking to contribute to a local organization that impacts student life and education. Please set up a recurring donation to a specific program within the organization.
Task 3:
You are a student at RCHS and need community service hours. Volunteer at a program with Friends of RCHS.
In addition to the task-driven portion of the user test, there were additional methods administered to learn more about the users and their experiences while completing the tasks. I administered the following methods:
Pre-test Questionnaires:
To understand users' experience with similar organizations
Post-Task Interviews:
To gather user reflections following the task
System Usability Scale:
Provides quantitative data on user reactions and impressions of the website (Appendix 4)
By evaluating these methods and task results, I identified current pain points and made recommendations to improve usability.
User Test Participants
During the initial meeting with Friends of RCHS, they identified their target audience demographic. I screened participants to narrow down the participant pool to reflect only those who:
Live in Richmond, Virginia
Between the ages of 18-65*
*Participants over the age of 18 were chosen to avoid any issues concerning receiving consent.
Participant Data Summary
All participants were adults over the age of 18. Of the total participants, 60% were female, and 40% were male. All participants live in Richmond, Virginia.
Findings and Recommendations
Study findings are detailed below, along with recommendations to improve the Friends of RCHS website's usability. Participant IDs are used alongside any quotes or reflections to preserve anonymity (ex: "Participant 3 "= "P3").Specific measurements and details regarding task difficulty, time to complete the task, and user success per task are in Appendix 3.
Finding 1: Users had difficulty fully and confidently understanding the purpose of Friends of RCHS.
Most users broadly understood the purpose of Friends of RCHS as an organization that provides programs to the community. Many stated that they could not find specific information about what the organization does. Many users did not understand the relationship between RCHS and Friends of RCHS.
Recommendation 1: Present information about the organization's mission statement and relationship with RCHS in simple terms that name specific actions that the organization does and provide evidence of the impact of those actions
Based on user testing results, I recommend clarifying the relationship between Friends of RCHS and RCHS in simple statements: "Friends of RCHS support the students of RCHS by providing them with X, Y, Z….". I would also include specific information about the organization's actions and the direct impact of their work in real-life statistics: "Friends of RCHS supported X number of students to college through the Dorm Life program since XXXX." This will help users better understand the organization's purpose through their achievements and actions;
Finding 2: Users had difficulty registering to volunteer with the organization and could not locate details about volunteering opportunities within specific programs.
Users had difficulty completing Task 3, which asked them to register as volunteers for one of the programs offered at Friends of RCHS. On the Volunteering page, the buttons under each program description provided have different labels, which confused some users as they were unsure if all opportunities were volunteer positions, despite finding them on the volunteer page. Additionally, only the button corresponding with the Nonprofit Office Support program leads the user toward a registration form or information page. The other two buttons lead users to a page with an image and the following text: "The requested form is no longer available." This confused P2, as they stated they were unsure if this page was a mistake on behalf of the website or an error with the testing computer's internet connection. This page also frustrated some users, such as P5, who stated: "If I cannot do what I want to do, I don't want to do this" after they could not register to volunteer with the Mindfulness Center or Community Gardens.
Recommendation 2: Update Registration Form and Contact Information for Volunteer Opportunities for Each Program
Based on user testing results, I recommend updating each program button from the Volunteering page to link users to a page with corresponding information about the program. This page should include program details, activity details, volunteering opportunities, necessary information to get involved, a registration form, and volunteer opportunity availability. The registration form should allow users to volunteer for specific activities on specific dates within their program of interest. Additionally, the program's brief description on the Volunteering page should include if the program is currently accepting volunteers.
Finding 3: Users had some difficulty and confusion navigating the website due to the inconsistent option of the back button feature and button labels.
Users consistently reported a feeling of being stuck or reaching a stopping point when they reached a page that did not allow them to use the "back" button feature as the button opened the page in a separate tab. I noted this frustration when users would click on a button on the Volunteering page that opened a new tab with an unavailable page. Navigating back to the original page from the unavailable form frustrated some users and interrupted every user's flow. Additionally, the buttons are labeled differently under each program on the Volunteering page, which confused some users about whether they could register for the program as a volunteer (Task 3).
Recommendation 3: Change all buttons 1) to open new links inside the same tab and not open a new tab or window and 2) to have consistent labels
I recommend changing all buttons to open new links within the same page instead of opening the link as a new tab or window to limit interruptions to the user flow. This change will allow users to use the back button feature on their browser to reach the previous page without confusion or disruption. Additionally, changing button labels to maintain consistency will help limit user confusion. Specifically, I recommend labeling the buttons under each program with the same label on the Volunteering page (i.e., "Register to volunteer").
Conclusion
I arrived at the above recommendations by performing five in-person usability tests and evaluating the quantitative and qualitative data collected by the tests. By implementing recommendations in this report, Friends of RHS will see a significant improvement in their users' experience with the website, user retention, donation conversion rate, and volunteer registration.
Limitations and Future Studies
The following are limitations of the usability test and reflections on improving future usability studies.
Some of the wording on the moderator script could have been clearer to help participants achieve tasks and gain more insightful information. For example, the warm-up questions included: "How do you feel about high education, such as university or college education?" and "How do you feel about community programs?". Changing the question to one about the experience instead of feeling would provide better feedback on the background of the participants. For future studies, the question should be changed from "How do you feel about higher education…?" to "What is your experience with higher education…?". Additionally, some participants were confused by the term "community programs" and were unsure of the definition. I recommend changing the phrase "community programs" to "nonprofit organizations focused on community outreach" to prevent further confusion.
Additionally, Task 1 ("Find out more about what Friends of RCHS does") could have been more specific to yield more insightful results. The task format allowed for a lot of broad, nonspecific answers. The task should be more precise, such as "Identify the relationship between Friends of RCHS and RCHS."
Appendix 1: Moderator Script
Script
Introduction
Hi <Participant name>,
How are you doing today?
Thank you for taking the time to talk to us today!
My name is Michelle and I will be leading the session today.
We’ll start with some questions, and then I will give you some tasks to complete on the computer. We’ll use the feedback we get from you and other participants to improve the website of the nonprofit Friends of Richmond Community High School, so please share your honest thoughts as we go along. There are no right or wrong answers!
Do you have any questions for me before we get started?
Before we begin, could I just confirm that you’re still okay with this session being recorded and shared alongside my User Research Portfolio? [Wait for reply]
Awesome! I will start the recording now.
[PRESS RECORD]
Warm-up questions
How do you feel about higher education, such as university or college education?
How do you feel about community programs?
Tasks
Tasks briefing
“Now I’m going to ask you to carry out some tasks on the computer. There are a few things to keep in mind here:
We're not testing you - we are testing the site! If you encounter difficulties, it's never your fault, and it's always the site's fault. Finding these problems is good because it will show us where we need to make improvements.
Be as candid as possible. If you don't like something or think it's just plain stupid, please say so!
Behave as naturally as possible. (I know this can be tricky with me watching what you’re doing, but still). If you get bored of something and you don’t think you’d continue any further please let me know.
Please ask me any questions you wish, but for this test, I might not be able to answer them for you.
Most importantly, please think aloud as you do this. Share with me where you're going to click, why you're clicking there, and what you expect to see after you do so. It helps us understand what is going through your mind and allows us to take good notes.
Task 1: Find out more about Friends of RCHS
Prototype link: https://friendsofrchs.org/
Relevant research objective: Make it easier for users to find information about the organization: How do we provide information to the community in a way that is relevant, professional, and easy to find?
Scenario: You are a parent whose child has been recently accepted into Richmond Community High School. You are looking for additional sources to support your child.
Task: Please identify the purpose of Friends of RCHS.
Single Ease Questions: Overall, this task was…”
- Extremely difficult
- Difficult
- Neither easy nor difficult
- Easy
- Extremely easy
Single Ease Question follow-up: Why did you give this score?
Task 2: Make a Donation to Friends of RCHS
Prototype link: https://friendsofrchs.org
Relevant research objective: Make it simple for people to donate and become sponsors: How do we increase donations and the rate of reoccurring donations?
Scenario: You are a donor looking to contribute to local organizations that impact student life and education.
Task: Please set up a recurring donation to a specific program within the organization.
Single Ease Questions: Overall, this task was…”
- Extremely difficult
- Difficult
- Neither easy nor difficult
- Easy
- Extremely easy
Single Ease Question follow-up: Why did you give this score?
Task 3: Volunteer with Friends of RCHS
Prototype link: https://friendsofrchs.org
Relevant research objective: Improve student engagement with the organization's activities and programs: How do we get students to register for, confirm, and attend organization programs and activities?
Scenario: You are a student at RCHS and need community service hours.
Task: Please sign up to volunteer in one of the programs offered at Friends of RCHS.
Single Ease Questions: Overall, this task was…”
- Extremely difficult
- Difficult
- Neither easy nor difficult
- Easy
- Extremely easy
Single Ease Question follow-up: Why did you give this score?
Wrap up questions
Adjective rating scale (SUS): Overall, I would rate the user-friendliness of this product as:
- Worst-imaginable
- Awful
- Poor
- OK
- Good
- Excellent
- Best imaginable
Adjective rating scale follow-up: Why did you give this score?
Anything else you’d like to add?
Closing words
These are all my questions for today.
Thank you very much for taking the time to speak with me.
As a thank you, I would like to give you a $5 Starbucks gift card. Thanks again!
Appendix 2: Consent Form
Consent Form (Adult)
I agree to participate in the study conducted by Michelle Saiyed. The purpose of this usability testing is to test the usability of the Friends of Richmond Community High School website. All demographic information will be kept anonymous.
I understand that participation in this usability study is voluntary and I agree to immediately raise any concerns or areas of discomfort during the session with the study administrator.
Please sign below to indicate that you have read and you understand the information on this form and that any questions you might have about the session have been answered.
Date:_________
Please print your name: ____________________________________________________
Please sign your name: ____________________________________________________
Thank you!
We appreciate your participation.
Appendix 3- Task Result Summary Graphs (graphs available on pdf version)
Figure 1. User Success
Task 3 had the most variability in user success, with scores ranging from extremely difficult to easy. Task 2 had the highest user success in completing the task, with 100% of users finding it easy or extremely easy to complete (graph available on pdf).
Figure 2. Average Task Duration
On average, Task 1 took the longest amount of time to complete, and Task 3 took the shortest amount of time to complete (graph available on pdf).
Figure 3. Average Task Difficulty Rating
On average, users experienced the most issues completing Task 3 and Task 1. Overall, users had the easiest time completing Task 2 (graph available on pdf).
Appendix 4- System Usability Scale
Adjective rating scale (SUS)
1- Worst-imaginable
2- Awful
3- Poor
4- OK
5- Good
6- Excellent
7- Best imaginable
Participant
SUS Score
1
4
2
5
3
6
4
5
5
4
Average
4.8
On average, users rated the user-friendliness of the website as good.