PAPA JOHNS USABILITY TEST
This project serves the purpose of demonstrating my abilities and knowledge on usability testing.
This is a usability test report on the ability of Papa John’s Pizza website to deliver the best user experience to its clients. It was designed to explore the usefulness of its navigation and overall performance.
Papa John's Pizza is an American restaurant company with a headquarters in Jeffersontown, Kentucky. It is the third largest delivery restaurant of take-out and pizza food in the entire world. Besides the traditional take-out options, Papa Johns’ Pizza offers to clients the opportunity to order pizza online, through the company’s website.
To assure Papa John’s website is usable enough to thrive in the market, it is essential to evaluate it once more to detect any potential flows. This is why Papa John’s decided to perform this usability test, to determine the extent that the company’s website interface facilitates the user’s ability to order pizza and complete specified routine tasks that are important to the company in a satisfactory way.
This document contains information about the participant’s feedback when performed the assigned tasks and further comments about the overall user experience and recommendations for improvement. It includes also a copy of the scenarios and questions provided.
Papa Johns is interested in optimizing the product by finding flaws that might interfere with the proper functioning of the website. The test performed provides insight about the satisfaction that the user experiences while interacting with the product. Papa John’s usability professionals conducted the test.
Three participants were selected using a screener test to isolate the segment of the population with the demographic characteristics of interest in the study.
The participants were asked to perform three specific tasks navigating the website using a computer.
As participants performed the tasks, the professional testers observed them, took notes, and interacted with them only when it was appropriate.
The testers followed precise protocols about the degree of verbal intervention they would use, ethical behavior, and other strategies known as the Golden Rules of Moderating.
A screen capture software, Screencast-o-Matic, was used to record the participant’s navigational choices and record his or her gestures, voice, comments, questions and overall reactions.
The tests were performed in the last week of September, 2016, in the company’s testing room.
All participants ordered a pizza online before, but not specifically from Papa John’s Pizza.
Two out of three of the participants had some level of confusion, at some point, when ordering a pizza.
All participants expressed their preference for making the complaints through human customer service rather than filling forms or writing emails.
Two of the participants, number 2 and 3, had trouble signing up for coupons without registering.
The half pizza icon was not intuitive for any of the participants.
Three participants were pleasantly surprised about the level of customization.
Two of the participants, number 1 and 2, were positively impressed by the “adding a topping” animation.
Two of the participants, number 1 and 2, were overall happy with the website experience.
The test administrator contacted the potential participants. A screening instrument was designed to isolate the segment of the population that we needed to test for usability. Three participants were selected for the usability test.
Once the participants were selected, the test administrator contacted the attendees by emailing them, to inform them of the test planning and confirming their availability and participation. At that point, dates and times for the test were arranged.
Before the session started, the test administrator explained the session organization and planning to the participant. At this stage, the test manager used his or her skills to assure the participant felt comfortable, appreciated, and easy.
The test administrator explained to the participant that he was not the object of analysis, but the product, assuring the user felt free to make mistakes.
The test administrator invited the participant to be as expressive and talkative as possible about every step he or she took. He was asked to express every thought that came to his or her mind while using the product, and every feeling he or she associated with the product and the experience.
At this point, the test administrator will ask for the participant consent to be observed and recorded.
Before asking the participant to enter the tasks, the test administrator asked him or her some questions that have the intention to get familiar with the participants and explore the user overall purchasing preferences related to our product.
Then, the participant was invited to use Google to browse for any pizza places close to his or her area code, to break the ice before starting to work with the intended Papa John’s Pizza website and submerge the user into the tasks.
The test lasted between 40 and 60 minutes, depending on the need and willingness of each participant motivation, style, ability and willingness to participate and express his or her feelings and ideas about the website performance and user experience.
The test consisted of three tasks designed to make the user navigate certain features of the product with the purpose of observing how fluently the participant could follow each path.
Task 1 - Ordering Pizza.
Task 2 - Signing up for deals and coupons without registering.
Task 3 – Contacting the corporate office.
For the participants to engage with the tasks, the test provided an imaginary scenario for each task.
Task 1 scenario: Imagine that the people you are with are fans of Papa John’s Pizza and that is where you will be ordering the pizza. There are a bunch of people at this party, and you need to order 3 pizzas.
Mushroom and Pepperoni.
Half onion and half sausage with the light sauce.
A specialty pizza (you want some variety).
Task 2 scenario: You want to sign up for deals and coupons, but you don’t want to register. You just want to give them your email. Can you do this and what do you think you will receive by email?
Task 3 scenario: Your pizza arrives and it is terrible! Your driver was rude and you are really upset about what just happened. You call the local store, but you get nowhere. You need to contact the corporate office; how would you do that?
At the end and in between each task was performed, the test administrator invited each participant to express his or her feelings about the navigation experience. When the test was finalized and the participant had nothing else to add, he or she was greeted and thanked for her collaboration.
All the participants, 1, 2, and 3, completed all the tasks required, with differences between them at the level of performance, discomfort, and easiness. Only one user’s overall satisfaction was negative. The other participants, in distinct ways, expressed appreciation navigating the product.
The following is a list of the navigation steps each participant took while performing each task.
In general, participants were very pleased with some of the web site features. These are some of the highlights.
Positive highlights about the ordering features
Participant 3 had a negative perception of the company. He had a not very pleasant experience through the whole test. Specifically, on task 3, when he was asked to imagine a frustrating delivery experience, he said: “That’s what I would expect of Papa John’s”
User comments about ordering pizza online
Remarks about The participants
Participant pizza most common ordering habits
User feedback about papa john’s pizza web site features
finding papa John’s pizza coupons within the site
contacting papa John’s corporate office
The following are recommendations based on the observation and analysis of the test results:
Considering that all the participants expressed, to one degree or another, some concern about the size and design of the half pizza graphic. I would recommend redesigning the graphic to make it clearer, perhaps bigger and more intuitive, and later test it for usability again.
Participant three tested using a Papa John’s site where a user had already subscribed. Then, the Start Order was not available for the participant, but instead the Delivery To user’s address button. This user was the one out of the three participants to struggle to place the very first order. I recommend redesigning this Delivery To button, and to include the words Start Order on it. This might prevent users that are using someone else’s site to place an order to get frustrated and change the provider because, as our participant, they don’t find a place where to order within the site fast enough. After changes are in place, test for usability again.
Participants hesitated about how to add the second pizza to the order. I recommend offering an Add another pizza button, or similar after the first pizza order was placed. This will speed the process. After clicking it, this button should give users the option of customizing it or picking one from any of the available already designed pizza offerings. After iteration, test for usability again.
Since all the participants at one point questioned themselves if creating an account would be a requirement to place an order. I would recommend testing the site offering the SIGN IN or Create an account button/option at the end of the purchase instead of at the very beginning. This might completely avoid any chance of users abandoning the site believing that’s the only option, as our participant 3 experienced it at one point. Placing an order as a guess would also work.
Since there was no consistency in the way participants found coupons, the navigational feature should be revised. Perhaps a Find Coupon button will be more effective. It could lead to a Rewards enrollment simultaneously.
When analyzing the results of this test, some considerations about the participant number three need to be highlighted. First, since he started the test, the participant number three was not comfortable with the testing equipment. This could have interfered with the participant’s performance. Second, he apparently had a deep rooted negative perception of the company.
The following is a suggestion for further research based on the observations and analysis of the test results:
Given the fact that two of the participants avoided cheese, and one of them identified herself as a vegan. It would be pertinent to research the potential Papa John’s vegan – as other special diets - target market, and the possibility of addressing them in a more direct and personalized manner.
Given the clear negative perception the third participant had about Papa John’s Pizza, it would be wise to research any possible deep rooted users’ negative perception about the company.
Two of our participants navigated the site with no major trouble. The third participant had several issues navigating the site. In general, all participants, to one degree or another, were positively impressed and satisfied with the customization features.