Pilot Test Case Study

It’s usability-study day. Your first participant walks in. You brief him, explain the session, and dive into the study. Only to realize that the participant does not understand what you’re asking him to do. It’s not a problem with the site design — it’s a problem with the study itself. You’ve just begun your study and you’re already digging yourself out of a hole, trying to quickly determine how to salvage the session. Now what?

When running qualitative usability studies, we often recruit only a handful of users. Small numbers of users can lend great insights into design and usability strengths and weaknesses. So if users struggle not with your site design, but your study design, it can be difficult to get the necessary results. Session data may need to be thrown away, depending on the severity of the problems.

Pilot testing can go a long way to alleviating such problems. In a pilot test, the usability practitioner (and team, if possible) runs through a session or two in advance of the main, scheduled study. Typically only a small number of sessions are needed to prepare for the full study and make sure everything is in order. The point of the test is to test the study itself, running 1–2 sessions to help ensure that the full study goes as smoothly as possible

Pilot testing is particularly important if you are:

  • New to running a usability test. Better to have your first try with a session that you can throw away if needed.
  • Testing in an unfamiliar subject area. If this is your first test of a site aimed, for example, at rocket scientists or nuclear physicists, and you aren’t an expert in the area, pilot tests can help prepare you. (Also do work ahead of the study to familiarize yourself with the topics and terminology, for instance by meeting with subject-matter experts as well as designers and developers.)
  • Running a remote, unmoderated study. Any time that study instructions need to stand on their own, they need to be tested to try to limit possibilities for misinterpretation. If you are running a diary study, unmoderated online study, or study conducted via email, all communications, from recruiting information to confirmation emails to study instructions to follow-up questions, should be pilot tested. These instructions need to stand on their own, as no one will be there to answer questions or make clarifications if a participant runs into a problem.
  • Running a quantitative study. Larger scale studies are typically done so that statistically significant results can be calculated. In such studies, each session needs to be run the same way. As such, a solid script must be created and tested.
  • Testing a high-visibility project. Even if you’ve worked in usability for years, there may be the occasional high-visibility project that requires some extra care. Maybe the results are going straight to the CEO. Maybe the test is on the company’s premier product. Of course, every usability project is important, but some may be a bit more important than others.
  • Doing a one-shot research project. If you’re doing many rounds of iterative testing, then the damage is limited if you make a few mistakes in the first test. The second study will be better. But if a design project only gets a single dose of user research, you want to get it right the first time, because there won’t be a second time.

Even veteran usability practitioners can benefit from running pilot tests. The longer you work in the field, the better you can get at writing strong tasks and instructions in the first place, but it never hurts to run a test plan past a participant in advance. Does every usability study require a pilot test? No, but it’s extra insurance that the study will run smoothly, leaving the team able to focus on the results, rather than the study itself.

Benefit #1: Dress Rehearsal

Pilot testing is a dress rehearsal for the study. It’s a great dry run to make sure the facilitator and team are prepared for the study. Are the materials printed? Consent forms copied? Payment prepared? Site ready and functional? Checklists can help, but running through the study in a low-pressure setting first is a good way to double check that the team is prepared.

Benefit #2: Test the Tasks

If a user is derailed by a poorly written task, it takes precious time away from studying the interface. The facilitator first needs to realize that the participant didn’t understand the task as it was intended, and then has to spend time redirecting or reinstructing the user. Any on-the-fly changes run the risk of accidentally giving users clues about how to complete an activity, or making participants feel they did something wrong. Tasks that can seem perfectly clear to you and your team can be confusing or misleading to participants. Find and fix issues with tasks before your study to help ensure a smoother test and stronger results.

Benefit #3: Timing

It can be hard to know how much time to schedule for user testing. Will participants complete 3 or 7 tasks during your testing session? Running through the tasks yourself and allowing more time than it took you is a very rough way to estimate task time. Running through the full study with 1 or 2 pilot users can help you better estimate how long users might take, allowing you to prioritize tasks effectively for the real study.

Benefit #4: Data You Can Use (Maybe)

If all goes well, or pretty well, in a pilot session, you’ve gotten a jump on the rest of the study. Run the session as though it’s the real thing, and if there are no major hiccups, the data can be used. Even if one or two tasks go awry, you may still be able to salvage information from the sessions. Don’t be too quick to dismiss any learnings just because it was a pilot test. At the same time, don’t be overeager to use the session as part of your final learnings if the session does not go well or something goes wrong at the beginning that might affect the reliability of the rest of the session (such as leading the user or overexplaining an offering or interface.)

Tips for Pilot Studies

To get the most out of a pilot session, schedule it at least 1 day in advance of the scheduled study, and longer if you are testing instructions for an unmoderated study where you might need to pilot instructions several times before they are solid enough to use. This allows you and your team time to make any necessary changes. (Assume there will be changes, because there always are.)

Recruit participants who match your target profile for pilot studies. This means that any feedback — about the study or about the site being tested— will be more relevant. In a pinch, recruiting someone who doesn’t quite fit the profile is typically better than not running a pilot test at all, but the results from those sessions then would not be applicable in the final study.

Does running a pilot study take extra time? Of course. Materials need to be ready earlier, extra participants need to be recruited, and time needs to be scheduled to run the sessions. The payoff, however, is that the final study will run more smoothly, making it easier to get the results that matter to your team and, ultimately, your product.

Case Studies/Pilot Test Findings

PTI-sponsored pilot projects and case studies are designed to help implement PTI within your company.  These pilot projects are designed to help answer implementation questions and to help develop well-informed best practices. Case studies document early adopters' experiences and learnings implementing the PTI within their companies. Postings may include information about pilots and case studies conducted and/or provided by industry members; any mention of a specific solution or technology provider they used does not constitute a PTI endorsement.

Pilot Project Participants

The following companies are among those who are participating in PTI pilot projects as of Jan. 1, 2012.
To volunteer to participate in a PTI pilot project, contact PMA's Ed Treacy and see the sign-up sheet:

  • Alpine Fresh
  • Castellini
  • Charlie's Produce (Seattle)
  • Chiquita Brands
  • Coastline Produce
  • Driscoll's
  • Duda Farm Fresh Foods
  • Food Lion
  • Growers Express
  • Jem D International
  • Liberty Fruit
  • M&A Products
  • Nunes Company, Inc.
  • Paramount Citrus
  • Pioneer Growers Cooperative
  • Royal Food Service (Atlanta)
  • Tanimura & Antle
  • The Oppenheimer Group
  • Three Rivers Produce, Inc.
  • Wada Farms Marketing Group
  • Westside Produce
  • Wishnatski Farms


Back to top

Pilot Test Findings

Produce Traceability Pilot Report Using ASNs (Part 1)

Oppenheimer, Safeway and iTradeNetwork began a pilot in May of 2011 to review the process for implementation of the Produce Traceability Initiative (PTI). This is one of several pilots initiated from the PTI Implementation Working Group charged with gathering pilot test results, identifying costs, challenges, opportunities, and testing Best Practice guidance documents.

Produce Traceability Pilot Report: ASN Test Cases and Sample Files (Part 2)

This report includes Advance Ship Notice (ASN) test cases and sample ASN files. 

Tracing Grapes from Chilean Field to U.S. Grocery

See how a traceability pilot implemented the PTI labeling and communications standards to follow red and green seedless grapes from Chile to retail stores across the Southern United States.

Produce Traceability Pilot Sponsored by Underwriters Laboratories (Video)

This multi-party traceability pilot conducted by LoBue Citrus, Associated Grocers, C.H. Robinson, and FoodLogiQ studied tracking of citrus from grove to grocer. This video highlights PTI milestones, details of PTI compliant case and pallet labeling, and documents the importance of critical tracking events throughout the supply chain.

Back to top

Case Studies

The Oppenheimer Group

Discover how this Global produce distributor uses the Advance Ship Notice for efficiently sharing traceability data.

Mother Earth Organic Mushrooms (May 2013)

Discover how Mother Earth Organic Mushrooms uses GS1 standards for produce traceability and real-time inventory management to deliver a safe and tasty product. 

JemD Farms (July 2012)

Read how JemD Farms demonstrates its commitment to food safety, enhances its reputation for innovation, and improves business operations by implementing all Produce Traceability Initiative milestones.

Paramount Citrus (May 2011)

Implementing the PTI is enabling this citrus industry leader to better trace its products forward and backward, while also providing faster door-to-door efficiencies, lower costs and greater customer satisfaction.

Growers Express (May 2011)

With 50,000 acres spread over multiple states and countries and crops rotated more than twice annually, Growers Express found lot integrity was hard to maintain and traceability suffered. By assigning case labels with GS1 GTINs, the company now has real-time traceability for every one of the 18 million cartons it ships annually. As a result, traceability is facilitated field to shelf, stringent food safety measures are followed and documented, the company has increased productivity and realized substantial savings and efficiencies.

Frontera Produce (October 2010)

Learn how Frontera gained visibility into its supply chain with GS1 barcodes for enhanced food safety.

Back to top

Volunteer for a Pilot Project

PTI Pilot Project Sign-up Sheet (November 2010)

Use this document if interested in participating in a PTI pilot.

Back to top


Leave a Reply

Your email address will not be published. Required fields are marked *