User testing – the foundation recipe
Improvisation from basics
Every good cook has some treasured foundation recipes: a simple muffin mix to which she can add nuts, chocolate or spices; perhaps a tomato and onion based soup to which she can throw in seasonal vegetables, pasta or chopped ham; maybe a spicey curry base that works well with prawns, chicken or vegetables.
To improvise in the kitchen, firstly master the basics then understand when each variation is appropriate. For a white sauce, add parsley to accompany fish. Add mustard for boiled bacon or cheese for savory pancakes. No onions? Chop a scallion. Left over Tarragon? Chop it up; chuck it in. Last night’s Salsa? Think again!
Experienced usability practitioners follow a similar approach in designing a usability test. It’s applied science; observation and analysis are fundamental. However, depending on goals and constraints, we can look for many things, observe in different ways and choose from a wide range of analytical techniques. As with cooking, there’s a foundation recipe and a wide range of variations.
User testing – the foundation recipe
Here’s a seven-step recipe that covers most types of testing. The two activities in parenthes are not strictly part of the method; they do, however, reduce risk and ensure that you learn from your experience.
1. Design study |
⇒ | 2. Recruit participants |
⇒ | 3. Prepare artefacts |
⇒ | (Pilot) | ⇒ | 4. Observe Measure Ask |
⇒ | 5. Analyse data |
⇒ | 6. Report results |
⇒ | 7. Brief client |
⇒ | (Project debrief) |
You can expect to have some activity for each step. However, the nature and scope of that activity will vary according to the needs of the client and the culture of the project. Consider a test to assess the safety of a remotely-controlled radiography device. You might plan for hypothesis-testing (design study) using a large sample size (recruit participants) to record error rates (measure) for statistical analysis (analyse data).
The report (report results) might become a formal project deliverable while a handover meeting (brief client) would be essential for a mixed audience of technical, business and medical specialists. It’s the equivalent of high-tea, muffins with chopped dates, walnuts and cinnamon.
For a small-scale “in-flight” study, the model is the same but the activities are smaller and simpler. A formative research design (design study) uses a small sample (recruit participants) to acquire data (observe, ask) for qualitative analysis (analyse data). The results are presented in a PowerPoint deck (report results) and reviewed by the design team and project manager (brief client). This situation is more like a simple dusting of caster sugar – good rather than fancy.
Variations
Here are the nuts, raisins and chocolate chips to add to the basic recipe.
1. Design study | Summative, Formative, Benchmark, Competitive, Comparative
User-driven, “chauffeured” |
2. Recruit participants | Quota sample, Stratified sample, Opportunity sample
Recruit directly, Use an agency Volunteers, Incentives |
3. Prepare artefacts | Paper, Static, PowerPoint, Axure etc, Wizard of Oz, Live code |
4. Observe | Direct, Indirect (video), Remote (e.g.TechSmith)
From a control room, Side-by-side In a lab, In an office, In the field |
4. Measure | Count, Time, Code, Checklist |
4. Question | Active, Passive Interrupt protocol, Debrief protocol, Before-and-after protocol |
5. Analyse data | Quantitative, Qualitative
Specific observations, Generalised issues Descriptive, Analytical Business impact oriented, Solution feature oriented |
6. Report results | Document, PowerPoint, Annotated video, Verbal
Formal, Informal, Standardised |
7. Brief client | Briefing, Review, Action-planning |
You can read more about these techniques in books such as Practical Guide to Usability Testing (Dumas) or Human Computer Interaction (Preece et al).
Checklist
The success of a user-test is pretty much determined by the quality of the thinking you do before you book a lab or approach a recruiter. Here’s a checklist that covers the main issues. Use it as the basis of a workshop or planning session before you start on design and logistics.
1. Design study |
|
2. Recruit participants |
|
3. Prepare artefacts |
|
4. Observe |
|
4. Measure |
|
4. Ask |
|
5. Analyse data |
|
6. Report results |
|
7. Brief client |
|
As in the kitchen, get the basics right but be prepared to improvise the detail. That way you’re still ready when you don’t have the right method in the store cupboard.
Nice list very concise
Comment by Matt — April 22, 2010 @ 9:21 pm