Pluralistic Usability Walkthrough

A usability test method employed to generate early design evaluation by assigning a group of users a series of paper-based tasks that represent the proposed product interface and including participation from developers of that interface.

A systematic group evaluation of a design in which usability practitioners serving as walkthrough administrators guide users through tasks simulated on hard-copy panels and facilitate feedback about those tasks while developers and other members of the product team address concerns or questions about the interface.

Related Links

Authoritative References

Bias, R.G. (1994) Pluralistic usability walkthrough: coordinated empathies. In J. Nielsen & R.L. Mack (Eds.), Usability Inspection Methods (pp. 63-76). New York, NY: Wiley and Sons, Inc.

Bias, R.G. (1991) Walkthroughs: efficient collaborative testing. IEEE Software, 8(5), 94-95.

Related Subjects

• Traditional Usability Walkthrough: The pluralistic usability walkthrough is a modification of the traditional usability walkthrough in that it incorporates representative users, product developers, members of the product team, and usability experts in the proceedings.

• On-line Prototyping: The pluralistic usability walkthrough can be used in parallel with on-line prototyping, as the prototype can be projected at the front of the room, along with the hard-copy screen-shot packets, to illustrate aspects of the user interface difficult or expensive to capture on paper, like color and cursor movement.

• Cognitive Walkthrough: Cognitive walkthroughs are also performed early on in the design cycle to determine how well an interface supports the tasks users might carry out, but developers and usability professionals —not users — perform that evaluation. Also, the cognitive walkthrough was intended as a method to test learnability.

• Participatory Heuristic Evaluation: Similarly involves several representative users, usability practitioners, and product team members who review a product against a set of general principles.

Detailed Description

Outcomes and Deliverables

• A list of potential usability problems.

• Suggested improvements to UI designs

Benefits, Advantages and Disadvantages

Benefits

Reduces test-redesign-retest cycle by generating immediate feedback and discussion of design problems and possible solutions while users are present. Can provide early performance and satisfaction data before costly design strategies have been implemented. The group atmosphere encourages collaborative, constructive comments from users, developers, and other members of the product team.

Advantages

Generates valuable quantitative and qualitative data on users’ actions by way of written responses. Product developers present during the session gain appreciation for common user frustrations or concerns about design.

Disadvantages

A fixed sequence of hard-copy panels limits the simulations that users can perform (no browsing or exploring). Alternative paths for the same task are not explored. Product developers might not feel comfortable hearing criticism about their designs. Because the walkthrough is dependent on all users finishing each task before discussion can begin, the session can feel laborious.

Cost-Effectiveness (ROI)

• Cuts down test-redesign-retest time by allowing developers to hear feedback before the interface has moved beyond the paper stage.

• Can be performed early in the product development cycle.

• Can gather data from multiple users in one session.

How To

Appropriate Uses

• Usually conducted early in the development cycle or when production time is limited.

• Appropriate for a group of 6-10 representative users.

Procedure

• The walkthrough administrator must prepare the product designers and developers in advance of the walkthrough. They need to be instructed to be thick-skinned, and to treat all user comments with positive regard. It helps to tell the product developers that we will NOT be making a recommendation for a product change in response to every user comment; we will be filtering all their comments through our design sense (that of the development team and the usability professional).

• Participants are presented with instructions and rules, in addition to task and scenario descriptions.

• Walkthrough administrator asks participants to write on the hard copy of the first panel the actions they would take in attempting the specified task.

• After all participants have written their responses, the walkthrough administrator (or a developer) announces the “right” answer.

• The users verbalize their responses and discuss potential usability problems, while the product developers remain quiet and the usability professionals facilitate the discussion among the users.

• As the discussion winds down, the developers are invited to join in, often with an explanation of why the design was the way it was.

• After each task, the participants are given a brief questionnaire regarding the usability of the interface.

Participants and Other Stakeholders

• Usability practitioner serves as walkthrough administrator, introducing tasks and encouraging group discussion about the design.

• Product developers (designers, engineers, etc.) answer questions about design and suggest solutions to interface problems users have encountered.

• Other members of the product team who are involved in making decisions that affect the user interface.

• Users representative of the target audience are the primary participants. Theirs are the data we are most interested in.

Materials Needed

• Printed screen-shots put together in packets in the same order that the screens would be confronted when the users were carrying out a particular task.

• Writing utensils for marking up the screen-shots and filling out questionnaires after each task.

• Room large enough to accommodate 6-10 users and a similar or smaller number of developers.

Who Can Facilitate

The walkthrough administrator (generally an experienced usability practitioner) must be able to guide users through tasks and facilitate collaboration between users and developers. It is usually best to avoid having a product developer/designer do it, as they tend to get defensive or move into “teach” mode rather than collecting user data.

Common Problems

Suitable for 6-10 representative users and products with linear tasks.

All users must complete each task before discussion and next task can begin, potentially affecting participants’ understanding of the design’s flow.

Data Analysis Approach

The users’ responses on their test packets, plus the answers to post-scenario surveys, can provide useful quantitative data to review how many completed tasks correctly and what percentage of users responded positively to the design. Also valuable are the qualitative data in the form of redesigns recommended by the users and the questions that both users and developers ask.

Next Steps

Walkthrough results provide developers and other product team members with constructive feedback about how to improve or modify design.

Special Considerations

Costs and Scalability

People and Equipment

• Photocopying expenses for screen-shot packets and questionnaires.

• 1 usability practitioner serves as walkthrough administrator.

• Another usability practitioner might be required to take notes.

• 6-10 representative users. The costs for these users would include screening costs, travel expenses, and any incentives that are given for participation.

• Product designers and developers must dedicate time for helping with the preparation of the materials, for attending the session, and for working on potential (implementable) solutions that address the design issues that emerged.

Time

Approximately three hours are required to carry out the walkthrough. It usually takes two or three days to develop the test tasks and build the test materials.

Accessibility Considerations

Allowing multiple visually handicapped users who use screen readers in the same room likely would not work.

International Considerations

Video conferencing capabilities can be used to allow communication between users and developers who are located at different sites or different countries.

Other Considerations

Walkthrough administrator should avoid characterizing users and developers as being on different “sides.”

Ethical and Legal Considerations

Pluralistic usability walkthrough involves the same legal considerations as for any user-based testing. These considerations include informed consent and non-disclosure agreements. To these, we add the concern that developers might not have had the same training in ethical treatment of test participants as usability practitioners have. Advise developers in advance to give positive regard to every user comment and approach users with courtesy and respect.

Political Issues

The usability practitioner must prepare the product developers and designers to be thick-skinned and to treat the users with respect.