Critical Incident Technique (CIT)

Critical incident technique is a method of gathering facts (incidents) from domain experts or less experienced users of the existing system to gain knowledge of how to improve the performance of the individuals involved.

The critical incident technique (CIT) is used to look for the cause of human-system (or product) problems to minimize loss to person, property, money or data. The investigator looks for information on the performance of activities (e.g. tasks in the workplace) and the user-system interface. Both operators and records (e.g. documented events or recorded telephone calls) can provide such information. The investigator may focus on a particular incident or set of incidents which caused serious loss. Critical events are recorded and stored in a database or on a spreadsheet. Analysis may show how clusters of difficulties are related to a certain aspect of the system or human practice. Investigators then develop possible explanations for the source of the difficulty.

The method generates a list of good and bad behaviors which can then be used for performance appraisal.

Related Links

Originators/ Popularizers

Flanagan, J.C. 1954. The critical incident technique. Psychological bulletin, 51(4), 327-358. Flanagan was first to describe the critical incident technique.

Chapanis, A. (1959) Research techniques in human engineering, John Hopkins Press: Baltimore, Maryland.

Ramsey (1977) Accident sequence model. Includes: perception of hazard, cognition of hazard, decision whether or not to avoid hazard, ability to avoid hazard and luck.

Authoritative References

Christensen, J. (1985). Human factors in hazard risk evaluation, Human Factors Engineering: Engineering Summer Conference, University of Michigan: Ann Arbour, Michigan.

Reason, J. (1990). Human error. Cambridge University Press: New York.

Published Studies

Nemeth, C. P. (2004) Human factors methods for design - making systems human-centered., CRC Press. Report case where invasive surgical procedure performed in a hospital catheterization laboratory present the potential to inflict serious trauma on the patient. Problems were found in the possibility of contrast dye introducing air bubbles into the patients bloodstream which could affect the nervous system or brain.

Casey, S. (1998) Set phasers on stun: and other true tales of design, technology, and human error, Santa Barbara: Aegean. Contains 20 short stories in the book covering situations from the accidental launching of a rocket to some horrific accidents in hospitals, illustrating how quickly and easily catastrophes can happen.

Related Subjects

• Task analysis: CIT might be a complementary technique when asking domain experts to describe the tasks they perform.

• Error analysis: Here a sample of users is observed when using a product to determine all of the errors that individuals might make while using it.

• Group Interviews: Flanagan (1954) describes a group interview approach where people are asked to respond to a particular question using a critical incident form. This method is used when time and personnel are limited.

Detailed Description

Benefits, Advantages and Disadvantages

Benefits

CIT identifies possible sources of serious user-system or product difficulties. The recommendations for improvement try to eliminate the potential for the same situation to result in similar loss. However only actual use demonstrating that the product no longer induces the problem ensures that it is currently safe.

Advantages

• Flexible method that can be used to improve multi-user systems.

• Focuses on important issues e.g. safety critical events, so may bring major benefits.

• The CIT is useful for identifying rare events that might not be picked up by other methods that focus on common or everyday events.

• Can be applied using questionnaires or interviews.

• Useful when problems occur but the cause and severity are not known.

Disadvantages

• It focuses on critical incidents therefore: routine incidents may not be reported

• it is poor as a tool for general task analysis

• The method can be used in situations where you cannot observe behavior because of hazards, security, or privacy

• Since critical incidents often rely on memory, incidents may be distorted or even forgotten if the incident is collected long after an event.

Cost-Effectiveness (ROI)

This can be very high as the study may result in major problems or loss being addressed and future losses reduced as a result of the study.

How To

Appropriate Uses

The method is useful when problems occur within a system but their cause (and sometimes their severity) is not known. However the method also takes account of helpful events that may have prevented loss or countered errors.

Procedure

• Preparation: Arrange interviews with individuals who may have knowledge of hazards, unsafe practices, or events that could have or actually did result in injury or loss. It may also be useful if operators or maintainers write down firsthand accounts of what they observed. Critical incidents can also be collected using questionnaires, diaries, phone interviews, or computerized incident reporting systems. Develop consistent instructions for participants.

• Obtain materials: Obtain records such as investigation accounts and accident records from departments or agencies that have responsibilities for safety and accident reporting. Possible sources include company safety departments, insurance companies or regulatory agencies

• Gather facts: Interview individuals who have experienced problems or who have observed others who have had problems. Questions may be asked: ‘Tell me what you know about what happened’. A more structured approach may be adopted: ‘Please think of what was happening when you were carrying out activity X. Were any events particularly good or helpful to you. Were any events particularly bad or unhelpful to you? Ask participants to describe an event, what let up to it, and what happened as a result.

• Analysis: Events and descriptions are stored in a database or spreadsheet. The analyst then looks for events that occur with some frequency, how often they occur and under what conditions the events occur. Create categories of these frequent events noting the kind of problem and the item that is associated with it e.g. a particular display on a control panel.

• Interpret: Develop possible explanations for the source of the difficulty (e.g. user needs to know trend in values not just snapshot readings). With reference to improving performance, the results of a critical incident technique can be fed back into system design and the user interface to reduce or eliminate the cause of loss.

Participants and Other Stakeholders

• Personnel with experience of incidents to be surveyed or interviewed.

• An investigator or investigation team (possibly representing a regulatory body).

• System staff with the task of implementing recommendations.

• The media and the public who in serious situations will need to see that changes are being made.

Materials Needed

• Interview schedule or survey forms.

• Database or spreadsheet software to assemble and analyze recorded incidents.

Who Can Facilitate

Human factors personnel and task experts with interviewing skills and experience in analyzing human activities. It is helpful to have personnel who are not too close to the situation being investigated who may have difficulty in being objective.

Common Problems

• Respondents may be reluctant to reveal incidents that reflect badly on themselves.

• Respondents may reply with stereotypes, not actual events (using more structure improves this).

• A common questions is “How many incidents do you need to collect?” The number will vary depending on the complexity of the system.

• The wording of critical incident questions could influence the type of incidents reported (Flanagan, 1954).

• The meaning of “critical incident” must be clear to participants. Critical incidents can be minor or major events that led to negative or positive consequences.

Data Analysis Approach

This can be done using a spreadsheet. Every item is entered as an incident first. Each of the incidents is then grouped into categories. The analyst looks at the frequency of similar incidents and under what conditions are events occur. Possible explanations for the source of the difficulties are generated and validated with the systems staff or participants. The results of a critical incident technique are improvements in the procedures and system to reduce and preferably eliminate the chance of similar incidents re-occurring.

Next Steps

The implemented changes are audited to check that they are effective and similar incidents are not continuing to take place, or resulting in other problems elsewhere in the system.

Special Considerations

Costs and Scalability

People and Equipment

The critical incident investigation team may consist of 1 or more investigators depending on the scale of the exercise.

Time

The critical incident investigation team may conduct a study within a few weeks. There is an advantage in conducting a study of a serious incident quickly while relevant staff can remember the details and while associated records exist.

Ethical and Legal Considerations

Interviewees are typical assured of anonymity so that participants can describe how a product operates or what another person did without being divulged as the source of information. This is of particular importance in communities that share a strong bond such as pilots or police officers.

Political Issues

Design teams may be sensitive to a very negative report identifying many usability problems. This can be addressed by grouping problems and offering a constructive approach in suggesting practical solutions. The report should also report the positive aspects of the system identified during the study.