Other Resources

Concerning User Assessment of Usability

Arnie Lund
alund@acm.org

USE HOME PAGE

Adams, D. A., Nelson, R. R., and Todd, P. A. (1992). Perceived usefulness, ease of use, and usage of information technology: A replication. MIS Quarterly, 16(2), 227-247. With minor changes, this study replicated Davis' work.

Ajzen, I. (1988). Attitude structure and behavior. In Pratkanis, A. R., Breckler, S. T., and Greenwald (Eds.), Attitude Structure and Function. Hillsdale, NJ: Erlbaum.

Ajzen, I., and Fishbein, M. (1980). Understanding attitudes and predicting social behavior. Englewood Cliffs, NJ: Prentice Hall. This offers a theoretical approach to understanding user evaluations of products, and is the basis of the Technology acceptance Model.

Bailey, J. E., and Pearson, S. W. (1983). Development of a tool for measuring and analyzing computer user satisfaction. Management Science, 29(5), 530-545.

Brooke, J. (1996). SUS: A "quick and dirty" usability scale. Usability Evaluation in Industry. Taylor and Francis. A free scale developed by DEC for use in evaluating software systems. Several of the items have a more general counterpart in the USE Questionnaire. See also www.redhatch.co.uk/sus.html

Carr, H. H. (1992). Factors that affect user-friendliness in interactive computer programs. Information and Management, 22, 137-149.

Chin, J. P., Diehl, V. A, Norman, K. (1988). Development of an instrument measuring user satisfaction of the human-computer interface, In Proc. If ACM CHI '88 (Washington, DC) 213-218. CS-TR-1926, CAR-TR-328. QUIS is a tool for evaluating software user interfaces based on a semantic differential approach. Some of the dimensions assess global views of system usability and user satisfaction, and are adaptable to Likert items. See also http://www.cs.umd.edu/projects/hcil/Research/1994/quis.html, and see lap.umd.edu/QUISFolder/quisHome.html for pricing information and other research using the instrument.

Churchill, G. A., Jr., and Surprenant, C. (1982). An investigation into the determinants of customer satisfaction. Journal of Marketing Research, 19, 491-504.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, September, 319-340. Davis used Brown's materials and developed scales to measure ease of use and usefulness, arguing that they might serve as fundamental determinants of system use. Past studies had found these variables are related to self-predicted use, and that they drive choices among alternatives, they influence the rate of adoption of innovations, and they impact the development of user preferences. Davis' survey of 37 published research papers dealing with user reactions to interactive systems helped build the initial instrument. The items he identified are very close to those that emerged in the USE Questionnaire studies.

Davis, F. D., Bagozzi, R. P., and Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982-1003.

Fishbein, M., and Ajzen, I. (1975). Belief, attitude, intention and behavior: An introduction to theory and research. Reading, MA: Addison-Wesley.

Hill, Smith, and Mann (1987). They demonstrated the importance of efficacy beliefs in the decision to adopt an innovation, and that prior experience appears to influence the beliefs indirectly. Efficacy is related to perceptions about ease of use and complexity. They also found an effect of instrumentality beliefs, which are related to usefulness. The two appear to affect intention, which in turn is related to behavior.

ISO 9241-11 (1998). Guidance on usability. Available in the US from the American National Standards Institute, info@ansi.org, www.ansi.org. This lists several widely cited components of usability. Although derived from practice and professional convention more than research, they bear a striking resemblance to the Usefulness, Satisfaction, and Ease of Use dimensions.

Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In Jordan, P., Thomas, B., and Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis. SUMI is widely cited for its usefulness for software evaluation. It has been carefully developed and fits within a larger methodology. See www.ucc.ie/hfrg/questionnaires/sumi/pricing.htm for pricing and ordering information and access to other information about SUMI. See also www.ucc.ie/hfrg/questionnaires/wammi/ for information about the WAMMI questionnaire being developed to evaluate Web sites. It is similar to the SUMI, but includes diagnostic items for the Web.

Kirakowski, J., and Corbett, M. (1993). SUMI: The Software Usability Measurement Inventory. British Journal of Education Technology, 24(3), 210-212. See Kirakowski (1996).

Lewis, J. R. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7, 57-78. Lewis conducted a factor analysis of 3 questionnaires used internally to IBM, and found 3 factors: usefulness, interface quality, and information quality. While some summary items were included, the focus of the questionnaires was diagnostic.

Lund, A. M. (1997). Expert ratings of usability maxims. Ergonomics in Design, 5(3), 15-20. A study of the heuristics design experts consider important for good design. See also www.ameritech.com/corporate/testtown/amerbldg/rules.html

Lund, A. M. (1998). Damaged merchandise? Comments on shopping at outlet malls. Human-computer Interaction, 13(3), 276-281. Comments on a critique of usability evaluation metrics, and the need for and importance of subjective ratings of usability.

Lund, A. M. (1998). The need for a standardized set of usability metrics. In the Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society, 688-691. A paper in the "unsolved problems" session. Includes a table showing how the scales can be used to compare products.

Mathieson, K. (1991). Predicting user intentions: Comparing the technology acceptance model with the theory of planned behavior. Information Systems Research, 2(3), 173-191. The Technology Acceptance Model has been very robust and has been replicated in a variety of contexts.

Melon (1990). A theoretical assessment of the user-satisfaction construct in information systems research. Management Science, 36(1), 76-91. A commonly used construct in MIS research is satisfaction. Satisfaction is assumed to be predictive of usage, and is influenced by experience.

Moore, G. C., and Benbasat, I. (1991). Development of an instrument to measure the perceptions of adopting an information technology innovation. Information Systems Research, 2(3), 192-222. Building on Rogers (1993) work, they designed a set of scales to study innovation adoption. Their work appears to confirm the importance of ease of use and usefulness in the diffusion of a new technology.

Morris, M. G., and Dillon, A. (1997). How user perceptions influence software use. IEEE Software, 14(4), 58-65. They applied the instrument developed by Davis and colleagues and the Technology Acceptance Model to interface design. They found that the usefulness and ease of use scales had a strong bearing on user acceptance and usage. Interestingly the items Davis used are very close to those identified as part of the USE Questionnaire studies across a wider range of domains.

Oppermann, R., and Reiterer, H. (1997). Software evaluation using the 9241 evaluator. Behavior and Information Technology, 16(4/5), 232-245.

Patterson, P. G., (1993). Expectations and product performance as determinants of satisfaction for a high-involvement purchase. Psychology and Marketing, 10(5), 449-465.

Robey, D. (1979). User attitudes and management information systemuse. Academy of Management Journal, 22(3), 527-538.

Rogers, E. (1993). Diffusion of Innovations. New York, NY: The Free Press. Rogers surveyed several thousand innovation studies and found that relative advantage and compatibility (which appear to be similar to usefulness and ease of use in content) are among the 5 key factors appearing throughout the studies and that influence the rate of diffusion of an innovation.

Schwartz, A. L., and Seifert, C. (1996). Can a usable product flash 12:00?: Perceived usability as a function of usefulness. In Proceedings of the Human Factors and Ergonomics Society’s 40th Annual Meeting, Santa Monica, CA, 313-317. This was the first Ameritech study, and it was designed to assess how consumers evaluate the various artifacts in their lives (anything from Caller ID and voice mail to light switches and VCRs). The two most important dimensions that were important to consumers were Ease of Use and Usefulness. Both factor analysis and multidimensional scaling were used to analyze a variety of rating and comparison data.

Shneiderman, B. (1998). Designing the User Interface. Reading, MA: Addison-Wesley Publishing Co.

USE HOME PAGE