Heuristic Framework for Evaluating Web 2.0 Applications

Please reference this post, if you happen to use the information for discussing heuristics for web 2.0 applications. I plan to publish an expansion on this sometime later this year!

My report framed in the previous post draws many of its conclusions on the basis of judgments applied to the 10-point framework for analysis detailed in this paper.

This framework describes both general heuristics and also design patterns embodied by Web 2.0 applications (O’Reilly 2005, Hinchcliffe 2006; numbered bullets 1-7 below). Though usability professionals have speculated on what defines success for Web 2.0 applications, some of the most popular sites do very poorly when judged by traditional usability metrics (Alexandra and Dix, 2007). This runs counter to expectation. If the user interfaces are riddled with problems, why are theses sites still so successful? Traditional usability heuristics do not have explanatory power nor are they even reliable indicators for user satisfaction in many instances. Thus, this paper presents a new framework for analysis that takes into account properties shared by successful Web 2.0 applications.

Web 2.0 Meme Map from O’Reilly 2005

Usability heuristics are general “rules-of-thumb” guidelines useful for explaining problems in an interface while design patterns explicitly focus on context and tell the designer when, how and why the solution can be applied. Hence, the patterns and heuristics used in this assessment structure evaluation in terms of “known good traits” for well-designed Web 2.0 applications and mash-ups (Preston 2007). In particular, this analysis was posed to support issues concerning map-based interaction. Specific criteria for analysis are included in Table 1. This framework incorporates what Tim O’Reilly (2005) calls the seven core “competencies” of Web 2.0 applications (five of the above come directly from Hincliffe 2006). But as Hincliffe remarks, “Web 2.0 is really a set of related forces, design patterns, and business models that are increasingly emerging onto the world stage. And these elements frequently defy detailed technical quantification.”

Bullet point 8 draws on the notion that aesthetics matter. “If you cannot attract a user to stay on a website, it doesn’t matter how well designed operational usability may be” (Sutcliffe 2002, Norman 2004). Interactive sites generate affective responses and are judged in terms of their engagement and appeal. The effects of aesthetics are dependent on the target audience, the domain and the application type. This assessment only includes a very limited perspective of aesthetics.

Additional, related heuristics for exploration / navigation and transactions (bullet points 9 and 10) have been incorporated into the framework. Most of these are traceable back to traditional usability metrics such as Nielsen 1994. The 10-point framework is described below:

1. Global platform (service, data, devices) - Global means end-to-end connectivity from device to machine to net. The focus is on services with cost-effective scalability.

2. Ease of data consumption, access and remix - In particular, we are concerned with user ownership of data. Users demand control over their data and value the ability to easily consume it, access it and mix it with other data. This is of great importance given the challenges of synchronizing data across home and work machines and maintaining copies of data across multiple devices.

3. Continuous and seamless updates - Gone are the days of “monolithic” releases. Users are realtime testers and your services can be instrumented to know how these users are interacting with the system. Changes and improvements are rolled out on a continual basis.

4. Customer self-service - ”Narrow niches make up the bulk of internet’s the possible applications. Therefore: Leverage customer-self service and algorithmic data management to reach out to the entire web, to the edges and not just the center, to the long tail and not just the head.” (O’Reilly 2005)

5. Richly interactive user interface - Ajax and Flash bring interaction capabilities to the web that rival those on the desktop. Some of the more important usability design patterns in this space have to do with making asynchronous server calls so that the user does not have to wait for http transactions (e.g., screen adjacent Google Map tiles load in the background). There are many useful and good design patterns listed at the Yahoo! Design Pattern Library.

6. Architecture of participation that encourages user contribution - Users add value by adding their data to that provided by the service-provider. By aggregating “user data as a side-effect of their use of the application”, network effects ensue. (O’Reilly 2005)

7. Trusted host - The largest concern by people who use hosted services is, “can I trust this host”? Taken explicitly, the question can mean a variety of things, “Is it my data or does it belong to the host? Is my data safe? Will it be there tomorrow? Will it be transformed or altered?”. Trust also refers to the ability of the user to trust the host as a source of information.

8. Attractiveness - Going beyond Web 2.0 competencies, attractiveness refers to aesthetics, minimalism, branding and message.

9. Exploration / Navigation - The ability for a user to be able to find information easily and navigate through a task space can define success or failure for an application.

10. Transactions - Transactions refer to the set of issues regarding communication between a machine interface and human. Thus, understanding, error diagnosis, and error repair apply.

While heuristics are very good at finding problems in the user interface and design patterns for making an impact with fairly little effort and time, additional techniques are needed lend assistance and insight into problems concerning task execution, and also assist with application re-design (Gray 1995, Ling and Solvendy 2005). So a third tool utilized in this study is an informal task analysis for the purpose of eliciting information about deeper structural and task-oriented problems as well as providing a useful framework for the re-design.

Alexandra, P. and A. Dix. 2007. Usability - not as we know it!. Proceedings of HCI 2007. The 21st British HCI Group Annual Conference, 3-7 September 2007, University of Lancaster, UK.

Gray, W. 1995. Discount or Disservice? Discount Usability Analysis–Evaluation at a bargain price or simply damaged merchandise? ACM CHI 1995 Mosaic of Creativity, May 7-11.

Hinchcliffe, D. 2006. The state of Web 2.0. Social Computing Magazine, April 2. .

Ling, C., and G. Salvendy. 2005. Extension of heuristic evaluation method: a review and reappraisal. Ergonomia IJE&HF, Vol. 27(3): 179–197.

Nielsen, J. 1994. Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 - 28, 1994). B. Adelson, S. Dumais, and J. Olson, Eds. CHI ‘94. ACM Press. New York, NY: 152-158.

Norman, D. 2004. Emotional Design: Why We Love (or Hate) Everyday Things. New York: Basic Books.

O’Reilly, T. 2005. What is Web 2.0?. O’Reilly.com. .

Preston, S. 2007. Deep into the AJAX user experience. Ajax World Magazine, 29 April. .

Sutcliffe, A. 2002. Assessing the reliability of heuristic evaluation for website attractiveness and usability. Proceedings of the 35th Annual Hawaii International Conference on System Sciences.