Sprint:Usability Evaluation

From ssc.aspirationtech.org

Usability Evaluation:

A usability evaluation is a quasi-experimental way of getting feedback on a design, a prototype or a working product from a target user population.

Participants: Typically, several (3-5) participants all of whom match a single target user population. How to recruit users? In this case, one participant in the Sprint evaluated the very rough draft of the running application and gave feedback.

Equipment In this case the developer's laptop served as the machine that the participant used. See our casual set up The developer served as moderator and a separate notetaker kept a record on paper. In ideal settings the observers sit in a separate room and view the session through one way windows. The moderator may sit near the participant or may sit in the separate room and communicate by a microphone connection.

Tasks: Typically, the users follow a "think-aloud" protocol, explore the application and then complete a series of predefined tasks. In this case the user explored the pages that were working, gave feedback about the behavior or functionality he expected to find, and tried to complete a task related to creating a list.

Observations from user:

• some terminology was confusing (e.g., "core") <li>liked organizing things by task <li>felt it would be challenging to get familiar with the categories <li>seeing [a larger number] (160) under "Donation Management" category suggests it's a strong category; vs. only (4) for "Email Management" <li>on a 1st visit, a low risk, high benefit activity would be research <li>expects the list to be cut by
  <li>software <li>topic/category (as "donation management") <li>organization served <li>recently reviewed or recently added/updated items <li>top N contributors to the site/to the category

  <li>(Asked whether the site should be a portal? E.g. childcare or other vertical sector as the entry point?) prefers overall view, as many NTAPs work across categories <li>wants to see the Feature set ("must be able to do X,Y,Z") for a particular software or solution <li>"If I have a weak bond, I would direct [clients] to use the SSC" (implies that not only NTAP could be target audience, but also clients of NTAPs) </ul>

  Defining a typical task: (Scenario) A non-profit needs to set up a listserve to develop a user support community.

  (Imagine) The non-profit knows something about email, their timeline, their budget, how large the audience will grow to and how to manage users. The NTAP will be knowledgeable about some specific dimensions of the software category and would ask questions related to features and functions, such as,

  <li>Are you willing to have ads show up? <li>Do you want to be able to add subscribers in batch mode? <li>Should the interface be brandable (by the non-profit org) or does it have a default look?

  The NTAP's goal is to minimize his time. His questions would help them evaluate or compare different software (which we assume is catalogued already by SSC). His role is "interpreter of attributes". He will prepare an annotated list or take the non-profit's list of N items and add his recommendations (N-m items). He'd make either his recommendations public, or would prepare an "Ok to recommend" list.

  The global "edit" function that showed up on the early pages is confusing, as each individual item has "edit" by it (appropriately).

  If one is browsing, then edit options are expected to be disabled. If one is not logged in, then can't edit. Even if logged in, some editing privilege needs to be controlled (not everyone ought to be able to edit everything). If one is contributing (signed in), then "edit" might best give values for each field, to require picking only from facets that apply (rather than creating values). Custom facets may also be needed beyond the default facets (values). Some discussion of whether private facets should be permitted, or whether all values would be public. Some specialized or custom criteria might be needed.

  Prefers a criterion list over going to each vendor's URL and not finding the info.

  Some discussion of whether and how to be able to sort lists: 1-10? ABC? Tiers? But in general, heterogeneity is preferred to strict sorting. Values for a field such as Support might be "Yes/No, Some, no data, unknown"

  Reviews are expected - or at least links to reviews. This, however, breaks the model previously shown of objective criteria (Name of Software Package, type of license, version number and date released, Platform(s) or OS required, ...)

  Discussion of whether SSC is a structured wiki? (Answer: Perhaps)

  
  Findings:

   This user is optimistic that the functionality shown in the rough prototype will indeed be useful to his consulting practice with non-profits.  He was intrigued by the potential to have even less-knowledgeable clients to use the system themselves with guidance. 
  

  More about usability evaluation:

   This pseudo-laboratory activity requires the session moderator to remain neutral.
   The moderator tries to get as much feedback from the individual user as possible by asking probe questions or open-ended questions.  
   Typical instructions or questions include 
  

  <li>"Before you click anywhere on the screen, tell me what you see and what you think each item does"; <li>"What do you think will happen next?"; <li>"What just happened? Did you expect that?"; <li>"What would you like to be able to do to complete your task?"; <li>"Were you able to complete the task?"

   If the engineer or designer who created the application or screens is also the evaluation moderator, the hardest parts of running a study are 
  

  <li>staying neutral, not explaining or defending the words, buttons, other visual or navigational devices <li>continuing to invite feedback <li>capturing the feedback to be able to adjust the attributes, such as size, shape, (color), functionality, layout and transitions after the study; this is why many studies keep the moderator role separate from the designer or engineer-implementer and why many use recording devices (video, audio, web-recording) <li>holding back from changing things on the basis of feedback from a single user (other than obvious bugs)