Users, Context, and Why Not to Test Your Coworker
As part of a four-person team at Olin College of Engineering, I designed an interface that allowed bar patrons to browse and choose cocktails based on drink attributes ("cherry," "rum," "citrus," "coffee," etc.) The opening screen showed a selection of "popular drinks" with a menu of addable attributes on the left:
When attributes were added, they appeared on the other side of the menu, and the list showed only the drinks that contained those attributes:
To get to this end product, we went through the entire design process -- interviewing users at upscale Boston bars (our job was horrible, right?), determining their needs, ideation, refinement, heuristic evaluation, testing... Although our "bar money" budget was limited by the testing stage, we were still very thorough -- we tested ALL the possible users available to us locally at our college: mechanical engineers, electrical engineers, bioengineers, chemical engineers -- heck, we even tested the computer scientists! Our interface went through repeated rounds of user testing with only slight modifications.
As a final round of testing before presenting our final prototype -- a culmination of months of work, we decided to test non-engineering staff members, business students from the school next door, and hapless parents who happened to be wandering through. Our interface failed, with every single one of the test subjects. Repeatedly, they landed on this screen, and had no idea how they had gotten there, or what to do next:
About a year later, I was working in a brainstorming session at Mobiquity with fellow mobile device developers/UX people and Jane (not her real name), from the marketing department. We were talking about the possibility of an app that would assist with package shipping. I jokingly threw out a suggestion about determining sea level and dropping the iPhone while tied to the package, to determine package weight. The marketing person suggested that the solution was much simpler than that -- just set the iPhone on a level surface, put a small package on the iPhone, and weigh it directly.
Jane was immediately corrected by someone else in the group -- the iPhone had no scale sensor, and no ability to weigh things. The moment stuck with me, however -- why had Jane, a very intelligent and successful person, been so certain that the iPhone could be trivially turned into a scale?
I realized that the reason she assumed the iPhone could be a scale was the same reason my cocktail chooser interface from last year repeatedly failed with non-technical users: in absence of context, users assume from their experience. Jane, for example, did not think of a "scale" as a particular type of sensor, in the same category as a thermometer, accelerometer, or GPS. Her experience told her that a scale was something that sat in her bathroom, was flat, shiny, and had a small digital screen on it and some simple electronics inside of it. Why couldn't an iPhone, which was much more advanced than a bathroom scale, and was also flat, shiny, and had a display, simply be programmed to be a scale?
Similarly, the "cocktail chooser" users in the more technically-inclined group performed well with an interface that was, to them, clearly a filter list. They observed what happened when items were added to the list, and compared it to filter lists that they had seen in the past (the Google Analytics interface, for example) and knew how to use it. When a group of non-technical users encountered this, they compared it against their knowledge of drop-down menus, which typically bring them to different parts of a website, and assumed they were simply viewing a page of "cherry" drinks, and then a page of "lemon" drinks -- not "cherry + lemon" drinks. This seemed reasonable when they saw the list of attributes across the top -- it could be a page history, breadcrumb, set of bookmarks or a myriad of other things they were more familiar with.
Because of my background I tend to collect examples of "computer people" vs. "not computer people," or "engineer" vs. "not engineer" but testing isn't just about making sure your grandmother could use it. What if the cocktail chooser interface were tested with drunk bar patrons, rather than sober ones? Would they be able to navigate the touch screen at all? Notice that attributes were popping up on the right side? How would a bartender or a botanist find their favorite attributes, as opposed to someone who rarely ventured outside "gin and tonic" and thought "mint" was just a type of chewing gum?
No matter how good you are, or how many years you've been in the business, it is actually impossible to "think through" an interface or a piece of software to see if it's usable by anyone other than yourself or those who are very similar to you. People use what the know, and there's no way of knowing what they know unless you go get them. Go forth, venture beyond your social circles or workplace, and test!