Thursday, September 7, 2017

A wealth of information creates a poverty of attention (Herb Simon)

Information density is one of my favorite issues. I’m pretty sure it crops up on every test I run.

And the particular problem I run into is typically the one that Simon points out here. If I had a dollar for every time I’ve written that n users couldn’t find x, and that we should “consider simplifying the UI,” I’d be a (very slightly) richer man. 

It’s also pretty obvious where it comes from too. Marketeers and execs are famous for wanting to provide as many features as possible, but to also make sure that everything is just one click away. Faced with this kind of “logic”, I try to point out the total fallacy of the old one-click rule and the primacy of information scent, but it can sometimes be a struggle.

I also like to point out that simpler interfaces are just easier to use. I think, these days at least, most people do get that. I only need to point to Google, or apps, or Nest, or even Tindr to make my point. 

I also, though, like to get them to think about the user’s experience as a whole. And that means not just whether users see each marketer or exec’s particular pet feature. 

I try to get across that a little here, a little there amounts to something I like to call “additive design,” a guaranteed way to get a system or site or app to sink under its own weight. I try to get them to consider that, as Shlomo Benartzi has written, “every new feature comes with a cost.”

Finally, I also like to tie in what I call the simple, basic “yuck factor.” My users are great at providing me with ammunition for that. If I had a dollar for every quote where a user basically says “TMI!” …  Even better, I’ve found that these quotes are typically some of the most pithy ones I get.

What I really try to get across to my clients in this instance, though, is that this visceral reaction on the user’s part has some very real consequences. In addition to making things harder to find, it oftentimes make the user not even want to try, and just give up. And that - and just clutter in general - can have a major negative impact on brand perception. And we all know how worked up marketeers and execs can get about that.


Herb Simon was quite the guy. A Nobel Prize winner, he also won the Turing Award, and made major contributions to psychology and sociology as well. He’s also the father of the “think-aloud protocol,” the basis of pretty much every usability test that's run. I was lucky enough to have met him in grad school.

Wednesday, August 30, 2017

For every complex problem there is an answer that is clear, simple, and wrong (H.L. Mencken)

A/B testing sounds like a pretty darn good idea.  In fact, on the face of it, it seems to have some real advantages over good, old-fashioned usability testing:

  • Numbers – If you’re working on a well-trafficked site and on a well-trafficked part of that site, you will probably be getting results in the 1000s. Now, compare that to the 7 to 12 you might get on a usability test.
  • Genuineness – A typical beef with usability testing is that it’s artificial – the users are in a lab, being observed, and typically doing some role playing.  In A/B testing, though, they are actually using your site, to do things they want to do, and without ever knowing they are being “tested.” 
  • Relevance – A/B results are typically tied to specific results.  These, in turn, are tied to specific business goals – account openings, purchases, downloads.  In other words, this stuff can have a genuine effect on the bottom line.
  • Concreteness – There’s a joke among usability engineers that the profession’s motto should be, “It depends.”  We sometimes don’t realize how frustrating this can actually be, though, for our clients who are more business- or numbers-oriented. A/B testing, on the other hand, is definitely something that speaks their language. 

At the same time, however, A/B testing also has plenty of possible issues:

  • Comparative only – A/B testing is a comparative method only.  In other words, you need 2 things.  But what if you only have 1?  Does that mean you’ll have to come up with a straw man in that situation?  (Usability testing doesn’t have that liability, and can also be used for comparative testing as well.)
  • Finished product – Though A/B testing is great for getting feedback in real situations, that also means you can’t try something out before it gets to a polished, ready-for-prime-time state.  (In usability testing, you can get feedback simply using pieces of paper.)
  • Emphasis on details – A/B testing tends to focus on smaller issues, like button size or color or location.  Great stuff to know, but a pretty incomplete picture.  Who knows, maybe there were things that weren’t even considered that could bump up conversion even more.  How would you ever know?  (Usability testing looks at whatever causes users problems, whatever that might happen to be.)
  • Cumulative effect – Because A/B testing often means tweak this here, tweak that there, attention isn’t always focused on the overall or cumulative effect.  Yes, that marketing tile was effective on the home page in week 1.  And, yes, that call-to-action that was added in week 6 worked well too. Does that mean, though, that we can keep adding whatever we want as the weeks go on?  I am actually familiar with a site that did just that.  And, as is right now, it’s also ready to pretty much sink under its own weight.

  • Short-term focus – As illustrated above, A/B testing is often very much about short-term effects.  Now, that fits in very well with an Agile methodology (which often relies on A/B), but that approach might also backfire in the long run.  How, for example, will that cluttered homepage impact conversion down the road, or overall?  
  • Scalability – Along the same lines …  So, increasing the size of a button bumped conversion up by 2%?  That’s great!  So, why not just bump it up again?  In other words, how can we tell when we’ve passed the line of too much of a good thing?  Heck, why should we even really care?
  • Neutral results – A lot of the press around A/B testing tends to focus around dramatic results.  From what I understand from people who are really doing it, though, typical results tend to be more along the lines of an “eh” and a shrug of the shoulders.  Now, was all that effort and expense really worth it for that 0.01% move on the needle?  Even worse, what if both designs tested equally badly?  What other data would do you have in that situation to come up with something new and different?
  • Effect on standards – One particular kind of dramatic result seems to be when the findings break conventional thinking, or go against established standards.  Now, that’s pretty fascinating, but what exactly do you with it?  Does that invalidate the standard?  Is there something about this particular instance that would be a good example of “it depends”?  Do we need to test every instance of this particular design (which is what I’m thinking the vendor might suggest)?

  • What happens next – A/B testing focuses solely on conversion.  As I mentioned above, that can be a good thing.  At the same time, though, conversion doesn’t come close to describing the totality of the user’s experience.  What if the user successfully signed up for … the wrong account?  What if the user successfully purchased the product … but did so offline?  What if they downloaded that newsletter … but never read it?  What if they signed up for your service ... but it was such a painful process that they really don't want to come back? Unfortunately, siloed marketeers often don’t care.  Just getting the user “through the funnel” is typically their only concern.   How the whole experience might affect, say, brand is something else entirely.  To quote Jared Spool, “conversion ≠ delight.”
  • Unmeasurables – Those brand concerns above hint at this one. Note, though, that unmeasurables can be a lot less lofty as well.  I, for example, do a lot of work with online banking systems.  Now, these people are already customers.  What key business metrics might they align with?  Actually, they typically have their own, very diverse goals.  They might be all about checking their balance, or looking for fraudulent transactions, or paying their bills on time.  All we really can do is support them.  Indeed, there are tons of sites out there that are more informational or that involve user goals that are vaguer or might not directly align with business KPIs.
  • Why – Perhaps the most important drawback with A/B testing, though, is a lack of understanding why users preferred A over B.  I mean, wouldn’t it be nice to learn something from this exercise so that you can apply it, say, in other situations?  Honestly, isn’t understanding why something happened a necessary precursor to understanding how to improve it?  Unfortunately, A/B testing is all a bit of a black box.  

A/B testing is basically only binary feedback.  You essentially get a thumbs-up or thumbs-down.  But perhaps there’s more to it than that.  Perhaps it does after all depend …


H.L. Mencken working on some early man-machine issues

Tuesday, June 27, 2017

Information is gushing toward your brain like a fire hose aimed at a teacup. (Doug Adams)

I think we can all identify with this one. I mean, honestly, how many emails do you get in a day?

The real question, though, is whether you can identify with your users in this regard. How complex is your website, or your app, or your software?  Does your system simply add to that information? Are you part of the problem? Or have you made an effort to be part of the solution? 

Now, as usability engineers, we have no shortage of opportunities to help out our company’s users in this regard. We also, however, need to remember our own users – the project teams we run tests for. In particular, there are a number of things we can do to make our primary deliverable to them – our reports – more effective. 

Our main goal here should be to take the fire hose of information that is the result of a usability test and turn it into something more like a tea cup’s worth. Now, the first thing to realize about this is that it is not easy. It can be very tempting to simply refashion that fire hose worth of stuff. It takes a lot of work to figure out what the flood of information actually means and boil it down to something a lot more consumable.

One thing I’ve found very useful is something an art director at a former employer shared once in a presentation about how to create better presentations. I forget the exact numbers, but it was something along the lines of 10-20-30 – meaning 10 pages, 20 words per page, and 30-pt font. I do pretty good with the last two, but found that I did have to double that first one. That said, when I subtract all the titles pages, appendices, and background info, it actually does come pretty close to the 10.

Here are some other things you can do to make your reports more easily processed:

  • Include an executive summary
  • Separate out methodological stuff and boil it down as much as you can (other than other usability engineers, very few people seem to care)
  • For results, try to include only one idea per slide
  • Use a simple, distinct, consistent layout – make it easy for readers to identify your finding, evidence (quotes, pix), and solution

Whatever you do, just be sure you can never be accused of do what I say, not what I do. 


Scott Adams, creator of Dilbert, always had a particular appreciation for usability problems

Friday, May 12, 2017

A foolish consistency is the hobgoblin of little minds. (Ralph Waldo Emerson)

Now, this one is another maxim for more advanced clients (as is this one). The challenge for less experienced clients is to recognize that consistency is actually quite a good thing. They’re the ones more likely to have a blue button on one page and an orange one on another, to have title case for one field and sentence case on the next, to have the call to action at the bottom left on one screen and the bottom right on the following one …

Unfortunately, once they really learn the value of consistency, these folks now have to start dealing with exceptions to the rule. Note, though, that this is a good problem for you to have. A rigidly consistent, even overly consistent, system is a ton better than a system that is just total chaos.

As an example of the former, I just had to deal with a consistency problem involving menus. My company offers multiple products – bank accounts, mortgages, car loans, etc. Our menus for each of these is typically a list of the actual products, but also some subsidiary and educational material as well, with these three areas marked out as sub-menus.

We are currently getting into an additional product area, and came up with a menu that includes 1 actual product, 1 bit of subsidiary material, and 1 tutorial. Needless to say, this is quite different from our other product areas, which typically include multiple examples of each type. Even so, some of the team wanted to still use the same organizing scheme – which basically amounted to three one-item lists.

Another common problem is parallel lists – i.e., lists whose items are all similar. I’m a huge fan of these, but sometimes it just doesn’t work. For example, you might have a menu with a bunch of tasks. It makes a ton of sense, then, to make these all start with action verbs – “edit,” “delete,” “create,” etc. Sometimes, though, these menus will also include some support material. But changing that from, say, “help” to something like “Getting Help” or “Understanding how to use …” is just wordy, unnecessary, and possibly confusing.

So, here are some things you can ask the overly consistent to get them to focus on, not just following the rules, but knowing when to break them as well:

  • Are these things truly the same? 
  • Is there a good reason to make this one stick out a little more? 
  • How will the user perceive them?
  • What do we accomplish by making these consistent?
  • What confusion, awkwardness, or other unintended consequences might we cause as well?

By the way, though, do not use this quote! I have a funny feeling that might not go over that well.


I’m thinking this one might have been Photoshopped

Thursday, May 11, 2017

Biased questions lead to biased answers. (Anonymous)

To be honest, I think – at least when it comes to usability testing – all questions lead to biased answers. 

As I was just telling one of our new usability engineers the other day, a usability test is not an interview. They’re supposed to be task-based. And what that means is that, other than prepping the user for think-aloud and giving them a scenario, you need never say another word. It’s all about the user completing the task and vocalizing what they are doing and thinking. The perfect test is one where you just sit there and take notes. 

Now, though I do get these every once in awhile, imperfect tests are much more common. Users only rarely will make it that easy for me. Most of the time, I have to work a little bit harder to earn my pay.

Indeed, there are no shortage of times when you have to say something. Most often, the user may simply fail to do the think-aloud. I think all usability engineers are familiar with the classic “What are you thinking?” just for those situations. Variations on this include, “What just happened?” “What’s going on?” “Is that what you expected?” and so on. And, once users do start talking, a simple “un-huh” is usually enough to keep them going.

Even if users are doing the think-aloud, though, sometimes that’s not enough. Frequently, they may leave thoughts hanging. They might say, “That’s a nice …” or “I don’t know if …” or “How did that …?”  Because we humans hate to leave anything incomplete, simply repeating what they just said will usually prompt them to complete the thought.

You can use a similar trick to get users to elaborate on vague comments like, “That’s not good,” or “I don’t like that.” Simply repeating their phrase will get them to almost magically add the “why” (and in a way that sounds a lot less accusatory than simply asking them why).

All in all, the less questions the better. And, if you do have to throw in a few, make it so they don't even sound like questions.


A lot of this advice came from an excellent talk Judy Ramey (at the University of Washington) gave at a UXPA conference many years ago

Wednesday, May 3, 2017

It’s not your job to guarantee a happy ending. (Philip Hodgson)

I like being a critic. It’s kind of fun pointing out everyone else’s foibles. And let me tell you, being a usability engineer is a great way to do that. Other people’s errors are on parade before you in test after test after test. Great fun!

Seriously, though, I’m actually not really a sadist. In fact, I’m known in particular for making a special effort to be positive when reporting results. I figure I’m just accounting for basic human nature here – nobody wants to hear their baby is ugly. And if you do need to broach the subject of some possible unsightliness, well, it’s generally a good idea to have some sugar to go along with that medicine. In general, I’d much rather be, not the scold who everyone hates and ignores, but the nice guy who actually has some valuable advice that might be worth listening to every once in awhile.

Now, it is pretty rare that I get ugly babies with absolutely no redeeming qualities. Almost everything has something good to be said for it. So, that part of it isn't really that hard.

That said, there are times when you do have communicate that, yes, we do indeed have a problem here, Houston. But even in those situations, there are still some things you can do to soften the blow.

One of these is to make sure that your client is aware that there might be a problem as testing progresses. In other words, don’t wait until the report out to bring attention to serious issues. No one likes surprises like that.

So, one thing you’ll want to do is make sure you debrief after every test. You can also send out topline reports – not full reports, but quick summaries – at the end of each day. Finally, you can also get your team to see if they can come up with a solution to any serious issues, say, midway through the test. (In fact, overcoming a problem can feel like an even more positive experience than simply having no issues pop up.)

Another thing I’ve found helpful is to allow the client to vent a little. I just try to put myself in their shoes (I know I’m a venter myself), and try not to take it too personally. Easy to say, but it really does take a little practice to get comfortable with.

Along similar lines, you’re going to have to also make sure that your data is pretty well buttoned-up. They say the best defense is a good offense, and I’ve seen plenty of clients who really go on the offensive when they hear something they don’t want to. In those situations, once again, I counsel remaining cool and calm as you fend off attacks on your methodology, users, prototype, personal integrity, whatever.

A final thing you can do is to do a good job of picking your battles. It’s pretty rare for me to fall on my sword for an issue. And that probably is something that just comes with experience. After doing this for 30 years, I know that it’s rare for something to be a real show-stopper. But there definitely have been some cases, over the years, where data from tests I ran caused products to be pulled, releases to be moved out, or projects to be shut down. 

Just be a little mindful about how to communicate results like those.


Philip is the owner of Blueprint Usability

Tuesday, April 18, 2017

Whereof one cannot speak, thereof one should be silent. (Ludwig Wittgenstein)

How tempting it can be though.

Imagine you’re in a report-out, and your client is eating up everything you have to say. You’ve really got them in the palm of your hand.  Why not just share that pet theory you had? Now, you can really only tie it to that one user. And their quote was actually pretty non-committal at that. But, heck, why not? You’re on a roll! Share this gem, and they’ll really love you.

On the other hand, you’ve also got much less positive scenarios as well. For example, one of your clients might have a great question, but – unfortunately – you don’t really have any data. Maybe it never occurred to you to collect it, and thus never made it into your test script. Perhaps you neglected to probe when the situation came up. Maybe you didn’t recruit enough of that particular type of user to really be able to say.

In fact, that last scenario is something I face all the time. Everyone needs to remember that we are dealing with qualitative data here – and the small number of people that qualitative research typically involves. Now, those numbers might be fine to show some usability mishap (a mislabeled button, a hidden link, a missing step), but when it comes to things that are more preferential, it can be hard to really say when all you’ve got is a couple of data points.

Another issue that sometimes comes up is when it’s just six of one, half a dozen of the other. In other words, there’s data for one side of an argument, and there’s data for the other. Now, you’ve most likely got a client with strong feelings in one direction (heck, you might even yourself). So,  they’ll probably feel just a little bit cheated: “Hey, I paid you all this money. I expect to see some real results. How am I gonna make a decision now?”

Basically, all it really comes down to is how comfortable you are saying, “I don’t know.” Interestingly, though, I’ve found that that will actually generate more respect for you in the long run.


And, yes, I know I’m taking this quote totally out of context  ;^)