Even a bad usability test will help improve your software. (David Travis)

Hmm, not sure how I feel about this one. And I’ve seen plenty of bad usability tests over the years.

Now, I’m a firm believer that you can always get something out of a test. And maybe that’s all that Travis is getting at here. 

I know I’ve definitely screwed up at times. When it comes to users, for example, I’ve had not enough, overweighting in one area, a turkey or two … And there are also times when the prototype will be a little rough as well. Finally, I might also find that parts of my script might be less than ideal.

But those kind of things really don’t matter all that much. For users, for example, I find it’s pretty easy to just delete the ones that don’t work out. You can always reschedule new ones or, say, just go with 8 instead of 10.

As for prototypes, if it’s something major (e.g., it won’t even load), I’ll just scratch those tests as well, get that fixed & then try again. If it’s something minor (which is much more common) I’ll just keep the data, make sure the issue get fixed, and plow forward. That said, if it’s something that might affect very specific results, I might just throw out the data for those initial users, but write up the issue anyway.

Something very similar can also happen with script issues as well. To be honest, though, I’ve been doing this for 30+ years, so this is probably the thing that comes up the least.

Now, you’re probably thinking to yourself, Isn't there something missing here? And, yes indeed, there is – facilitation. Now, having stopped counting users at 5,000, I’ve pretty much seen everything. Though I did have some slip-ups & befuddlements in my earlier years, I feel pretty confident I can prep for & handle anything now.

If, however, you’re not a seasoned veteran, facilitation can be a real issue. Are you biasing the user? Are you giving away the answer? Are you interacting too much, turning your usability test into a friendly get-together with an old friend? Now, these can be biggies.

Something similar can happen with your screener too. A poor screener will simply give you the wrong audience, and what might be a problem for who you ended up getting may not be for your true audience (and vice versa, of course). But how would you ever know?

As for scripts, the cardinal sin here is typically giving the game away in the scenarios. You know, pseudo-tasks like "go to page x" - instead of going to page x for a very good reason (you want to buy y, you need to transfer money from a to b, you need to get from your house to place z …)

For prototypes, it’s mainly a question of whether it supports your tasks. And let’s hope those tasks are the right ones as well – common tasks, high-impact tasks & tasks that get at questions that your project team wants answers on. In other words, if you don’t have the right tasks, you’ll have incomplete (and possibly misleading) results.

You know, it could be just about anything in your test plan, to tell you the truth. What happens, for example, if your usability test should really be a focus group, or in-depth interviews, or an unmoderated test instead of a moderated ones?

Heck, though, if you’ve got methodology wrong, I’m not sure David’s quote would still stand. Hard to get orange juice out of an apple.

If you don’t know where you’re going, you’ll end up someplace else. (Yogi Berra)

I’ve been doing usability tests for 35 years now. And I still am never attempted to just “wing it.”

For me, even the simplest test needs at least some kind of test plan & some kind of script. In fact, I’m a firm believer that:

1. A good planning meeting means a good test plan
2. A good test plan means good prep
3. Good prep means easy facilitation
4. Easy facilitation means good test sessions
5. Good test sessions mean good data
6. Good data means good analysis
7. Good analysis means good reporting
8. Good reporting means your team’s actually motivated to changes that will affect the user experience in a positive way

Now, I’m not saying that every test needs to be a formal effort. If you’re at least cognizant of the above steps & maybe write some of them down on paper, you should be fine. The bigger the effort, though, the more formal you’ll probably want to make it.

Whatever the size of your project, probably the biggest benefit you’ll get out of good planning is something that you may not have considered, confidence. It might just be me, but I don’t work very well when I’m flustered. I forget things, I get things out of order, I say the wrong thing, I hem and haw, I don’t exactly inspire confidence in my user (or my observers) … 

Now, I do know some other folks who honestly do thrive under pressure. In fact, those kind of people often need a little jolt of some kind like to really get things rolling. You know the types – that guy in college who was always pulling all-nighters, the exec who can make off-the-cuff remarks in front of 200 people, that usability engineer who’s never run a pilot test in their life.

To each his own, but usability tests typically have way too many moving parts to ever really “wing it.”

Yogi and some cutting-edge tech (for the 1950s, that is) 

Effective error messages inform users that a problem occurred, explain why it happened, and provide a solution so users can fix the problem. (Microsoft)

Honestly, is this still a thing? If memory serves me correctly, my first conference presentation was about this very topic. Forgive me for having a little trouble remembering, though, as that was all the way back in the 1980s.

I guess, though, not everyone in UX these days was around back then. Heck, I would imagine the great majority of them probably hadn’t even been born yet. 

That said, I do see a lotta UXers without much background in UX as well these days – designers with fine arts degrees, content strategists who studied Shakespeare or did the sports beat at the local paper, project managers who have backgrounds in sports marketing … Funny, though – it was much the same when I started.

But I digress …

I’ve always liked to see these ideas in user terms – i.e., the things that a user might actually ask. Is something wrong?  What is it?  How do I fix it?

In addition to that, I think it’s also important to speak the same language as the user. So, no “invalid operators,” “bad requests,” or “system error codes.” (By the way, a lot of these come from the days when developers wrote error messages. Don’t be like them!)

Related to that is to “speak human.” Just think of that poor user, who is obviously already having problems. The last thing they need is a “fix” that doesn’t fix anything, or that talks down to them, or confuses them, or makes light of their situation. No better way to guarantee their going from mildly annoyed to seriously pissed off.

Finally, be specific. Be concrete. Use examples. This extends to both how to fix the problem as well as explaining what went wrong (within reason, for the latter – remember who this is being written for). So, no more “enter a valid number,” “please try again,” or “correct name and resubmit.” Human minds tend to be a lot more concrete that abstract. What exactly constitutes a valid number? I can try again, but is there something I should be doing differently? What exactly was wrong with the name I entered?

Overall, think in the terms that the user might. And that’s really the main idea here. No matter who you are or where you came from, just think like the user. And if you look at it that way, it is true – the basics never really go away.

The more things change …

There are downsides to everything; there are unintended consequences to everything. (Steve Jobs)

Unintended consequences are a huge part of the tech world.  Facebook was going to bring the world together, remember? Instead, we got fake news, reality filters, abuse & stalking, depression & alienation, erosion of privacy, and the destruction of local journalism. So much for social media.

How about the sharing economy? Well, for Uber, we’ve got price surging, physical abuse of drivers & passengers, drivers who can barely make ends meet, drivers with criminal records, cabbies being put out of business, and a corporate culture that looked like a cross between Animal House & The Wolf of Wall Street. For Airbnb, we’ve got party houses, destruction of historic neighborhoods, stolen & damaged items, breaking of local laws, exorbitant fees, and illegal subletting. So, so much for that.

Amazon? Twitter? WeWorks? Don’t make me laugh.

I wonder if Jobs had any regrets. I mean other than Foxconn suicides, sweatshops, slave & child labor, overpricing, walled gardens, planned obsolescence & incompatibility, anti-competitive practices, tax dodging, spin, secrecy, environmental issues, digital addiction, Apple maps, scandals (Fortnite, batterygate, John Wiley & Sons, Think Secret), as well as what an awful person Steve Jobs actually was ... 

Honestly, maybe the only unintended consequence he really cared about was getting found out.

Or maybe that reality distortion field really did work - perhaps even on himself.

Trust the data! (anonymous)

Well, yes and no.

Just think about Covid.  There’s plenty of data out there.  Which, actually, might be the main problem.  There’s just too much of it for the average Joe or Jane.  

In that regard, this really becomes more of a question of, “Who am I going to trust to make sense out of it all?”  Indeed, it all kind of boils down to the old question, “What’s your source?”

Now, I’m not going to do my own research on this particular issue. I will do plenty of research, though, on where my data's coming from.  

Even if I trust the source, though, I’m going to be a lot less trustful when that source simply dumps a bunch of data in my lap.  How am I supposed to make sense of it?  Isn’t that their job?  Why would they put the burden on me?  I’m not buying it.

How does all this apply to UXR?  Well, think of your own client.  They’re probably drowning in data.  Your report out for a usability test, or competitive analysis, or 3rd party research, or whatever is not the only data they will be dealing with that week.  They’re probably also got a ton of stuff from analytics, vendors, dashboards, KPIs – you name it.

So, the question is really, "How can my stuff rise to the top of the pile?"  And the answer to that should be, “How can I make it easier for them?”

Clients need reports that:

• Are short & sweet
• Get to the point
• Are clear & well-organized
• Are actionable
• Are convincing
• Involve at least some data (number of users, quotes, clips)
• Build trust

Remember, the report is not about you - your methodology, findings that only you find interesting, interminable FYIs, clever but hard-to-interpret diagrams, simple brain dumps (because those are so much more easy to do), fascinating one-offs or rat holes ...  It’s about the system you’re evaluating.  In particular, it’s about how it’s performing & how that can be improved.  And that needs to get to your client.  When it really comes down it, it’s really all about that audience of yours.

A user interface is well designed when the program behaves just as the user thought it would. (Joel Spolsky)

Ah yes, mental models. How I love them. And how helpful they are. They also actually come up a surprising amount. 

I find that they are often to blame when it comes to sneaky, more complex problems – i.e., ones that aren’t obvious from the start, like labeling or inconsistency or text size ... When you really have to wrack your brains over what just made that test bomb so badly, mental models are often the culprit.

Think about it though. When there’s a mismatch between your model and the user’s, it can be a lot like talking with someone at cross purposes or, worse, speaking two wholly different languages. 

I mean, honestly, hand gestures and middle school French are only going to get you so far. For a truly smooth interaction, you and your user really need to speak the same language, be on the same page, sing from the same choir book …

Now, all mental models are are simply ways for people to understand complex systems or ideas. I don’t need to know the details of the internal combustion engine to know how to drive a car, or a degree in library science to check out a book, or machine language to use a word processor. 

All I have to know is enough so I can use the car or library or spreadsheet to accomplish what I set out to do. Too much detail, and I’m overwhelmed. Not enough, and I can’t predict how the system is going to work and what I‘m supposed to do. When my model is just right, though, it’s a super-handy way to make my way around the system and accomplish my goals.

There are three main problems with mental models – in particular, when the system:

• Totally lacks a model 
• Has a model, but it’s too complex or inconsistent to figure out
• Has a model, but it doesn’t match the user’s model

In general, though, they’re great things. Probably the most important part is making sure that we understand that last bit (the user’s mental model) and then build around that. Using a mental model that makes use of what the user knows from the real world can also be very helpful (you know, a metaphor). Perhaps, most importantly, we just need to use them more – and more consciously.

Joel’s a developer with a real eye for UX. He created Trello; was the major factor behind Excel, Visual Basic, StackOverflow; has authored 5 books; and may be most well-known for his blog, Joel on Software.

Would you like to find out about your problems in the lab, or on Twitter? (Susan Campbell)

This actually happened to me once. 

It started out as a nice New Year’s Day. I hadn’t partied too hard the night before, and wasn’t finding it too difficult to wake up.

Just then, though, I got a call from, of all people, my boss. Now, she wasn’t the type to call and wish me a Happy New Years, so I scrunched my eyes, tilted my head, frowned, sighed, and picked up the phone.

What I could make out from her somewhat incoherent call was that something was terribly wrong with the new look and feel for our authenticated site that we had just rolled out on the stroke of midnight. Turns out users did not like it. The issue, which seems to indeed have started on Twitter, had gotten all the way up to the second-in-command. 

To remedy the situation, the higher up wanted to do some emergency usability testing. “Some what?” I enquired. My boss tried to make some sense of it and explain it to me. We talked some more, and eventually agreed to come in to the office to just to make the higher-ups happy. They were going to supply us with some customer names, and involve some others so I could coordinate testing times with them. It was all a bit crazy.

Luckily, none of this could really happen (on New Year’s Day, mind) without a major act of government, so it all eventually petered out.

Now, here’s the kicker … We had actually done some usability testing – actually, lots of usability testing. Turns out they had just ignored all my results. But that’s a topic for another post.

Susan is VP of UXR at HBO Max