Thursday, March 29, 2018

There are three kinds of lies: lies, damned lies, and statistics. (Benjamin Disraeli)

I have a mixed relationship with statistics. 

On the one hand, I know their an important part of our history. I also know there some of us out there who still do a bang-up job with them (Jeff Sauro and Jim Lewis come immediately to mind), Finally, I also realize that stats definitely have their use.

I myself, however, am much more qualitative. I generally like to leave stats to things where they are a much better fit – surveys, web analytics, A/B testing, big data ...  I’m more than happy with my rich, why-based qualitative data.

That said, I do use stats.  Because of the qualitative nature of my data and the much smaller number of participants I typically have, though, my stats tend to be much more on the lighter side.  For example, if I’ve got my typical 10 users, I think it’s good for my audience to know if it was 2 users who had a certain problem versus 9 of them.  

I also typically eliminate any 1-offs. I find they typically draw too much attention to themselves by their mere inclusion.  That also has the added benefit of making my report easier to write, and easier to read as well. 

I also might do error rates.  Once again, though, I try to keep it simple.  If the user aces a particular task, that’s a 100.  If they veer off somewhere but eventually get back on track, or if they complete everything but miss 1 small part, that’s a 50.  If they give up and look to me (and even then complete it with some help from me), that’s a 0.  I add ‘em all up, and then generally use a simple academic grade scale – 90 and above is an A, 80 to 90 a B, and so on.  I just want to give my audience a rough idea of how the system performed.  (BTW, time on task – because of think aloud – means absolutely nothing to me.)

Finally, I also like to do a SUS. Note, though, that scores for individual tests mean almost nothing on their own.  I do keep a running total of all tests we run however. And that let’s me compare 1 particular score to all the rest, to an average for the type of system or user group, or for previous tests of the same thing. Very helpful.

So, yes, I do use stats.  But only in the simplest, most direct ways that pretty much anyone – business types, developers, writers, graphic designers … could understand.  


I have no idea why he’s on a phone case, but there you have it

No comments:

Post a Comment