Update: Thanks to some great commenters, I have been told that most people call this Guerilla testing. I will change the title of the post if anyone can find a guerilla-related image that is more badass than the one of the Guvernator above.

Last week I aggregated all the most current resources I had found on lean usability testing into one post. As promised, this follow up post is about my personal experiences and lessons learned from doing Commando Usability Testing (CUT). I did CUT with around 30 different users in at least 5 different locations (all cafes): 3 in Palo Alto last summer and fall, and 2 in NYC this fall and winter.

What is CUT?

I had never heard it called this until Andre‘s usability presentation last week. But what an awesome name . CUT is the most informal and low-cost style of usability testing. Basically you go into the field (usually a coffee shop) with your laptop and find random strangers to do user testing/customer development on.

Below is Andre’s Slide on Commando Testing:

Andre's slide on CUT

Why do CUT?

As Andres noted, CUT is:

• Cheap
• Requires no planning
• And makes you feel like a bad ass

But CUT’s real advantage is Flexibility.

CUT let’s you make game-time changes. Most major usability issues become very obvious very quickly. These major problems will require significant redesigns. If you’ve spent a lot of time and money recruiting 6-7 people and discover these issues after the second user, then you’ve wasted a lot of resources on those next 4 users since additional testing on the same major problems won’t be as valuable.

Cindy Alvarez recommends planning for this by intentionally leaving time to make changes after the first few user tests. Of course, there’s no way to know how much time you’ll need and when, or if, you’ll need it.

The point is that, like all things in the startup world, it’s impossible to tell how user testing will go and CUT gives you the ultimate flexibility to deal with that uncertainty. I found that after my first two tests, there were obvious changes to be made. So I made them, got rid of that road block, and immediately found new issues on the new design with new users.

Having users at arms length to test out any idea or design decision makes the iteration loop spin very fast and it’s amazing how much your design can evolve in just a day.

However, I’m not advocating CUT over other lean usability techniques. CUT is most valuable in very early stages of customer and product development. I did CUT with just mockups (using Balsamiq). Once you have a more complex prototype built out, or are just testing new features with a site that already has a userbase, the value of CUT over other in-house techniques diminishes.

How I did CUT

CUT is great in theory, but when you show up in the coffee shop and realize that you’ve got to approach people, you will suddenly realize why it’s not very popular. Since moderating usability tests is very well covered (see my last post for resources) I’m going to focus on the greatest challenge of CUT which is simply approaching people and getting them to agree to be your guinea pig.

1. Choosing who to approach

First, think about who your target user is and break down everyone in the cafe by market segments. Then break each of these groups down based on a scale from 1 to 10 of how likely they are to respond to your approach….. Actually, that was a complete joke. Don’t do ANY of that.

It sounds like something you might learn in business school, but this kind of over-analytical thinking is the reason why CUT never gets anywhere. Because we obsess over the perfect person to approach, we end up paralyzing ourselves and making excuses to never approach anyone. I tend to be a huge over-analyzer and at first I found myself constantly thinking this way and not actually doing any approaches or testing. In fact, on my first day of doing CUT I only got 3 tests done in the entire day because I kept over-thinking and was too nervous to talk to anyone.

Instead, approach anyone and everyone.

You are looking for the boulders in the road here so it doesn’t matter too much whether you find your exact target users. And most importantly, you have no idea who your target user is until you talk to them. By the time you’ve convinced yourself that there is no way you can approach that particular person, you probably could have done it and found out for yourself.

Get momentum. Once you start approaching people, it gets a lot easier. In fact, think of your first 3 approaches as warm-ups. All you’re doing is getting them out of the way so that you feel more comfortable with your opening line. If you need to really take off the pressure, go to one cafe and tell yourself that you’ll approach 3 people and then leave to never see any of them again.

However, there are a few patterns I noticed that could save you some time. Avoid approaching people on the street and trying to get them to come into the cafe. I did this when I ran out of people in the cafe and had terrible luck with it because people are usually on their way to doing something as opposed to just chilling out. I also found that people reading books tended to be really in the zone and unresponsive. But still approach them.

Don’t be afraid to approach people who look unfriendly, people reading newspapers, magazines, working on laptops, listening to music on headphones etc..

Also, definitely go for groups! I found that groups are great because they are more likely to be in a social fun mood and agree to work with you. And groups can also be great for ideation because it will often turn into a brainstorming session.

2. What to say

OK this is the one place where my experience has taught me to disagree with all the pros on usability.

Do not offer to pay them. Do not even offer to buy them coffee.

I think that paying people for traditional usability testing is obviously necessary since you are asking people to travel to your office. In that scenario, people are weighing pros and cons and making a rational decision.

In CUT scenarios, the decision is more emotional than anything else. The potential tester has just been approached by someone they don’t know and asked to do something weird when they thought they were just here to get coffee. It’s a lot easier for people to mentally justify not participating when they can say to themselves “whatever, I was already going to buy my $4 coffee anyway”

Don’t make it about the money. Appeal to their curiosity and sense of goodwill.

I like to say “Hi, excuse me, sorry to bother you but I’ve been working on a website for artists and people who love art, and I was wondering if I could quickly get your feedback on something.”

Adjust this based on who you’re approaching, and how many people there are (groups are actually great), but whatever you say, focus on communicating:

• You are doing something interesting that should spark their curiosity to find out more.
• Their commitment will be minimal – maybe only a minute or two. Of course, they can end the testing after 1 minute or whenever they want, but I find that most people get into it and end up staying until I’m finished (20-40 minutes).
• Even though this requires minimal commitment, they are doing you a big favor (which they are) and they will feel good for having helped you out.
• Even though you are approaching them, they have been pre-selected somehow. This is communicated in your description of your site. Describe your site in a way that should interest your target audience. If they’re not interested, then you’ve saved yourself some time already because they will likely not be your users. If they are interested, then they will feel a more personal connection with what you’re doing and be more likely to want to participate.

For instance, when I approached a group of young students wearing more trendy clothing, I would say something like “Hey guys, sorry to bother you, but I’ve been working on a website for artists and people who love art, and I was wondering if I could get your feedback on something. Again, sorry to interrupt, but you guys look more artistic than most people around here so I figured it was worth a shot.”

Also, make sure you’re not leaning over the table. Smile, and lean back. Come off as friendly and unthreatening. Acknowledge that what you’re doing is kind of weird!

Last Tuesday I attended Andres Glusman’s presentation on lean usability testing at the NYC Lean Startup meetup (at Meetup.com’s HQ).

Of all the startup events I’ve been to so far in NYC, none was so dense with interesting and actionable information as Andres’ presentation. So I would highly recommend you check out the slides embedded below.

This presentation reminded me of Cindy Alvarez‘s presentation on usability testing that I saw last summer at the Facebook Fund in Palo Alto. I would also highly recommend that you read through all of her slides as well since both Cindy and Andres have had a lot of experience doing these tests yet came to their conclusions completely independently. So it’s interesting to see how their experiences differ.

Helpful tools for conducting lean usability tests

silverback.com (for macs)

techsmith.com/morae.asp (for PCs)

Lean user testing with The Flip: http://www.webgrrls.com/blog/2009/05/21/best-new-tool-for-user-testing-the-flip/

A real world example of how to conduct a usability test from Steve Krug:

Here is the link to see Steve Krug on the Least You Can Do About Usability Testing (Kind of long. Fast forward to 5 minutes in.)

Outsource Your Usability Testing

Short on money, time, and man-power? I would highly recommend looking into outsourcing your user-testing. As Jacob Nielsen notes, 5 user tests will typically identify 85% of errors but 0 user tests will find 0%. Cheap and outsourced usability can be extremely effective if in-house testing isn’t an option.

Outsourced user-testing options:

I have heard the most good things about usertesting.com

And here is some extremely helpful advice on how to effectively use usertesting.com from Chris Neumann (re-printed with his permission from a post to the Lean Startup Circle google group):

“I’ve been a long time Usertesting.com customer, so I thought I’d share
a few  tips from conducting a lot of tests with a bunch of existing
and MVPs with them:

1. The testers tend to want to tell you how awesome your product is,
so I try to hide who it’s for.  I’ll say the URL to test is google.com
and to search for a solution to the problem.  In Ash’s situation, I
might say something like “you want to share pics of your kids with
other parents and family.  Search for a solution for this.  Spend no
more than 5 minutes, and then go to www.cloudfire.com and see if
that’s a good solution.

2. If you get the testers to search Google, then pay attention to what
they type in.  These words can be valuable in SEO, SEM, headlines, etc

3. You can say who can accept the test, so getting someone who
actually wants your solution makes a BIG difference.  There are lots
of stay at home Moms on there, so finding busy parents was pretty
easy.  I’m happy if I get 1 of 5 testers who want whatever I happen to
be testing.

4. The videos can be incredibly persuasive to others inside and
outside the org.  Share them.  If on step 2 of your registration flow
users are flailing, then forward the videos to the UI designer,
engineer, whoever is responsible for that page.  I’ve been shocked at
the speed at which that stuff gets fixed.  I’ve even sent them to
partners.

5. You can also use the videos to end internal debates about feature
requirements.  Recently I was working on a project where a consumer
had to choose from a list of about 20 things.  One person on the team
suggested we add a search box.  I didn’t even respond, but conducted
the user test I was doing anyway that evening, showed him the videos
and said that I didn’t think we needed search.  That was way easier
than getting into an internal debate.  In other words, these videos
can kill HiPPOs.

6.  If you already have a product, I like to say that I’m testing a
competitor, and then have the tester also test my product as a
comparison.  The testers will tend to bash my product since they think
I’m the competitor, so I’ll get some data that I might not have
otherwise gotten.

7. The support person (Aimee) is good, so if you have a question or
problem, send an email to support and they’ll generally get right back
to you.

This one is for Cindy Alvarez: in-person testing gives better data.
However, it takes more time and is harder to coordinate, so I think
that usertesting.com is way better than no testing.”

Other outsourced usability resources I’ve heard good things about:

http://www.easyusability.com/

http://fivesecondtest.com/

http://www.feedbackarmy.com/

And here’s a great post that goes through all these resources, plus many more, and does Pros and Cons:

http://www.usefulusability.com/24-usability-testing-tools/

Guerilla Style Usability Testing

And finally, there is Guerilla or, as Andres calls it, “Commando Style” Usability testing. This lies in-between outsourced and in-house testing. I haven’t yet outsourced usability testing, and have done little in-house testing. But I have ‘gone Commando’ in several coffee houses in Palo Alto and NYC, and think it’s a very powerful – not to mention free and fun – way to perform customer development.

Here is an excellent and entertaining post on how to do $5 Guerilla Usability Testing in bars.

And here is my follow-up post on my own experiences and hacks for doing Commando Style Usability testing for $0!