Update: Thanks to some great commenters, I have been told that most people call this Guerilla testing. I will change the title of the post if anyone can find a guerilla-related image that is more badass than the one of the Guvernator above.

Last week I aggregated all the most current resources I had found on lean usability testing into one post. As promised, this follow up post is about my personal experiences and lessons learned from doing Commando Usability Testing (CUT). I did CUT with around 30 different users in at least 5 different locations (all cafes): 3 in Palo Alto last summer and fall, and 2 in NYC this fall and winter.

What is CUT?

I had never heard it called this until Andre‘s usability presentation last week. But what an awesome name . CUT is the most informal and low-cost style of usability testing. Basically you go into the field (usually a coffee shop) with your laptop and find random strangers to do user testing/customer development on.

Below is Andre’s Slide on Commando Testing:

Andre's slide on CUT

Why do CUT?

As Andres noted, CUT is:

• Cheap
• Requires no planning
• And makes you feel like a bad ass

But CUT’s real advantage is Flexibility.

CUT let’s you make game-time changes. Most major usability issues become very obvious very quickly. These major problems will require significant redesigns. If you’ve spent a lot of time and money recruiting 6-7 people and discover these issues after the second user, then you’ve wasted a lot of resources on those next 4 users since additional testing on the same major problems won’t be as valuable.

Cindy Alvarez recommends planning for this by intentionally leaving time to make changes after the first few user tests. Of course, there’s no way to know how much time you’ll need and when, or if, you’ll need it.

The point is that, like all things in the startup world, it’s impossible to tell how user testing will go and CUT gives you the ultimate flexibility to deal with that uncertainty. I found that after my first two tests, there were obvious changes to be made. So I made them, got rid of that road block, and immediately found new issues on the new design with new users.

Having users at arms length to test out any idea or design decision makes the iteration loop spin very fast and it’s amazing how much your design can evolve in just a day.

However, I’m not advocating CUT over other lean usability techniques. CUT is most valuable in very early stages of customer and product development. I did CUT with just mockups (using Balsamiq). Once you have a more complex prototype built out, or are just testing new features with a site that already has a userbase, the value of CUT over other in-house techniques diminishes.

How I did CUT

CUT is great in theory, but when you show up in the coffee shop and realize that you’ve got to approach people, you will suddenly realize why it’s not very popular. Since moderating usability tests is very well covered (see my last post for resources) I’m going to focus on the greatest challenge of CUT which is simply approaching people and getting them to agree to be your guinea pig.

1. Choosing who to approach

First, think about who your target user is and break down everyone in the cafe by market segments. Then break each of these groups down based on a scale from 1 to 10 of how likely they are to respond to your approach….. Actually, that was a complete joke. Don’t do ANY of that.

It sounds like something you might learn in business school, but this kind of over-analytical thinking is the reason why CUT never gets anywhere. Because we obsess over the perfect person to approach, we end up paralyzing ourselves and making excuses to never approach anyone. I tend to be a huge over-analyzer and at first I found myself constantly thinking this way and not actually doing any approaches or testing. In fact, on my first day of doing CUT I only got 3 tests done in the entire day because I kept over-thinking and was too nervous to talk to anyone.

Instead, approach anyone and everyone.

You are looking for the boulders in the road here so it doesn’t matter too much whether you find your exact target users. And most importantly, you have no idea who your target user is until you talk to them. By the time you’ve convinced yourself that there is no way you can approach that particular person, you probably could have done it and found out for yourself.

Get momentum. Once you start approaching people, it gets a lot easier. In fact, think of your first 3 approaches as warm-ups. All you’re doing is getting them out of the way so that you feel more comfortable with your opening line. If you need to really take off the pressure, go to one cafe and tell yourself that you’ll approach 3 people and then leave to never see any of them again.

However, there are a few patterns I noticed that could save you some time. Avoid approaching people on the street and trying to get them to come into the cafe. I did this when I ran out of people in the cafe and had terrible luck with it because people are usually on their way to doing something as opposed to just chilling out. I also found that people reading books tended to be really in the zone and unresponsive. But still approach them.

Don’t be afraid to approach people who look unfriendly, people reading newspapers, magazines, working on laptops, listening to music on headphones etc..

Also, definitely go for groups! I found that groups are great because they are more likely to be in a social fun mood and agree to work with you. And groups can also be great for ideation because it will often turn into a brainstorming session.

2. What to say

OK this is the one place where my experience has taught me to disagree with all the pros on usability.

Do not offer to pay them. Do not even offer to buy them coffee.

I think that paying people for traditional usability testing is obviously necessary since you are asking people to travel to your office. In that scenario, people are weighing pros and cons and making a rational decision.

In CUT scenarios, the decision is more emotional than anything else. The potential tester has just been approached by someone they don’t know and asked to do something weird when they thought they were just here to get coffee. It’s a lot easier for people to mentally justify not participating when they can say to themselves “whatever, I was already going to buy my $4 coffee anyway”

Don’t make it about the money. Appeal to their curiosity and sense of goodwill.

I like to say “Hi, excuse me, sorry to bother you but I’ve been working on a website for artists and people who love art, and I was wondering if I could quickly get your feedback on something.”

Adjust this based on who you’re approaching, and how many people there are (groups are actually great), but whatever you say, focus on communicating:

• You are doing something interesting that should spark their curiosity to find out more.
• Their commitment will be minimal – maybe only a minute or two. Of course, they can end the testing after 1 minute or whenever they want, but I find that most people get into it and end up staying until I’m finished (20-40 minutes).
• Even though this requires minimal commitment, they are doing you a big favor (which they are) and they will feel good for having helped you out.
• Even though you are approaching them, they have been pre-selected somehow. This is communicated in your description of your site. Describe your site in a way that should interest your target audience. If they’re not interested, then you’ve saved yourself some time already because they will likely not be your users. If they are interested, then they will feel a more personal connection with what you’re doing and be more likely to want to participate.

For instance, when I approached a group of young students wearing more trendy clothing, I would say something like “Hey guys, sorry to bother you, but I’ve been working on a website for artists and people who love art, and I was wondering if I could get your feedback on something. Again, sorry to interrupt, but you guys look more artistic than most people around here so I figured it was worth a shot.”

Also, make sure you’re not leaning over the table. Smile, and lean back. Come off as friendly and unthreatening. Acknowledge that what you’re doing is kind of weird!