They also paid well. Really well. Even more than my friends received working 100 hour weeks at I-Banking jobs. In retrospect, no college student should ever have been paid that much (on the bright side, the savings were enough for Art.sy’s initial funding).

But that summer it meant I could go out to nice dinners with my girlfriend, and never worry about paying for drinks at expensive clubs. It meant I could afford fancy clothes, an iPhone, and plane flights to Asia. Having always worked in labs prior to that job, it redefined how I thought about money.

So what is wrong with this picture? I had an extremely fun and challenging job, working with awesome people, that let me afford an incredible lifestyle. It was a dream come true.

But at the end of the summer, the CTO brought me into the corner office and closed the door. I had worked with him all summer and this was my last day, so I was expecting a performance evaluation. Instead, after some chit chatting, he asked me a question:

“Have you ever wondered what value we create here?”

Value? This wasn’t what I was expecting at all.

“Not really.”

“I’ll tell you. We increase the liquidity of the secondary bond market. We shave basis points off of spreads.”

I’ll never forget that question. It turns out that our CTO was saving every penny and had plans of leaving as soon as he had enough cash to pursue his dream.

He didn’t care about the fancy clothes, the clubs, or being a master of the universe. All he cared about was how he would add value to the world. At this point, my story starts to sound cliche, but it was a cliche I needed to experience in person because it radically changed my perspective.

“How am I creating value?”

I realized that the programs I had spent all summer writing were great, if they could make people money and save them time. But if all it resulted in at the end of the day was slightly more efficient markets, well, what was the point of that?

I was so caught up in the fun and camaraderie of my job, so high with the rush of money, I never considered such a simple question.

This probably won’t change the minds of people who have already chosen career paths. But to any students who are thinking about their futures, I hope my story illustrates how easy it is to get swept up by short term pleasures, and how important it is to always ask this question when making important decisions.

Now I’m in NYC starting a company, and he’s leading marines through combat exercises in the woods of Quantico, Virginia. Yet despite our completely different jobs, I was fascinated to discover that we are learning very similar things.

For one, my roommate’s training gives him an intuitive understanding of lean startup principles. The lean startup methodology has only become popularized recently in the startup world. And I think that’s because its principles are very unintuitive. Most startup founders plan everything out years in advance with no expectation for their product or business model to change.

Marines are trained to never plan beyond first encounter with the enemy because “you can never predict in advance how the situation will change.” Instead marines must learn to make quick decisions and become comfortable with uncertainty as the situation changes unpredictably from moment to moment. As my roommate described the process of dealing with enemy combatants, I couldn’t help but think of an entrepreneur performing customer development and pivoting his product and business model around market feedback.

We’ve also experienced similar lessons in leadership and agree that a lot of leadership comes from practice and experience. Effectively leading a platoon of 30-50 marines has a lot to do with the confidence that comes with comfort over time. At first my roommate was very nervous about giving orders to large groups of marines, but over time he started to care less and less. Now he doesn’t bat an eye.

When I was preparing for business plan competitions, I realized the same thing. I was incredibly nervous at the prospect of public speaking. I realized that I just needed to make myself comfortable. So I practiced my pitches hundreds of times until I could visualize all my slides and give the pitch with my eyes closed. I even walked around campus saying my pitch out load to get used to people giving me weird looks or laughing so that audience reactions wouldn’t throw me off on the big day.

Last week there was a great NextNY event on Disruptive Commerce Models or E-Commerce 2.0. Jordan Cooper later wrote a post claiming that the line between emerging commerce models and ad models is blurring.

I couldn’t agree more, and I think there is a 3rd blurring of the line in the commerce space: and that is social media, or more specifically, User Generated Content that is linked (implicitly or explicitly) to commerce.

I am not talking about user generated inventory, as that is nothing new, just consider the success of product companies like Etsy and Threadless, and also service companies like oDesk and eLance. These are great examples of “commerce light” companies that avoid inventory risks and costs. Jordan argues that these companies can be thought of as more advanced advertisers since they perform the same function of driving consumers towards the suppliers. However, what about social media companies that focus on enabling consumer to consumer interactions and converting them into sales?

For other companies, like SponsoredTweets.com,Twitter is a platform for creating conversations and social engagement that leads down the funnel to sales.

In the same way that Commerce 2.0 companies avoid inventory risk and focus on discovery, these social commerce companies aim to avoid discovery risk, and focus on user generated content. Instead of optimizing for search and advertising, they optimize for the viral loop embedded in social interactions. Yet ultimately these services are all part of the same customer acquisition funnel that results in a sale between a consumer and supplier.

Although no-one wants to think about social interactions as ad units, in the abstracted framework Jordan presents in his blog post, there is no doubt that the user generated, or C2C, commerce model of companies like Blippy and Hunch are also becoming harder to distinguish from the new generation of Ad and Commerce startups.

Update: Thanks to some great commenters, I have been told that most people call this Guerilla testing. I will change the title of the post if anyone can find a guerilla-related image that is more badass than the one of the Guvernator above.

Last week I aggregated all the most current resources I had found on lean usability testing into one post. As promised, this follow up post is about my personal experiences and lessons learned from doing Commando Usability Testing (CUT). I did CUT with around 30 different users in at least 5 different locations (all cafes): 3 in Palo Alto last summer and fall, and 2 in NYC this fall and winter.

What is CUT?

I had never heard it called this until Andre‘s usability presentation last week. But what an awesome name . CUT is the most informal and low-cost style of usability testing. Basically you go into the field (usually a coffee shop) with your laptop and find random strangers to do user testing/customer development on.

Below is Andre’s Slide on Commando Testing:

Andre's slide on CUT

Why do CUT?

As Andres noted, CUT is:

• Cheap
• Requires no planning
• And makes you feel like a bad ass

But CUT’s real advantage is Flexibility.

CUT let’s you make game-time changes. Most major usability issues become very obvious very quickly. These major problems will require significant redesigns. If you’ve spent a lot of time and money recruiting 6-7 people and discover these issues after the second user, then you’ve wasted a lot of resources on those next 4 users since additional testing on the same major problems won’t be as valuable.

Cindy Alvarez recommends planning for this by intentionally leaving time to make changes after the first few user tests. Of course, there’s no way to know how much time you’ll need and when, or if, you’ll need it.

The point is that, like all things in the startup world, it’s impossible to tell how user testing will go and CUT gives you the ultimate flexibility to deal with that uncertainty. I found that after my first two tests, there were obvious changes to be made. So I made them, got rid of that road block, and immediately found new issues on the new design with new users.

Having users at arms length to test out any idea or design decision makes the iteration loop spin very fast and it’s amazing how much your design can evolve in just a day.

However, I’m not advocating CUT over other lean usability techniques. CUT is most valuable in very early stages of customer and product development. I did CUT with just mockups (using Balsamiq). Once you have a more complex prototype built out, or are just testing new features with a site that already has a userbase, the value of CUT over other in-house techniques diminishes.

How I did CUT

CUT is great in theory, but when you show up in the coffee shop and realize that you’ve got to approach people, you will suddenly realize why it’s not very popular. Since moderating usability tests is very well covered (see my last post for resources) I’m going to focus on the greatest challenge of CUT which is simply approaching people and getting them to agree to be your guinea pig.

1. Choosing who to approach

First, think about who your target user is and break down everyone in the cafe by market segments. Then break each of these groups down based on a scale from 1 to 10 of how likely they are to respond to your approach….. Actually, that was a complete joke. Don’t do ANY of that.

It sounds like something you might learn in business school, but this kind of over-analytical thinking is the reason why CUT never gets anywhere. Because we obsess over the perfect person to approach, we end up paralyzing ourselves and making excuses to never approach anyone. I tend to be a huge over-analyzer and at first I found myself constantly thinking this way and not actually doing any approaches or testing. In fact, on my first day of doing CUT I only got 3 tests done in the entire day because I kept over-thinking and was too nervous to talk to anyone.

Instead, approach anyone and everyone.

You are looking for the boulders in the road here so it doesn’t matter too much whether you find your exact target users. And most importantly, you have no idea who your target user is until you talk to them. By the time you’ve convinced yourself that there is no way you can approach that particular person, you probably could have done it and found out for yourself.

Get momentum. Once you start approaching people, it gets a lot easier. In fact, think of your first 3 approaches as warm-ups. All you’re doing is getting them out of the way so that you feel more comfortable with your opening line. If you need to really take off the pressure, go to one cafe and tell yourself that you’ll approach 3 people and then leave to never see any of them again.

However, there are a few patterns I noticed that could save you some time. Avoid approaching people on the street and trying to get them to come into the cafe. I did this when I ran out of people in the cafe and had terrible luck with it because people are usually on their way to doing something as opposed to just chilling out. I also found that people reading books tended to be really in the zone and unresponsive. But still approach them.

Don’t be afraid to approach people who look unfriendly, people reading newspapers, magazines, working on laptops, listening to music on headphones etc..

Also, definitely go for groups! I found that groups are great because they are more likely to be in a social fun mood and agree to work with you. And groups can also be great for ideation because it will often turn into a brainstorming session.

2. What to say

OK this is the one place where my experience has taught me to disagree with all the pros on usability.

Do not offer to pay them. Do not even offer to buy them coffee.

I think that paying people for traditional usability testing is obviously necessary since you are asking people to travel to your office. In that scenario, people are weighing pros and cons and making a rational decision.

In CUT scenarios, the decision is more emotional than anything else. The potential tester has just been approached by someone they don’t know and asked to do something weird when they thought they were just here to get coffee. It’s a lot easier for people to mentally justify not participating when they can say to themselves “whatever, I was already going to buy my $4 coffee anyway”

Don’t make it about the money. Appeal to their curiosity and sense of goodwill.

I like to say “Hi, excuse me, sorry to bother you but I’ve been working on a website for artists and people who love art, and I was wondering if I could quickly get your feedback on something.”

Adjust this based on who you’re approaching, and how many people there are (groups are actually great), but whatever you say, focus on communicating:

• You are doing something interesting that should spark their curiosity to find out more.
• Their commitment will be minimal – maybe only a minute or two. Of course, they can end the testing after 1 minute or whenever they want, but I find that most people get into it and end up staying until I’m finished (20-40 minutes).
• Even though this requires minimal commitment, they are doing you a big favor (which they are) and they will feel good for having helped you out.
• Even though you are approaching them, they have been pre-selected somehow. This is communicated in your description of your site. Describe your site in a way that should interest your target audience. If they’re not interested, then you’ve saved yourself some time already because they will likely not be your users. If they are interested, then they will feel a more personal connection with what you’re doing and be more likely to want to participate.

For instance, when I approached a group of young students wearing more trendy clothing, I would say something like “Hey guys, sorry to bother you, but I’ve been working on a website for artists and people who love art, and I was wondering if I could get your feedback on something. Again, sorry to interrupt, but you guys look more artistic than most people around here so I figured it was worth a shot.”

Also, make sure you’re not leaning over the table. Smile, and lean back. Come off as friendly and unthreatening. Acknowledge that what you’re doing is kind of weird!

Last Tuesday I attended Andres Glusman’s presentation on lean usability testing at the NYC Lean Startup meetup (at Meetup.com’s HQ).

Of all the startup events I’ve been to so far in NYC, none was so dense with interesting and actionable information as Andres’ presentation. So I would highly recommend you check out the slides embedded below.

This presentation reminded me of Cindy Alvarez‘s presentation on usability testing that I saw last summer at the Facebook Fund in Palo Alto. I would also highly recommend that you read through all of her slides as well since both Cindy and Andres have had a lot of experience doing these tests yet came to their conclusions completely independently. So it’s interesting to see how their experiences differ.

Helpful tools for conducting lean usability tests

silverback.com (for macs)

techsmith.com/morae.asp (for PCs)

Lean user testing with The Flip: http://www.webgrrls.com/blog/2009/05/21/best-new-tool-for-user-testing-the-flip/

A real world example of how to conduct a usability test from Steve Krug:

Here is the link to see Steve Krug on the Least You Can Do About Usability Testing (Kind of long. Fast forward to 5 minutes in.)

Outsource Your Usability Testing

Short on money, time, and man-power? I would highly recommend looking into outsourcing your user-testing. As Jacob Nielsen notes, 5 user tests will typically identify 85% of errors but 0 user tests will find 0%. Cheap and outsourced usability can be extremely effective if in-house testing isn’t an option.

Outsourced user-testing options:

I have heard the most good things about usertesting.com

And here is some extremely helpful advice on how to effectively use usertesting.com from Chris Neumann (re-printed with his permission from a post to the Lean Startup Circle google group):

“I’ve been a long time Usertesting.com customer, so I thought I’d share
a few  tips from conducting a lot of tests with a bunch of existing
and MVPs with them:

1. The testers tend to want to tell you how awesome your product is,
so I try to hide who it’s for.  I’ll say the URL to test is google.com
and to search for a solution to the problem.  In Ash’s situation, I
might say something like “you want to share pics of your kids with
other parents and family.  Search for a solution for this.  Spend no
more than 5 minutes, and then go to www.cloudfire.com and see if
that’s a good solution.

2. If you get the testers to search Google, then pay attention to what
they type in.  These words can be valuable in SEO, SEM, headlines, etc

3. You can say who can accept the test, so getting someone who
actually wants your solution makes a BIG difference.  There are lots
of stay at home Moms on there, so finding busy parents was pretty
easy.  I’m happy if I get 1 of 5 testers who want whatever I happen to
be testing.

4. The videos can be incredibly persuasive to others inside and
outside the org.  Share them.  If on step 2 of your registration flow
users are flailing, then forward the videos to the UI designer,
engineer, whoever is responsible for that page.  I’ve been shocked at
the speed at which that stuff gets fixed.  I’ve even sent them to
partners.

5. You can also use the videos to end internal debates about feature
requirements.  Recently I was working on a project where a consumer
had to choose from a list of about 20 things.  One person on the team
suggested we add a search box.  I didn’t even respond, but conducted
the user test I was doing anyway that evening, showed him the videos
and said that I didn’t think we needed search.  That was way easier
than getting into an internal debate.  In other words, these videos
can kill HiPPOs.

6.  If you already have a product, I like to say that I’m testing a
competitor, and then have the tester also test my product as a
comparison.  The testers will tend to bash my product since they think
I’m the competitor, so I’ll get some data that I might not have
otherwise gotten.

7. The support person (Aimee) is good, so if you have a question or
problem, send an email to support and they’ll generally get right back
to you.

This one is for Cindy Alvarez: in-person testing gives better data.
However, it takes more time and is harder to coordinate, so I think
that usertesting.com is way better than no testing.”

Other outsourced usability resources I’ve heard good things about:

http://www.easyusability.com/

http://fivesecondtest.com/

http://www.feedbackarmy.com/

And here’s a great post that goes through all these resources, plus many more, and does Pros and Cons:

http://www.usefulusability.com/24-usability-testing-tools/

Guerilla Style Usability Testing

And finally, there is Guerilla or, as Andres calls it, “Commando Style” Usability testing. This lies in-between outsourced and in-house testing. I haven’t yet outsourced usability testing, and have done little in-house testing. But I have ‘gone Commando’ in several coffee houses in Palo Alto and NYC, and think it’s a very powerful – not to mention free and fun – way to perform customer development.

Here is an excellent and entertaining post on how to do $5 Guerilla Usability Testing in bars.

And here is my follow-up post on my own experiences and hacks for doing Commando Style Usability testing for $0!

Becoming a State Space Traveler

Imagine that life and the pursuit of happiness is really an optimization problem. Imagine that we spend our lives traversing a gigantic endless landscape full of mountains and valleys. The landscape represents the space of all different possible beliefs, values, and experiences that you can have at any moment: your state. And the height of the landscape represents how happy you are at that state. Imagine that the lower you are in the landscape, the happier you are.

That’s why most people take the path of least resistance. They spend their lives walking straight down hill or squatting at the bottom of a valley; because that’s the most comfortable place to be. But this presents a problem: when you’re at the bottom of a valley, you can only see places that are higher up than you, in this case, states where you would be less happy.

So it is easy to make the mistake of thinking that there is no point in exploring your landscape. It’s hard to imagine that there might be deeper valleys beyond the ridgeline of your own.

Here is a common example of how we get stuck in local minima: many people spend their lives in ‘the valley of money,’ where the local minimum is defined by having as much cash as possible. When you consider the ultimate goal of the money valley, you realize the futility of such optimization. Whether you have ten dollars or ten billion dollars, you are no closer to your ultimate goal of infinite money. Consider that most people who win the lottery end up no happier one year later. They’ve got the money, fame, and freedom, yet they can’t get themselves any closer to being truly happy. Why is this? Why can’t they escape the valley of money?

Our ancestors evolved to optimize for local minima such as sex, security, and resources. Once we have settled into local minima, we grow comfortable and see no reason to leave. And it doesn’t help that we are all automatically born into the local minima that our parents occupy. That is why the vast majority of the world believes in the same religion as their parents and holds the same cultural values. These local minima are vast and very difficult to escape. Most of us have no idea that a great landscape even exists beyond the walls of the local minimum into which we were born.

Leaving the comfort of your local minimum is unsettling. Even though taking that step requires confidence, you appear less confident to the outer world because you are not sure what you believe in. You lose your values and your convictions. You no longer know what is right and what is wrong. You can no longer make yourself feel superior by passing judgement on others. You lose your sense of identity and must expose your ego directly to the fact that you don’t actually know everything. Many people who have been through this share a common shocking experience: “I suddenly realized that I actually knew nothing.”

But for those of us trapped in a local minima, once we decide to climb out and ascend a high ridge of uncertainty and discomfort, we will see a whole beautiful world that was invisible to us from the valley, a vast landscape full of far deeper valleys than the one you came out of. And you will also see even higher mountain ranges in the distance that stretch up into the clouds. And you will know that to find out what is beyond them, you must climb their steep cliffs. When you occupy these high places of discomfort, you feel lost and exposed. It is windy and cold.  But from there it is much easier to descend back down and ultimately reach deeper valleys and get closer to finding your global minimum: true enduring happiness.

Simulated Annealing: A Mathematically Proven Way to Find True Happiness

So we’ve established what you have to gain by leaving your current local minimum. But you also can’t spend your whole life exploring randomly through the state space because one day you’ll drop dead. So given the constraints of the human life time, what is the best strategy for traveling the state spaces and achieving true happiness?

1. First of all, give yourself as much time as possible.

In other words: exercise regularly and don’t eat crap food. The longer you live, the more exploring you can do, and the more time you can spend enjoying the benefits of your exploration and changing the world. And maybe you will live forever.

2. Increase the size of your state space.

This means consider every possible idea, belief, and value no matter how crazy or wrong they seem at first. Think about how many great scientific discoveries came from mistakes in the lab, or how many great companied evolved from bad initial ideas. People with the largest state spaces change the world. They are the ultimate creative types and big dreamers whose singular visions define humanity’s achievements. These people are always open to the newest experiences and cherish the craziest ideas. They surround themselves with people from diverse backgrounds and don’t let their egos blind them to critical feedback.

3. Use simulated annealing to search your state space.

Simulated annealing is an algorithm that can find the global minimum of a state space. Which, in our case, means finding ultimate happiness. In fact, it has been proven that under certain conditions simulated annealing can always find the global minimum of any arbitrarily large state space. This is a very exciting idea but how does it apply to our lives?

To answer that we need to know how simulated annealing works. Imagine that there is a ping pong ball and you want to get it to the lowest point in a landscape. To perform simulated annealing, you would heat up the temperature of the landscape so that the ping pong ball starts bouncing around crazily and randomly between all the different places. After heating up the ping pong ball to very high temperatures, you slowly lower the temperature. As the temperature lowers, the ping pong ball bounces around less energetically and slowly starts to settle down in lower parts of the landscape.

It has been mathematically proven that if you heat up the landscape to an infinite temperature and lower that temperature slowly enough, the ping pong ball will always end up in the global minimum of the landscape. Even for an infinitely large landscape.

So how can we apply this mathematical proof to living our lives? Well if we heat the ping pong ball up to an infinite temperature, this means it randomly moves around the landscape at an infinite speed and is simultaneously at all locations and also no location. The ping pong ball doesn’t occupy any single state. For us, this means we must do the same and completely clear our minds of all beliefs and values. This is very hard since from birth we’re inundated with belief systems. However, meditation is a powerful practice for emptying our minds and releasing ourselves from preconceptions.

From this place of clarity, we are completely free to explore any part of the state space without bias and beliefs that were embedded in us as children. This is the part where you slowly cool down your state space and start considering new beliefs and values. However, given that our state space is very large, this must be a very careful and slow process. But if we live long enough and chose those values slowly and carefully enough, we will eventually reach ultimate happiness. It is a mathematical certainty.