Chapter 2: measuring product market fit

Good morning Predictable Revenue community,

Thanks to everyone who replied last week, the overwhelming votes were for quantifying product market fit so I skipped right to Chapter 2. This one is also long because it’s the bones of the chapter, there will be more added and there is still a bunch of cleaning up to do.

Here’s my ask, what are 3 things that you like and 3 things you’d change?

tl;dr:

• Product Market Fit Strength can be measured

• I use a funnel and track the conversion rate & time

• The four stages are Acquisition, Activation, Retention, and Referral

• The formula is = Referral Rate x (1/number of days) x 1000 

Thanks for your support, here’s Chapter 2: Measuring Product Market Fit.

Collin

In a previous email I introduced the idea that Product Market Fit Strength is a measure of the pull of the market, similar to measuring the strength of a magnet. In this email, I’ll share my method for quantifying product market fit strength. I’m a salesperson at heart so it’ll be no surprise to anyone that I used a funnel. Before I get into the stages of the funnel, I think it’s important to address where and when we’re going to use this funnel. I use it as a ‘meta funnel’ that measures the strength of PMF across a number of markets, segments, personas, etc… It gives me a somewhat objective measurement of the relative strength of each variable. 

The funnel.
I started using this funnel to track my customer development process when I was struggling to articulate why one segment felt better than another. We only had subjective data, collected from our interviews, and found that our personal opinions were working their way into the conversations as facts. I had already tracked every person I talk to in a big CRM-like sheet so it wasn’t too hard to add in a few extra columns. The hardest part was filling it out (thanks Vanessa!) and keeping it up to date. Here are the stages: 

Acquisition - when we tell them about it, are people excited to try it?
Activation - do they actually sign up/pay/follow through on your CTA?Retention - when they sign up, do they keep using the product?
Referral - if they use it, was it so great that they tell their friends?

Great products that serve a large market with unmet needs tend to be so exciting that the people you encounter feel a need to tell everyone they know. When I first started interviewing people for Carb.io (something that did quite well), people would stop me part way through an interview and breathlessly ask, “wait! Are you working on solving this?!” I could feel their excitement that finally the painful experience of not having a tool for X would be over. 

Tracking the referrals you get from each customer is a great proxy measurement for excitement. It’s objective and reinforces something you should be focusing on, making people so happy they tell their friends. I mostly cared about two numbers, the percentage of people we talked to that referred us to another user and how long it took to go from first conversation to first referral. 

Interviews —> Referrals —> Customers.
Some of you may have noticed that these are Dave McClure’s Pirate Metrics minus Revenue. I intentionally left Revenue out because I start measuring these stats when I’m pre-revenue and usually pre-product. This is not a perfect model because there are some problem spaces where referrals are not natural or allowed. If this is the case for you, then revenue is a fine proxy. 

What we’re trying to capture by measuring the referral ratio is the % of your users that think your product is so great that they have to tell their friends. The reason this is the primary focus is that the people you run your customer development interviews with should be your first customers. Your next customers are very likely to come from referrals from your first customers. If I’m not able to walk someone from first conversation to referral it is a strong indication that growth will be challenging. It doesn’t mean I don’t have a product or a market to sell it to, it’s just that the market isn’t pulling my product from me very strongly. I’m trying to measure that pull. 

It’s important to tag your users by different variables when you’re exploring. With Athlon, we had 15 characteristics that we tracked under the user profile, 10 for the reason they exercised, and 1 ICP/Persona category. When you can filter your funnel by different variables, it can help you identify trends that you might miss when looking at the broader audience. When we first started working on Athlon, we thought we’d focus on semi-pro athletes but we also interviewed some random friends of ours just to build a wide sample. After the interviews, it was clear that semi-pro athletes weren’t interested, I didn’t need a fancy spreadsheet to figure that out, they straight up told me. 

When we looked at the data, it was pretty mediocre. What was interesting, was that there was a cohort of folks that, when we filtered by only their data, had really high scores. It was a cohort of folks that used to play sports or work out 4-5 times a week but were struggling to regain the habit after having kids. Many of them were exercising because they wanted to get back to a previously lower weight. I belonged to this group but it was my buddy Karl, who also belonged to the cohort, that gave it the name “fat dads”. It started out as a bit of a joke but the data backed up that it was a pretty interesting niche. 

A low referral rate does not necessarily mean you do not have product market fit and won’t be able to build a business on top of it. It means that your go to market investments won’t go as far. 

Athlon, a recent example.
Athlon is the most recent startup that I tried to validate and I interviewed ~47 people. No matter where I am in my product development stage, I always end every customer development interview by asking if, when it’s ready, they’d be interested in trying it out. It doesn’t have to be a “will you pay for it” conversation, just a casual ask if they’d try it. Especially in the early days. As the product becomes usable, I’ll start to ask for some money in exchange for using it but usually at a 95% discount. The goal isn’t to get rich but to see if people will give you some cash to try the shitty first version.

Here are my customer development interview stats for Athlon: 

Total conversations - 47 
Conversations are Stage zero in my funnel. Everyone I talk to starts here. If the meet the entry criteria of the next stage, they get moved down. If you follow this model, make sure you and everyone you’re working with are using the stages in the same way. 

Acquisition - 38 (80%) 
Step one in my customer development funnel is Acquisition, when I ask people if they want to try the thing, do they say yes? For Athlon, the last product I tested with this method, roughly 80% said yes. What I learned from the 20% was that most of them belonged to an ICP that didn’t care about the problem we wanted to solve. 

Activation - 26 (68%)
My next move was to call their bluff, let’s call this Activation. For the 38 people that said they’d be interested in checking it out, I sent them an invite to try the platform. This is where the rubber met the road because it is much easier to say yes to a hypothetical offer than a real one. For Athlon, the numbers weren’t so great, in my opinion, and only 26 people actually clicked the link and signed up. 

Retention - 10 (38%) 
Once people were live on the platform, I wanted to see how many actually continued to use the product. This is where we ran into trouble. We ended up retaining 38%. It was an indication that our product wasn’t amazing yet, which we knew because it was super early. As people exited the funnel, I would reach out to book interviews with them so I could understand why they dropped off. 

Referral - 9 (90%) 
The thing that surprised me the most was the referrals. Of the 10 people that ended up using the platform, almost all of them referred at least one additional user that signed up. This was a really strong indication that we had found something interesting. 

While we did find that people had the pain we thought existed, we couldn’t find a way to create a real business around it. The users were consumers but we wanted to pursue a B2B model selling to employers and couldn’t find a good way in. Our team of 3 ended up running out of steam and we went our separate ways. There’s probably still an opportunity here, we just weren’t able to make progress on it given the time and resource constraints we had. 

The Funnel.
At least you can learn from our experience, here’s what our final funnel metrics looked like: 

Total conversations - 47 
Acquisition - 38 (80%) 
Activation - 26 (68%)
Retention - 10 (38%) 
Referral - 9 (90%) 

We tracked it all in a big spreadsheet called SHARED: External Interviews. 

Our total funnel efficiency was 9 / 47 or 19% from conversation to referral, which in hindsight is pretty strong. It took us an average of 30 days to move someone from Conversation to Referral which wasn’t as great. 

Your numbers will be different and that’s ok. The reason to represent this as a funnel is it tells you where you need to focus if you want to make things better. For Athlon, I could see that we were losing most of our customers at the retention stage. That told me that while we had something interesting, our tool needed to be better if we wanted users to onboard unassisted. 

The Formula.
Sometimes the market just pulls the product out of you. In these situations, the conversion rate to referral doesn’t accurately represent the full strength of your product market fit and you need to add a time variable into the mix. A funnel that converts to referral in 7 days is significantly better than one that takes 30 days to accomplish the same feat. Here is the formula I’ve used to calculate my Product Market Fit Strength:

PMF-S = Conversation to Referral Rate x (1/Time) x 1000 

*Time = number of days from Conversation to Referral

If I compare the scores of Athlon, the startup we worked on for 14 months and never made any revenue, with Carb.io, our startup that scaled $0 - $1m in a few months, it’s pretty clear that one is better than another. 

Athlon Strength of PMF = 0.19 x 0.033 x 1000 = 6.27

Carb Strength of PMF = 0.40 x 0.14 x 1000 = 56 

These scores are snapshots from a specific time period, they’re only useful as a way of comparing against one another. This is just a proxy I use when I’m comparing projects that I’m considering. Be careful when you compare your score to someone else’s. If there’s one thing I’ve learned from helping fix sales teams it’s that everyone uses their stage definitions a little differently. 

The End + CTA.
Wow, you made it, congrats and thank you. I know some of those sections are still super rough.

What are 3 things you’d keep and 3 things you’d change?

Collin

The PS.
For next week - I’m thinking of chapter 1 (introduces the idea of pmf strength + a formula for growth) or the strategy of a sale. Let me know what you think.

I’ve taken all week off to work on finishing the book so I’ll hopefully have plenty of content by this time next week.