Product Operations: do you need it, or is it just a fad?

Product Operations: do you need it, or is it just a fad?

Recently, I’ve been hearing about more and more companies building “Product Operations” functions.  This appears to be especially true of startups with high operational-complexity; particularly marketplaces.

So, what is Product Operations? Do you need it? Or, is it just another fad?

I spoke with several founders/leaders at a number of high-growth and operationally complex startups who have invested in Product Operations functions, including Uber, Thumbtack, and Zipline.

So, what is it?

The definition of “Product Operations” varies somewhat between companies but here are the main themes I hear:

  • can be part of the Product organization, but distinct from Product Managers
  • can also report into the Operations organization
  • works very closely with operational teams
  • very detail-oriented and data-driven, especially as it relates to process optimization
  • provides the bridge between the operations teams and Product Managers, solving problems and being supportive, and ensuring communication is effective

The simplistic distinction between Product Management and Product Operations seems to be as follows:

  • Product Management – “what should we build?”
  • Product Operations – “is what we’ve built working?”

So, is it a fad?

Product Operations vs Product Management

This is where my own cynicism started and where I got the first whiff of a possible fad.

To me, being very close to customers, even if internal customers, and having a great finger on the pulse of whether what you’ve built is working is a major aspect of any Product Manager’s job.  If a Product Manager is not doing that, they’re not a good Product Manager, right?

Pragmatism Rules

Everyone I spoke to agrees with me…in theory.

However, the reality is that it’s very hard to find Product Managers who have a natural affinity for what’s involved in operating a complex business “at the coal face”.  This makes sense because most Product Managers come up through the ranks of product and engineering organizations.

Although a Product Manager’s job is to understand and empathize with users, it seems it can be more pragmatic to hire people with an operational background who just more naturally “get it” and put them in a Product Operations role as the go between operation users and the product team.

Put simply, one person I spoke to said that Product Operations “makes sure operations are getting respected” by the Product organization.

What to look for

So, who makes a great Product Operations person?  Here’s a summary of what I’ve heard:

  • have an operational background (vs a product/engineering background)
  • adaptive
  • empathetic
  • data-driven
  • strong personalities willing to fight for what they think operations needs

Side effects

Are there any undesirable side-effects of introducing a Product Operations function? The consensus seems to be that two problems can occur:

  • there is some duplication of effort/ownership between the Product Management and Product Operations and therefore some potential ambiguity and politics that needs to be carefully managed.  To mitigate this, the distinction between the two functions summarized above must be made very clear to both sides.
  • the introduction of Product Operations risks Product Managers retreating further to their ivory towers, allowing them to get more divorced from the (internal) customers they serve.
Advertisements

Marketplaces: Scaling with Operations vs Engineering?

There are many things that make building and scaling marketplace businesses hard: for example, there’s the quintessential chicken-and-egg problem of building and balancing supply and demand, and there’s the need to build two or more products in parallel to serve the needs of the different participants in your marketplace.

There is also the question of how you scale your marketplace once you’ve got product-market fit established and some unit economics that seem to work.

Electrons vs Atoms

Most marketplaces have to deal with the tangible, real world: unlike pure software/SaaS companies, marketplaces have to deal with whole atoms, rather than just electrons.

Those atoms might make up people, or cars, or meals, or apartments but they are physical resources that have to be managed. This is why marketplaces tend to need significant operational headcount.

However, most marketplace companies aspire to be, and actively position themselves as, technology platform companies.  This of course requires an ongoing investment in product/engineering.

Given finite resources, how do you choose between scaling a market place through operation headcount versus product/engineering investment?  How do you strike the right balance?

The Comparables

I did some quick research to look at what other marketplace businesses are doing.

I took a basket of marketplace companies at varying funding stages and looked at their employee counts on LinkedIn by role.

Firstly, let’s set the scene by looking at the absolute number of engineers that various marketplaces have and compare that to their funding stage:

Perhaps no surprises here: as marketplaces develop, they hire more engineers. I am struck, though, by the widely varying number of engineers that the earlier stages marketplaces seem to have.

Now let’s look at the ratio between operational headcount and engineering headcount in these same companies:

This is also what you might expect.  Although the data is noisy*, it seems that as marketplaces grow, they become less dependent on operational headcount. Presumably, their investments in product/engineering payoff in terms of automations and efficiencies.

Of course there’s also survivorship bias here – these are only the marketplaces that are still around. Perhaps the ones that didn’t make it had wildly different ratios.

What would be great is to get historical data on these ratios and see how that correlates with outcomes. Unfortunately, I don’t have that data (if you do, let me know!).

My bet would be that a higher ratio of operational to engineering headcount is hard for marketplaces to wean themselves off – i.e. it’s hard to change the ratio over time.  If you organization gets accustomed to scaling and solving problems by hiring ops people rather than hiring engineers to automate, that just gets amplified over time.

* my methodology here was simply to search on LinkedIn for people with “engineer” and people with “operations” in their job title. This is obviously error prone for a number of reasons.  For example, some “engineering” roles have “operations” in their job titles, and not all headcount are necessarily on LinkedIn, especially if a company outsources or off-shores some functions. However, given a sufficiently large sample set, one would hope that these effects blend out.

Brute Force Growth vs Long-term Value

Like any business, marketplaces have to continue to show top-line revenue growth in order to maintain the faith of investors and employees and be able to continue to raise money.  The first, second, and third rules of business are “don’t run out of money“.

However, while it’s possible to “brute force” growth of many marketplaces through reliance on operational headcount in the short to medium-term, I believe this strategy has large associated dangers in the longer term.

In a perfect world, you could scale both operational and product/engineering headcount as needed but, in reality, you will be forced to choose between spending each $1 on one or the other. Here’s my quick take on the pros and cons:

In summary, the biggest danger with scaling by adding operational headcount is that it works…in the short-term.  It’s also cheaper.  But, the danger is that you win the battle but not the war.

Agree or disagree, please leave a comment.

Are you confusing Optimization with Growth?

Are you confusing Optimization with Growth?

In a startup, there are always many things that aren’t working as efficiently as they could be – acquisition funnel conversion, manual processes, customer acquisition costs, etc.  This may be incredibly frustrating, especially for the team members who have to deal with it on a day-to-day basis.

It’s very tempting to direct precious money, time, and energy to resolving these frustrations, especially as its your team’s tired faces that you have to look at every day.  It’s always tempting to give the squeaky wheel some oil.

However, it’s vital that you remain focused on growth and don’t confuse growth with optimization. Burning lots of time optimizing at the expense of growing is not a recipe for success for early- to mid-stage, venture-backed startups.

Of course, there is some nuance here: if things are so broken that your team starts to leave, you have to address that – no team; no company.

Also, the smart investors (i.e. the ones you want) realize that, if your unit economics fundamentally don’t work, you will simply lose more money as you grow.

However, conversely, it’s unlikely that a Tier 1 investor will invest in the also-ran, #3 player in any category in terms of growth rate and/or absolute revenue, however optimized and healthy the acquisition funnels, gross margins, etc. Investors are in the business of selecting for the biggest return on their capital, not the best run or most efficient business.  The biggest return comes from the biggest exit and the biggest exit goes to the category winners.

As a venture-backed startup, the most important thing is to stay as one of the leaders in your category – this is what allows you to maintain team confidence and morale, attract the best talent and investors, and continue to raise money when you need it.  Note: there are usually only 1 or 2 “leaders” in any category.

Let’s take two startups:  to start with, Company A and Company B are neck-and-neck.  Both have a $5M in gross revenue, with a average revenue of $5,000 per customer per year and a customer acquisition cost (CAC) of $2,000.  Both have revenue that is doubling each year.  Both are mid-stage startups – they’re not yet profitable and don’t expect to be any time soon.

Both companies also know that their CAC is too high and, by some optimizations, the CAC can be reduced significantly.  The high CAC drives some members of the team crazy – so many opportunities lost, so many wasted marketing dollars.

So, the CEO of Company A directs the team to work on CAC.  Over 6 months, they manage to effect a series of changes process and product changes in their customer-acquisition funnel, through A/B testing, cost reduction, etc.  These compound and end up halving the CAC to $1,000 – that’s a huge improvement.  Company A’s gross margin has significantly improved.

Meanwhile, the CEO of Company B ignores the CAC for now and instead directs the team to focus on increasing the size of the sales and marketing teams significantly and filling the top of the sales funnel with as many leads as possible.

One year later, Company A’s revenue has doubled again and they’re netting an average of $4,000 per customer per year – 33% more.  Not bad.

However, by focusing on growth, one year later, Company B’s revenue has tripled rather than just doubling.  They still net an average of $3,000 per customer per year but there are 3 times more customers.

Both Company A and Company B need to raise more money.  So does a 3rd player in the category; Company C.  Company C is going gang-busters, beating both Company A and Company B on growth rate and total revenue.

You know how this story ends:  Company B and Company C are able to raise giant C-rounds from Tier 1 investors at great valuations.  Meanwhile, Company A has fallen behind – its unit economics are better than Company B’s but it’s now an also-ran and struggles to raise money.  Without that money, it cannot continue to grow and falls further and further behind Company B and Company C.  Perhaps it’s acquired by Company C at a fire-sale valuation or perhaps it’s a giant smoking crater.

Of course, this is a contrived story.  In reality, you can probably achieve growth and some optimization in parallel.  But, the key is not to confuse one with the other.

So, grow and optimize as you go, as long as that optimization doesn’t slow your growth.  Don’t optimize hoping that it will deliver meaningful growth.

tl;dr – in a startup, you can’t optimize your way to success – you must out-grow your competitors.

Are You Hiring in Your Own Image? Avoid Your Blindside

Are You Hiring in Your Own Image? Avoid Your Blindside

You’re sitting in a quiet meeting when a stranger suddenly bursts into the room, screaming and ranting – what is your immediate reaction?

Your answer might say something about your personality. If your first instinct is to act – perhaps to tackle the person, or run and hide – you’re likely a “doer”.  If your first reaction is empathy; to wonder why the person is so upset, you’re probably a “feeler”.  Lastly, if your first reaction is purely internal – to consider why the person is so mad  – you’re a “thinker”.

This simple personality model – doer, feeler, thinker – is of course just one of many. Like any model, none of us fits perfectly.  No one is purely a doer, feeler or thinker (we’d be a weird bunch if we were), but we do tend to have a primary or dominant characteristic.  Further, it’s important to be aware of the weaknesses of each of these dominant characteristics.

One of the complimentary comments that several members of the team at Wonolo have said to us over the past 4 years is that they think Yong (CEO), AJ (COO) and yours truly (CTO) are a good exec team because we are all very different.

It’s of course very nice to hear and I think the truth here is that, by compensating for each others’ weaknesses, we achieve more than the sum of our parts.

AJ (COO) is a doer; a man of action. His catchphrase might be “Just Do It”.

Last year, when he heard that the US government recommends people walk 10,000 steps per day,  he set himself a goal of doing it every single day.  At the time of writing,  he’s not missed 1 day in 326 – despite weather, holidays, travel, vacation, etc – not one!   It’s hard for me to imagine having the consistency and commitment to achieve this.

Yong is a feeler. We use Slack for internal communication and Yong’s handle is “sobstory” (handles are generally chosen by the team).

I think it’s only right and proper that we have a feeler as a CEO, given that we are in the people business.  Yong uses his “sob stories” to motivate and inspire the team and to bring empathy and humanity to our business.

That’s not to say that Yong’s not a thinker and a doer too – as well as being one of the nicest guys I’ve ever met, he’s also one of the hardest working. AJ, too, is one of the smartest people I know.  But, again, we’re talking about the the dominant aspect of their personalities.

On the flip side, one weaknesses of “doers” tends to be that their bias to action or impatience can make them act before necessary analysis of all the options is done – doers have a low tolerance for long discussions and theory.

Doers also tend to be highly competitive. AJ’s Slack handle is “bookie” owing to his tendency to bet on anything he thinks he can win.

Being competitive is of course a positive quality in many situations.   But, for doers, it can also mean that needing to win the argument is more important than making the right decision, and that achieving the goal can become more important than considering whether the goal in question is actually valuable or important.

As to feelers, their empathy can mean they tend to focus too much on how something feels rather than how it is. It means they struggle with decisions they know are right but which negatively impact people.  They can be subject to emotional manipulation by others who know they can exploit the feels.

I myself am a thinker. My Slack handle is “dirtyprofessor” (“knowitall” was another candidate).

As a thinker, one of my weaknesses is that, once I’ve worked out how to do something, I’m less interested in actually doing it. I’m more interested in the theory than the practice; the abstract over the concrete.  Learning for the sake of learning is fun for me.

Another weakness is that by being very analytical and data-driven, I can tend to get disconnected from the real-world, human impact.

AJ and Yong, respectively, definitely help counter these weaknesses.

I first met AJ and Yong in the summer of 2014 when they were working on Wonolo  inside Coca-Cola. I’d love to say that we immediately saw this complementary set of personal styles and that’s why we decided to join forces but the reality, like many things in startup life, is that we simply Got Lucky.

As I’ve aged and learned, I believe I’ve managed to compensate for the weaknesses of being a “thinker” and become a more rounded person but it’s nice to know that Yong and AJ have my back.

What this experience reminds me of is the importance of recognizing your own weaknesses and not hiring solely in your own image.  By hiring people who are different to you, you can compensate for our own weaknesses; your blindside. There are no “right” personality types and no one fits precisely in one bucket but, by having a well-rounded team, you will avoid many pitfalls.

Please leave a comment if you have one.

12 Cognitive Biases that Will Kill your Startup

If you asked me to choose the 3 most important things that determine success or failure of a startup, I would say:

  1. company culture – it has a profound and pervasive effect and it’s essential to consciously and deliberately nurture it.
  2. focus – doing many things badly rather than a few things well is surefire startup suicide.
  3. cognitive biases – the cognitive weaknesses of the human brain can if, unrecognized, wreak havoc.

I’d argue that luck is an even bigger factor than any of these 3 but, by definition, luck is outside your control.

So, let’s talk about #3 – Cognitive Biases…

Definition

“A systematic pattern of deviation from rationality in judgment and decision making.”

Despite what you might like to think, your brain is not a rational, logical computer – you’re a human being.  Your human brain has limitations.  It has bugs.

Cognitive Biases have been experimentally proven (again and again) to exist.

So What?

Cognitive Biases are potentially dangerous to all people and organizations but I think are particularly deadly to startups because:

  • Startups involve making a series of low-data, high-risk decisions. With limited data, Cognitive Biases have a stronger sway.
  • People working on startups are often stressed and working long hours, making them more susceptible.
  • The room for error is often very small because of limited funding runway.
  • Startups are commonly started and staffed by younger people who have less prior knowledge to counteract the impact of Cognitive Biases.

I believe that understanding and being aware of Cognitive Biases leads to better decision making and that better decision making, on balance, leads to better outcomes.

So, let’s look at 12 Cognitive Biases that I have seen are particularly prevalent and particularly dangerous to startups, and how you might spot them and counter them:

1. Correlation vs Causation

Did you know that sleeping with your shoes on is strongly correlated with waking up with a headache?…

…therefore, sleeping with your shoes on causes headaches!

Of course it doesn’t; this is a classic example of confusing correlation with causation.

For some more amusing examples, Tyler Vigen created a great series of charts over at tylervigen.com.  Here’s just one for illustration:

In my experience, confusion between correlation and causation is rife in startups.

Partly it’s the limited availability of data…but I’ve also seen that more data can actually make the problem worse.  With so many metrics tools available, people tend to get lost in the data and miss the bigger context.

Startups run at high speed and many things are changing in parallel, with little ability to control variables.  For example, on several occasions I’ve worked to optimize conversion in a product funnel and been happy to see that the optimization worked, only to discover later that conversion improved due to an unrelated change elsewhere – e.g. positive media coverage.

Remember to always ask yourself, does A really cause B?  Always consider that, perhaps:

  • B causes A
  • both A and B are caused by C
  • A and B effect each other
  • it’s a coincidence

2. Confirmation Bias

Confirmation Bias is the tendency to search for, interpret, favor and recall information in a way that confirms one’s pre-existing beliefs or hypotheses.

More commonly, you might call it “cherry picking”.

I think that Confirmation Bias is a particular problem for startups because they generally have a lot invested in a particular view of how the world should be (or will be) but have very little solid data to go on, especially at an early stage.

Life at a startup is ambiguous; startups struggle and go through hard-times. Therefore, belief is often what carries a team through the tough times.

Startup founders tend to be “true believers”, with a tendency to get high on their own supply.  One aspect of Confirmation Bias is that it can maintain or strengthen beliefs in the face of contrary evidence.

Always ask yourself whether you’re truly open to evidence that contradicts your existing views and beliefs.  Startups regularly pivot but often pivot too late.

3. Overconfidence & the Planning Fallacy

“the most pervasive and potentially catastrophic of all the cognitive biases to which human beings fall victim” – Svenson (1981)
sydney_opera_house_-_construction_-_phase_2_1966

Sydney Opera House:

  • Planned Completion Date: 1963, Planned Cost: $7M
  • Actual Completion Date: 1973, Actual Cost: $102M

Unfortunately, your confidence in your judgments is reliably greater than the accuracy of those judgments.  You are overconfident.

The real kicker is that this is especially true when your confidence is relatively high.

Read that again: you’re probably wrong and the more confident you are that you’re right, the more likely you are to be wrong.

One particular aspect of Overconfidence is what is referred to as the “Planning Fallacy” – most people who are involved in software development are probably familiar with it.  The Planning Fallacy is the primary reason that software development projects are almost always late.

The Planning Fallacy is the tendency for people to be overly optimistic in how much time will be needed to achieve a task.  Counterintuitively, experience doesn’t seem to eliminate the problem – i.e. knowing that similar tasks have taken longer than expected doesn’t solve the problem.

The good news is that the bias behind the Planning Fallacy can be mitigated with a couple of relatively simple “hacks”:

  1. ask someone else – overconfidence generally only occurs when people are estimating their own tasks and disappears when people are estimating for others. So, never ask the person who will be performing the work how long it will take – ask someone else.  Better still, ask a number of other people.
    In software development, a common technique is “Planning Poker” where a group of people provide blind estimates (so they don’t influence each other) of how long a task will take and the median is used.
  2. break tasks into smaller chunks – experiments have shown that the estimation for how long it will take to do a task is generally always less that the sum of the sub-tasks once they are broken out.

4. Group Think

life-of-brian

“Yes; we are all different!”

“Group Think” is probably a term that most people are familiar with.

My experience is that, in startups, it’s closely linked to the Confirmation Bias problem.  As discussed above, startups are carried forward by true believers (founders) and dissent is often considered heretical.

Perhaps the best way to understand Group Think is to look at what people do that makes it happen:

  • Minimize conflict – with a few exceptions, most people don’t want to fight.  Hey, startups are stressful enough. However, there are sometimes necessary conflicts.
  • Suppress dissenting viewpoints – startup founders are usually, almost by definition, very strongly opinionated and convincing. The flip side of this is that they tend – consciously or otherwise – to stifle viewpoints that contradict their worldview.
  • Isolate outside influences – some startups have company cultures that are cult-like. While this may be very useful in getting everyone aligned on an objective, taken too far it is dangerous, leading to “not invented here” syndrome and hubris.
  • Appeal to authority – startup teams must be encouraged to “speak truth to power” and call out the elephant in the room.

5. The Curse of Knowledge

It’s extremely hard to imagine what it’s like to not know what you know.  i.e. you can’t “unknow” something.  This is the Curse of Knowledge.

In the context of a startup, one of the biggest problems is trying to put yourself in the shoes of your user or customer.  The reality is that you can’t.  You’re so deep into the problem that you’re trying to solve that it’s impossible to see it in the same way that an outsider would.

This really underlines the importance of doing User Testing on your product using real users, not internal team members.  Watching people unfamiliar with your business and the problem you are trying to solve use your product is always extremely revealing.  There are always many implicit assumptions that you’ve made and these are only exposed through contact with real users.

6. Anchoring

Anchoring is the tendency to rely too heavily on the first piece of information you get and a tendency not to adjust your position based on further evidence that contradicts it.

Anchoring is what skillful negotiators – e.g. car salespeople – exploit to get the deal they want.  The first price discussed tends to anchor the subsequent negotiation.

In a startup, your first customer deal, your first hire, your first customer loss, etc tend to set a mental template for “how things work” with your business.  It’s important to periodically ask yourself if you’ve become anchored in a world view that isn’t necessarily correct.

7. Sunk Cost Fallacy

You might call this “flogging a dead horse” or “throwing good money after bad”.

Sunk Cost Fallacy is a tendency to continue to rationalize decisions and actions when faced with increasingly negative outcomes.  A “sunk cost” is a cost that you’ve already paid and can’t get back – money that is already spent whether you continue or not.

Sunk Cost Fallacy is, I think, one of the primary reasons that many startups pivot too late and end up running out of money.

Being rigorously data-driven is probably the best antidote to Sunk Cost Fallacy (and other Cognitive Biases).  Set specific metrics that need to be achieved by specific dates in order to assess whether a particular initiative or direction is working and hold yourself and your team to them.

8. Attribution Bias

bro

Attribution Bias is the tendency, when evaluating the causes of the behaviors of a person you dislike, to attribute their positive behaviors to the environment and their negative behaviors to the person’s inherent nature.

We all have to work with people we don’t necessarily like.  It’s important to realize that we probably can’t accurately understand others’ internal motivations and try not to take personally the behaviors that we consider negative.

9. Ostrich Effect / “The Elephant in the Room”

ostrich

The Ostrich Effect is the avoidance of risky/difficult situations by pretending they do not exist.

The “elephant in the room” is an obvious truth that is being ignored or going unaddressed.

It’s critical to build a company culture in which people are encourage to call out the elephant in the room.  If you don’t, you will be trampled by the elephant.

10. Hindsight Bias

tswift

“I knew it all along!” …actually, you didn’t.

Hindsight Bias is our tendency to see an event, after it has occurred, as having been predictable.

Hindsight Bias is a large and fascinating topic.  I can’t possibly do it justice here.

It’s unfortunately one of the hardest to counteract. The only known way is to ask whether or not alternate hypotheses and predictions would have been equally believable ahead of time.

11. Survivorship Bias

Ever had a conversation like this?  “Uber did X therefore we should be doing X!”

In reality, it’s likely that there were many other companies that did X but which don’t exist anymore…so you won’t be hearing from them.

Survivorship Bias is concentrating on the people or things that “survived” some process and overlooking those that didn’t because of their lack of visibility.

Survivorship Bias is one of my favorites simply because it is extremely common in Silicon Valley.  Beware of people claiming that companies succeeded because of specific reasons without data showing that those reasons were in fact the reasons they succeeded.

12. Bias Blindspot

Lastly, we all tend to think of our own perceptions and judgments as being rational, accurate, and free of bias.

In a sample of more than 600 residents of the United States, more than 85% believed they were less biased than the average American.

This is despite the overwhelming amount of experimental evidence that they are not.

Summary

Firstly, forget any idea that you can eliminate these biases – you can’t.  However, you can educate yourself about them, build awareness in your team and encourage people to question themselves and call them out when they see them.

Additionally, the #1 thing you can do to help counteract these biases in your startup is be Data Driven.  Data of course does not solve all problems but, by asking the right questions and getting accurate answers, you can cut through many Cognitive Biases.

Lastly, it’s important to note that these Cognitive Biases are subtle and pernicious. Companies do not usually fail because of one Cognitive Bias affecting one decision. Instead, the impact Cognitive Biases have is across a series of decisions over time.  Be mindful.

Further Reading

There are a few other Cognitive Biases that didn’t quite make the list but which I still think are hugely relevant and I’ve observed in startups:

  • illusion of validity – belief that additional information generates additional relevant data for predictions, even when it evidently does not
  • information bias – the tendency to seek information even when it cannot affect action
  • zero-risk bias – preference for reducing a small risk to zero over a greater reduction in a larger risk
  • loss aversion – the tendency to focus more on what you might lose from a particular decision than what you might gain

The best place to start for these and many others is the List of Cognitive Biases on Wikipedia.

I Crave Your Feedback

Good, bad or indifferent, please leave a comment.  Thanks.

How Startups can work with Big Companies and not get Killed

(Note: this post was originally published on the Wonolo blog under the title “Collaborate. Innovate. Top Tips for How Large Enterprises and Startups Can Have a Winning Partnership”)

Recently, I was invited by Unum, one of our FORTUNE 500 customers, to participate in a panel session about corporate innovation at Maine Startup and Create Week. At the heart of our discussion was how large companies like Unum can be more innovative and how startups and large companies can work together toward this goal.

It’s a topic that’s close to my heart: I spent the first part of my career in the wireless industry, and back in 1998, I was a part of the founding team of Symbian, one of the first operating system platforms for smartphones (although they weren’t yet called “smartphones” at that point).

Symbian was a joint-venture between Nokia, Ericsson, Motorola, Psion, Panasonic and, later, several others. Their rationale for investing in Symbian was a desire to have a common software platform for smartphones. However, what these companies actually had in common was that they were large, bureaucratic, and they were arch-competitors.

Getting these large companies in the Symbian joint-venture to work together was somewhere between very hard and impossible. (For the full story, see David Wood’s excellent book.) The term of art at the time was term “coopetition,” and it didn’t work. None of the participants in Symbian really had any desire to share their product plans with their competitors.

So, the irony was that, while Symbian was arguably at the spearhead of technology innovation, it was frequently stymied from actually being innovative by the inertia and culture of its participants. This left the market open to more focused, agile and independent companies like Google and Apple to dominate the smartphone market of today. In contrast, Nokia had an ignominious end – broken up and sold off, with billions of dollars in market value destroyed.

So, fast-forward to 2016 and my panel discussion at Maine Startup and Create Week…How can big companies be more innovative, and how can startups and large companies work together to the benefit of both?

Designer, Builder or Maintainer?

First, let’s take a look at the kinds of people that tend to work at startups versus larger companies: I have a simplistic but hopefully powerful model that divides people into three groups – “designers,” “builders” and “maintainers.”

Let’s use an analogy: here in San Francisco, arguably our best-known symbol is the Golden Gate Bridge – just look at any tourist tchotchke.

If we think about the Golden Gate Bridge, first there were the designers. In our culture, the designers generally have the “sexy” job – they are the visionaries.

GG_Bridge_Plans.png

Next come the “builders” who actually constructed the Golden Gate Bridge.

GG_Bridge_Maintainers.png

Last come the “maintainers.”  These are the workers who hang on ropes off the bridge, scraping off rust and continuously repainting it in International Orange.

International_Orange.png

This is the least sexy job in most people’s eyes: which would you rather be – the visionary designer of the Golden Gate Bridge or someone who hangs off it on a rope, scraping rust?

Now, in a startup, what you need for success are just a few designers – these are typically the founders.  You can’t have too many because they tend to butt heads.

What you really need for a startup is a boat-load of builders: these are the doers – people that create and Get Shit Done (GSD). Builders are the backbone of any startup.

What you don’t need in a startup is maintainers: everything in a startup is being created anew so there isn’t anything to maintain. You’re also focused on growth rather than optimization.

Contrast that to a big, established company: there, most people are maintainers. Their job is to ensure that an already successful business continues to be more successful. They are there to grease the wheels and optimize.

So What?

What this means is that there is a cultural mismatch between a large company and a startup.

At the core of the startup mindset is a willingness to fail and an acceptance of it. In a startup, failure is the norm – as the cliché goes, you fail your way to success. Since “failure” has negative connotations, I think it better to simply reframe it as “learning.”

Another important aspect of building a startup is understanding the art of the “good enough.” Because you’re bandwidth-constrained, you are forced to be very selective and very efficient in how you do things. You have to get them done quickly. You have to not let the great be the enemy of the good. You have to focus on delivering 80% of perfection for 20% of the effort.

Naively, when large companies aspire to become more innovative, they trot out clichés like “we reward risk-takers.” This is a lie. The last thing you want when you have a large company generating billions in revenue it to have some cowboy risk-taker come in and break it. What you want are maintainers to keep it working and keep it generating billions of dollars.

To take it back to the Golden Gate Bridge example, would you want a maintenance worker who said, “Let’s see what happens if we take all the bolts out”?

I think it would be better to rephrase it as, “We reward people who make small, smart bets.” Making a series of small, smart bets to test various hypotheses is the basis of iteration, and iteration is the how you build great products and great companies.

Recognizing these problems, many large companies have started to take a different approach – they have created specific initiatives intended to foster and drive innovation. Wonolo itself was created through The Coca-Cola Company’s innovation program.

Creating a Great Corporate Innovation Program

So, how can a large company create an innovation group and/or program likely to succeed? These initiatives can take various forms, but I think these are the most important elements:

  1. Set money aside – the budget for innovation can’t come out of the normal, operating budget for any existing business unit. If it does, it competes with the budget needed for maintenance of what’s already working.
  2. The innovation group must report directly into the CEO – this demonstrates genuine commitment to innovation and also helps unblock bureaucracy.
  3. Be clear with objectives – what specifically are you hoping that the innovation program does for your business? What are you looking to achieve? How does it positively impact your core business?
  4. Build the right team – a good mix is designers and builders from outside of the organization, along with some inside players who can help navigate the existing organization, as long as they carry enough weight. You will also need to reassure your best maintainers that they should stick to what they do well rather than trying to join the innovation program because it’s sexy.
  5. Accept failure – as discussed above, you must accept that failure is a vital part of the process. Not all initiatives you start or companies you fund will be successful, but you will learn something important from each.

How Can a Startup Engage with a Big Company and Win?

Big companies can kill startups. I’ve seen it happen.

Big companies can lead startups on and consume lots of their time and bandwidth with no pay-day. At the end of the process, the large company has perhaps lost a few hundred thousand dollars. Meanwhile, the startup has run out of funding and is dead.

Here’s what I’ve learned (the hard way) to avoid that outcome:

  1. Find your champion. Ted Reed is our champion at Unum. Not only is he an all-round great guy, but he also understands the need to be completely transparent with us. A great champion is your guide to the large company – its structure, how it makes decisions and the key players you’ll need to win over.
  2. Seek trust, honesty and transparency – any great relationship is built on mutual trust. Get feedback early and often (from your champion) on your likelihood to succeed.
  3. Don’t over-invest until you have clear commitment – be prepared to scale back or end the relationship if it’s not clear you are on a path to success. The opportunity cost of your time in a startup is huge. Don’t do anything for free – free means there’s no value, and it won’t be taken seriously.
  4. Ensure clarity in objectives, value and define success – if both sides are not clear on the business value that your product or service is providing to the large customer, be very cautious. Make sure both sides agree on what success means.
  5. Start with a small, well-defined trial – rather than trying to boil the ocean, it’s wise to start with a trial that demonstrates the value your product or service provides to the large company. This has less risk, requires less investment and has a higher likelihood of success. For more tips on how to best go about setting up a pilot, check out our related blog post.

How Can a Big Company Engage with a Startup and Win?

On the other side, how can big companies successfully engage with startups and win? Here’s my personal recipe:

  1. Be honest and transparent – don’t lead startups on. Be honest about chances of success and what it will take.
  2. Be respectful of bandwidth and provide funding, if possible – realize that a startup’s most precious commodities are bandwidth and funding. Do everything you can to reduce the sales cycle. Structure the deal to provide the funding and/or revenue necessary for the startup to succeed.
  3. Have realistic expectations in terms of maturity of a startup and its processes -don’t try to apply your pre-existing vendor onboarding process when engaging with a startup. For example, a 10-person startup won’t pass your 50-page IT security audit.
  4. Respect the need for independence – you may be providing a startup with revenue, funding and a great customer reference. However, a startup needs to be in control of its own destiny and own its own product roadmap. Don’t treat a startup like a consulting company or development shop, unless that’s how the startup sees themselves.

Overall, the relationship between a large company and a startup can be a marriage made in heaven. I would marry Ted Reed if I could.

Meaningful Metrics: 6 Steps to Metrics Heaven

Surely, one of the benefits of technology-based businesses is that it’s much easier to make data-based decisions – to “drive by the numbers”.

Superficially, this seems like it should be easy – just look at the data and decide what to do, right?  However, in every company I’ve seen, it’s always been painful.

Hopefully, this will help you avoid some of that pain.

Six Steps to Metrics Heaven

Six Steps to Metrics Heaven

I believe there are 6 steps to “metrics heaven”:

  1. Define
  2. Measure
  3. Present
  4. Discuss
  5. Action
  6. Iterate

Sorry, there are no short cuts; you have to get all 6 right to be successful.

Step 1 – Define

Ask the Right Questions

Your path to Metrics Heaven starts simply by asking the right questions.

Firstly, try to state the question behind each metric as unambiguously as possible.  Secondly, explain in plain English what this question means and why it’s important to your business – the context.

Example:
Question:  “Of the users that have signed up in the past 12 months, what percentage have logged in in the past 1 month?”

Context:  “This metric measures our ability to keep users coming back to our site.  This is vital to our core business model as we pay $40 on average to acquire each new user and can only make a profit on them if they keep coming back.”

Clear definition of the question is particularly important because the person responsible for actually answering the question – writing the database query, wrangling Google Analytics, etc –  is often not the person framing the question.  So, any ambiguity has the potential to result in an answer that is misleading or just plain wrong.

Fight Data Overload

If you try to measure, track, discuss and action too many metrics, you’ll lose track of what’s important.  You’ll lose sight of the forest for the trees.

Data overload is an insidious problem.  The addition of each new metric seems innocuous so it’s hard to say no.  Plus, there’s a tendency for people to want to get their group’s name “up in lights”.  But it’s vital that you do say no.  Otherwise, here’s what I’ve seen happen every time:

  • pulling and calculating all the metrics becomes a chore and takes forever
  • you end up with a big and unwieldy dashboard
  • the dashboard takes more than 30 minutes just to run through with everyone during the metrics meeting
  • people fall asleep or get distracted by their laptops/iPads/iPhones
  • you run out of time to discuss actions
  • people start to view the metrics meeting as a pain and find excuses to avoid it

Top 10 Metrics

To avoid overload, you need to identify a maximum of 10 metrics.  This “Top 10” constitutes your top-level dashboard.

By “top-level dashboard”, I mean the dashboard that is shared with the whole company, which the executive team uses to drive the business and the first – and maybe only – set of metrics you review when you meet.

Each group or department may have its own dashboard too.  There may be multiple levels of dashboards that dive deeper and deeper into the internals but don’t let those pollute your Top 10.

Your Top 10 metrics dashboard will probably be something you’ll want to work to automate so that it’s available live, displayed on a big flat-screen in your office, etc (more on that later).

To help ensure that you don’t grow beyond 10 metrics, use his rule:  if someone wants to add a metric to the top-level dashboard, they have to choose one to remove also and justify why the one they want to add is more important than the one they want to remove.

Now, let’s look at how to identify which metrics should be in your Top 10…

MAIN Metrics

I have come up with a handy-dandy acronym to help identify the metrics that you should choose, particularly your Top 10 metrics; “MAIN”:

  • Meaningful
  • Actionable
  • Isolatable
  • Not misleading

Let’s look at these 4 in more detail…

Meaningful

Is the metric a true and valid indicator of success in your core business?

A good way to ask this is to pose the following question:

“if this metric changes in the right direction, and nothing else changes, will my business be doing better?”

If the answer to this question is no, then you may have identified a useful metric to measure and try to impact, but it’s unlikely that you’ve identified a Top 10 metric.

Actionable

There’s not really much use measuring something if you can’t do anything about it.  You must be able to “move the needle”.

Metrics where you can’t move the needle are potentially interesting in terms of giving you a better understanding of what drives your business but almost certainly are not Top 10 metrics.

Isolatable

It’s vital that you choose and define metrics in such a way that they do not vary based on other, independent variables, whether those variables are in your control or not.

You need to be confident in measuring the effect of the actions you take to move the needle.  You need to avoid the age-old problem of confusing correlation with causation.

A classic example in web businesses is not insulating your metrics from the impact of fluctuations in traffic caused by other factors.  For example, you may have been measuring the number of user adds on a weekly basis and be working aggressively to drive up that number by reducing friction in your sign-up funnel.  The week after you launch a slew of sign-up funnel optimizations, you get 25% more users.  Yay for you – it worked – gold star.

Not so fast.  It turns out that what actually happened was that an article in a local style magazine that your PR agency setup some weeks back was finally published which drove 25% more traffic to your homepage.  Your conversion rate actually stayed the same.

To isolate a metric as far as possible, first frame the question in such a way that it’s not impacted by other variables.  The easiest approach to this is to always define metrics in terms of rates, not in terms of absolute numbers – e.g. “what % of users that started down the signup funnel completed it last week?” Not, “how many users signed up last week?”

Lastly, always be suspicious if it seems too good to be true.  Dig deeper to check whether your action really was the cause of the change.

Not Misleading

Lastly, even if a metric is isolatable, it may also be misleading in other ways.  It may give a false negative or false positive, or otherwise cause you to react in the wrong way.

It’s important to define metrics in such a way that they are hard to misinterpret.  Even if that’s not completely possible, it’s important to highlight the ways the metric could be misinterpreted when it is discussed.

Another classic example from the web is measuring the cost effectiveness of various CPC (cost-per-click) advertising campaigns.  You may find that certain keywords or certain channels allow you to buy traffic at a lower cost.  So, you decide to spend more money on those channels and reduce your spend on other channels.  However, not all traffic is created equally.  You may find that the users/customers brought in by the cheaper CPC rates actually generate proportionally less revenue for you so it’s worth paying more for higher quality traffic.

The danger of misleading metrics is one of the main reasons why stating the context of each metric (as discussed above) is so critical.

Step 2 – Measure

So, you’ve defined your Top-10 MAIN metrics in a precise and unambiguous way.  Now you just need to measure them.

Easy, right?

Um.  Mostly not.  Measuring is usually hard for a variety of reasons:

  • the relevant data is spread across multiple systems (your database, Google Analytics, MixPanel, AddThis, etc)
  • data is buried in the depths of your system and extracting it requires in-depth knowledge of your systems (and probably wicked SQL skills too)
  • the people with the know-how to extract the data are probably engineers who are busy with other things and consider extracting metrics beneath them
  • metrics were not really considered when the original system was built so the “instrumentation” has to be added retrospectively; this work contending with other product feature work
  • lots of manual steps are required, making it a chore

Tools

The good news is that there is a growing range of tools available to help.  The bad news is that any tool, if badly used, can be worse than no tool at all.

Let’s look at a few…

Google Analytics (aka “GA”)

I hate Google Analytics.  However, I can’t argue with the price.  And it’s incredibly powerful…if you can work out how to use it.  It’s unfortunately incredibly unintuitive.  It’s a camel (horse designed by committee).

The good news is that it’s fairly ubiquitous so you probably already have people on your team who have experience with it.

Mixpanel

I like Mixpanel.  I have no relationship with them other than as a happy customer.  Mixpanel does the 10% subset of Google Analytics that you actually need and does it well – the charts make sense, things are consistently referred to by terms that make sense.  It’s not free but it’s not expensive compared to the value it provides, in my opinion.

KissMetrics

Personally, I had a bad experience with KissMetrics but that’s just one data point.  They are probably Mixpanel’s main competitor.

SQL Queries

The data in your app is probably in a relational database.  That means you need at least one SQL ninja to write queries on that database.  SQL is a technology that is older than I am and therefore is definitely not cool but it does what it does better than anything else, if you know how to use it.

Unfortunately, modern web frameworks mostly insulate developers from having to use SQL directly.  Whilst this may be a good thing from the perspective of developer productivity and “separation of concerns”, it means that the number of engineers that can write complex SQL queries is reducing.

Microsoft Excel

There ain’t no shame in Excel, in my opinion.  Likewise, Google Sheets, although the graphs are much better in Excel.  Both have plugins to pull in data from various sources, including Google Analytics.

Hire Someone

You might want to consider hiring a “data analyst” – i.e. someone who runs the metrics process as their main job, rather than it being done badly as a part-time job by others.  I use parentheses around the job title because I’ve found that people who self-identify as Data Analysts have a tendency to be quite academic and unable to roll their sleeves up and actually grapple with Google Analytics to extract the numbers.

I would suggest you’re better off starting with a technically-minded digital marketing person.  They’re like unicorn poop in the current job market so good luck.

Step 3 – Present

Once you’ve got the actual data, you now have to present it in a clear way that can be digested by others.

Build a Dashboard

The first step in presenting metrics is to build a dashboard that quickly and clearly communicates your “Top 10” metrics.  Don’t worry about finessing the dashboard for metrics outside your Top 10;  it’s better to get the Top 10 dashboard as good as possible before you spend time elsewhere.

Excel is the most common tool used for this purpose and, to repeat myself, I believe there is no shame in Excel.  However, as discussed above, there are an increasing range of more automated options.

Use Charts

Some people are able to “see the matrix” – they can see patterns and trends in raw numbers.  But, most people can’t, including a lot of very technical people.

So, show charts in your dashboard, not numbers – they really help highlight trends over time, progress versus budget and other patterns.

“One Number” Nirvana

I already talked about limiting yourself to a maximum of 10 top-level metrics.  I’m going to go a stage further and say that every business has just ONE number that represents how successfully the business is performing.  This may seem impossible at the outset but but I bet you it’s not.

So, set yourself the stretch goal of identifying that One Number.  You’re unlikely to get to it immediately so start with your Top 10 and see if you can iterate towards your One Number.  It may be one of your Top 10 or it may be a calculation based on a number of metrics.

Simplistically, you might say that revenue or profit is the One Number for any business but absolute revenue or profit is driven as much by scale as anything else.  Before you can scale to maximize revenue or profit, you need to have a business model and product that is worth scaling and you’ll also likely need to persuade investors to give you the cash required to scale.  That’s where your One Number comes in.  So, your One Number is likely to be something like customer acquisition cost (CAC/CPGA) or profit per user add.

Build a Live Dashboard

Data junkies will argue that the ultimate nirvana is to have a live dashboard that shows your One Number and Top-10 metrics, with graphs, on a web page that anyone in the company can view at any time to see a real-time view of how the product and business is performing.

But, however easy it is to get view the metrics, that does not remove the need to meet to discuss them and to action them.  So, let’s talk about that.

Step 4 – Discuss

Having your metrics beautifully presented is no use if you don’t actually talk about them as a team.

Meeting to discuss the metrics regularly keeps them in the front of everyone’s minds, makes sure that everyone understands what actually drives your business and, most importantly, is the venue to come up with your action plan for pushing them in the right direction.

Meet Weekly

The only cadence that I have seen work when it comes to reviewing the metrics in an online business is weekly.  A month is an aeon in Internet time.  Daily is good for quota driven sales team but too much for a startup team that is trying to build and rapidly iterate a product.

So, set a fixed time to meet every week.

The most important people to have at the meeting to review metrics is the people in the company that can actually impact those metrics.  Others can attend but you don’t want too many cooks.

Manage the Time

It’s very easy for the weekly metrics meeting to turn into a mechanical run through of all the metrics.  This is especially true if you’ve let your dashboard grow beyond 10 metrics or if you spend time diving into lots of very detailed metrics below your Top 10.

Most important is that you spend at least as much time on actions as you do on reviewing the numbers.  People who attend the weekly metrics meeting should be expected to have reviewed the metrics ahead of the meeting and come to the meeting armed with questions and suggestions for actions.

Step 5 – Action

Having perfectly accurate and beautifully-presented metrics that you discuss weekly as a team is useless if you don’t actually do anything about them.

What’s surprising is how often this happens.  People come to the weekly data meeting, review the metrics, ask questions and then go back to their jobs.

If this is what’s happening, you’re wasting time preparing the metrics and might as well get a job rearranging deckchairs on a sinking cruise ship.

Set Targets

It’s way easier to move a metric in the right direction if you specify a target of where you’re trying to get to.

Frankly, any target is better than none but it’s of course better to have a target that is achievable.  It’s better to start with a small, achievable increment and then move the target as you get better and more sophisticated in impacting your metrics through action.

To learn more about this, read a good dieting book.  Trying to either “stop being fat” or “lose 40lbs” is much, much harder and more demotivating than setting an achievable target for each week.

Actions for every metric

Every metric in your Top 10 should have a corresponding list of actions aimed at moving the needle in the right direction, along with an owner for each metric.  These can be marketing efforts, new features, A-B tests, whatever.

If you repeatedly don’t come up with actions for a Top 10 metric, ask why.  It probably means you should remove that metric from your Top 10 and replace it with something you feel is more important and which you can actually impact.

Likewise, if agreed actions don’t get performed by their owners, you can keep people honest by asking whether people still think the metric is important in terms of making the business successful.  If it is, then ask what other tasks they are doing that are more important.  If it’s not, remove the metric.

Step 6 – Iterate

I would say that the chance that you’ll choose the right set of metrics, be able to extract all the data you need, present them clearly and  build an action plan to move the needle first time around is approaching zero.

Expect it to be painful initially.  Expect it to take time.  Expect that you’ll need to iterate.

Summary

If there are just 4 things I suggest you take away from this, they would be:

  • Ask the right questions
  • Fight data overload
  • Focus on action
  • Iterate and expect it to take time to get right

Agree/disagree?  Please leave a comment (link is top-right of the page).