(2021-03-21) Chin Product Development As Iterated Taste

Cedric Chin: Product Development as Iterated Taste. Working Backwards is the first book to describe how Amazon really works, written by two insiders who were in the room when the techniques were first invented. I finished the book a few weeks ago, and I’m still chewing on many of the ideas

One way you can think about the Working Backwards process is that it is a response to the fundamental difficulty of launching a new product.... With stakes that high, you don’t really want to fail. So what do you do?

For many years the answer was to ‘write a business plan’ or ‘do market analysis’

But in the early 2010s, a new orthodoxy emerged with Eric Ries’s The Lean Startup. “Go build a ‘minimum viable product’”

Talk to a startup person today and you'll likely hear things like ‘MVP’ or ‘pivot’ or ‘product/market fit’; these ideas have spread so far that they’ve become conventional wisdom. But this, in turn, implies something interesting: whenever you come across a successful company doing things differently from the standard ‘Lean Startup’ method, it is always worthwhile to pause, and pay attention, and ask: “wait, why does this work?”

Amazon’s ‘working backwards process’ is a pretty fascinating example of an alternative that works — I should know; I’ve been collecting alternatives to the Lean Startup for a number of years now.

Amazon is, after all, in more industries and product categories than pretty much any other Western tech company. It's a bit ridiculous that it is able to enter, invent and dominate so many niches repeatedly; on this basis, I think the Working Backwards process belongs on the pantheon of other successful product development methods, alongside The Lean Startup, Apple’s Creative Selection Process, and Pixar’s Braintrust.

Amazon’s PR/FAQ Method

The total document — both PR and FAQ — should not exceed six pages in length.

Once submitted, your PR/FAQ is reviewed in a one-hour meeting with the relevant stakeholders present. The first 20 minutes is dedicated to reading the document in silence.

When everyone has finished, the writer asks for general feedback. The most senior attendees tend to speak the last, to avoid influencing the others.

The authors describe a process early on in the PR/FAQ cycle for Amazon S3:

that day we never got to the next question. (emphasis added). We kept discussing this question. We really did not know how developers would use S3 when it launched. Would they store mostly large objects with low retrieval rates? Small objects with high retrieval rates?

All those factors were unknown yet could meaningfully impact our costs. Since we didn’t know how developers would use S3, was there a way to structure our pricing so that no matter how it was used, we could ensure that it would be affordable to our customers and to Amazon?

the discussion moved away from a tiered subscription pricing strategy and toward a cost-following strategy. “Cost following” means that your pricing model is driven primarily by your costs, which are then passed on to your customer

But of course you cannot, you need to go think about the right costs to follow.

You eventually figure out that storage and bandwidth are the most expensive cost factors; in the end, everyone decides on storage and bandwidth as the right costs to follow.

This turns out to be mostly right. After S3 launched, Amazon discovered that they had missed out on one other cost factor... after running for a while, we realised that the number of requests was an equally important resource. If customers have many tiny files, then storage and bandwidth don’t amount to much even if they are making millions of requests.

And so the end result of a PR/FAQ meeting is often a decision to continue working on the … PR/FAQ. This is not a typo: most PR/FAQs are rejected, and the ones that pass often generate further questions that need to be answered

In the end, it took 18 months of PR/FAQ iterations before Amazon’s software engineers wrote the first line of code for AWS. (I relayed this anecdote to a programmer friend of mine, and we both agreed that we would much rather quit than spend 18 months iterating on a Word document).

So why does this process work?

preserves your company’s resources to build products that will yield the highest impact for customers and your business

Another one of the biggest benefits of a written PR/FAQ is that it enables the team to truly understand the specific constraints and problems that would prevent a new product idea from being viable and aligning on them.

Does this process work 100% of the time? No, it doesn’t. The authors argue that the PR/FAQ process merely increases the odds that your product would succeed; it doesn’t guarantee it.

the Amazon Fire phone, for instance, was a flaming wreckage of a product, and it went through the PR/FAQ process all the same

Bryar and Carr say that by forcing multiple iterations on a PR/FAQ, you implicitly gain the approval of everyone who reviews it, demands iterations on it, and subsequently approves it. This then means that when a product passes the PR/FAQ process but subsequently fails in the market, people are more likely to take collective responsibility.

I think it is highly dependent on cultural values that exist in Amazon

If you take a step back and squint, however, you can see certain commonalities between the Working Backwards process and others like it.

For instance, Apple’s Creative Selection process is driven by a demo-first culture — you are invited to sit in demos only if you have demonstrated good taste in product development in the past.

• hmm is "iterated taste" just a "credible track record" of outcomes?
• cf core group

Or take Pixar’s Braintrust structure, which workshops story ideas as ‘reels’, showing them to a core group of storytellers for debugging before committing resources to production... making reels... Then the Braintrust watches this version of the movie and discusses what’s not ringing true, what could be better, what’s not working at all. Notably, they do not prescribe how to fix the problems they diagnose. They test weak points, they make suggestions, but it is up to the director to settle on a path forward. A new version of the movie is generated every three to six months, and the process repeats itself.

I think one way you can think about this is that for a product to succeed, lots and lots of things have to go right. And if even one critical thing goes wrong — be it software design, or some hardware flaw, or market size, or manufacturing costs — the odds are good that the entire product flops.

all of the product development processes that I’ve described — Lean Startup, Working Backwards, Creative Selection, and Braintrust — are simply ways of iterating cheaply through an idea space, with sufficient feedback, in the hopes of checking enough boxes for success.

If there’s anything we’ve learnt from the decade after Lean Startup, it is that sometimes, even building an MVP can be too expensive. (or impossible)