book Signals & Levers: Systems Thinking Tools to Unblock Software Delivery, (Systems Thinking, Thinking Tools) by Elisabeth Hendrickson and Joel Tosi, ISBN:1966280297

Partial Excerpts (antilibrary)

2026

Table of Contents

Foreword by Gene Kim

Across the board, people’s worst projects had the most testers

when you know that no one’s going to find your problems for you, you often end up with spectacular quality, because developers know they are fully and solely accountable and responsible

Over the years, I’ve encountered many pieces of this puzzle presented individually: the MIT Beer Game, systems thinking, fast feedback loops, Conway’s Law, Lean principles, statistical process control, DevOps, and so forth

as a software leader, what are the concrete tools I can use to correctly diagnose and steer a complex system toward my goals—something that is notoriously difficult to do.

What we’ve lacked is a coherent framework for seeing them clearly and doing something about them. This book answers that question

Preface

Here in 2026, as we write this book, AI is reshaping how organizations build and deliver software

We believe the organizations that already excel at delivering software will continue to do so because they’re in the best position to take advantage of AI without succumbing to the risks inherent in shipping generated code

that no one really understands

we believe the real value of AI is likely to be less about generating raw code and more about supporting people in reasoning about complex systems.

Every new age brings with it a new set of challenges specific to the technology. However, most of the problems that plague organizations today are not new

Flashback: 1968

the conference that coined the term “software crisis” and popularized the term “software engineering.”

If you could go back in time and eavesdrop on the discussions, you would find the topics hauntingly familiar: Software development took too long and cost too much

Dijkstra neatly summed up why these challenges will always be with us in his 1972 Turing Award Lecture:

…as the power of available machines grew by a factor of more than a thousand, society’s ambition to apply these machines grew in proportion, and it was the poor programmer who found his job in this exploded field of tension between ends and means

Every age brings new approaches—process improvements, tools, and supporting technologies—intended to help the poor programmer keep up with that growing tension between ends and means

Every one of these advances was truly a revelation, unlocking new capabilities, making things possible that hadn’t been possible before

It seems we will always be running faster than our legs can carry us

Let’s return to the phrase “software crisis.” If the problems with software delivery are endemic, is it really a crisis? Consultant, author, and teacher Jerry Weinberg wrote in his book Secrets of Consulting: A Guide to Giving and Getting Advice Successfully, “It may look like a crisis, but it’s only the end of an illusion

The biggest illusion is that it is possible to make software development cheap, 100% predictable, and control able

Peer deeper into the crisis and you’ll spot the illusions: estimates mistaken for commitments; promises made when they shouldn’t have been; insufficient practices dressed up in the language of bureaucracy; security theater

Worse, organizations tend to have processes, policies, and incentive structures that act to preserve illusions

The longer an illusion

holds and the bigger it is, the worse the crisis appears when the illusion is finally dispelled.

Ultimately, this is a book about piercing the veil of illusion sooner, providing you with signals to identify the illusions and levers to address and correct the challenges

We offer a set of systems thinking tools drawn from a range of disciplines

You’ll learn how to use variability analysis from statistical process control so you can separate signal from noise and not over- (or under-) react

Causal modeling, a technique from systems thinking, will aid you in uncovering hidden connections and show you how to discover levers that you might not have known existed

And you’ll learn how to use U-curves from economic theory to explore a range of possible ways of using those levers, allowing you to optimize for the variables that matter most

we give you heuristics for choosing between levers and recognizing the signals that your intuition is leading you astray.

A warning here seems appropriate: Once you learn to see, you won’t be able to unsee.

Worse, the tools in this book enable you to show others what you see, to telll your story, enabling everyone to see the same reality. If you can’t unsee it, they can’t either. They may not thank you for destroying their comforting illusions.

The upside is that when you deal in what’s real, you can make reality better. Eventually, illusions lose their appeal.

Introduction

When something goes wrong, it’s so tempting to look for a root cause, as if there’s a single originating source of the problem. There rarely is. Reality is far more complicated, and the real challenge isn’t finding the culprit—it’s seeing the system.

we start by looking at the nonlinear nature of software development and why attempts to control outcomes directly don’t work

To tackle these ideas, we’ve put together a fictionalized account that represents a composite of our own and others’ experiences

Each chapter begins with a scene from their story that il ustrates the concepts we’ll explore, then steps back to examine the systems thinking concepts that explain what’s happening and what to do about it, and wraps up with an exercise that helps you apply these ideas to your own challenges

Let’s get into the story.

The Day Begins

The one-on-one meeting with his boss, Gus, the CTO, less so. “What are they dragging you into those meetings for?” Gus would ask. “Your job as VP of Product is to sit on Engineering and get them to execute on the roadmap.” Rohan felt his soul wither a little at the idea that his primary job was to squeeze more features out of Engineering. That wasn’t what he signed up for.

But he couldn’t convince Gus, and without Gus, Preston, the CEO, would never take that kind of gamble.

Engineering was behind, and Rohan was frustrated—Preston was frustrated too. But he was more frustrated that he couldn’t count on Gus as a strategic partner. Gus didn’t listen to new ideas and micromanaged his team, and instead of looking ahead to find ways technology could put the company into a stronger strategic position, he was busy telling his leaders how to do their jobs

she pulled up her metrics dashboards. After the last delayed release, she had challenged the engineering team managers to improve efficiency—and every team showed substantial improvement in their velocity. Despite all the pressure, seeing that pay off put her in a good mood

It’s a System

three smart, experienced leaders who are trapped in patterns they can’t see. Rohan blames Engineering’s lack of urgency. Preston blames Gus’s micromanaging and obsession with technical details. Theresa looks at her velocity metrics and sees progress. They’re all wrong, or more precisely, they’re all partially right while completely missing the bigger picture

Why are organizations still struggling? Because they’re still trying to control outcomes by applying solutions that tweak parts of the system instead of understanding and influencing the system as a whole.

Before we continue, we must establish a few foundational concepts that we’ll use throughout this book

First, software delivery is a system. Borrowing heavily from the work of Russel Ackoff and Donel a Meadows, we define a system as a set of interrelated parts that does something.

Second, software delivery is also inherently a social activity that is achieved through creating and using technical tools

Finally, systems are adaptive

Consider how organizations dealt with the COVID-19 pandemic. Even companies with no history of remote work suddenly found themselves operating with an entirely remote workforce

The shift to remote work that initially seemed to be cataclysmic ended up being empowering. Changes don’t need to be so dramatic to shift the system, though. Adding one new remote employee to a team that is otherwise colocated shifts the communication dynamic. Small changes, ripple effects, and the system adapts.

The idea that software delivery is a system implies that it has three key properties

1. The whole is indivisible. The system’s behavior emerges from the relationship between its parts, not from the parts themselves.

2. Parts are interdependent. Change one part of a system, and it affects other parts, often in unpredictable ways

Interdependence also means that locally optimizing one part does not guarantee the overall system benefits

3. Systems are fractal. Each part of a system can be a system itself.

In our story, Rohan blames the engineers for not having a sense of urgency. He’s not thinking of software delivery as a system. He’s acting as

though speed is a dial

But it doesn’t work like that. Theresa observed that Rohan’s pressure in the past had led to shortcuts that were still biting them

In any organization struggling with capacity, risk, execution, adaptability, trust, or economics, you’ll hear the same frustrations—just aimed in different directions. Everyone’s pointing at someone else. And if you take a truly unbiased look, you’ll find they’re all at least a little bit right

Every attempted fix has a catch

Any serious problem is the result of multiple interrelated issues

There’s no single culprit, no root cause, and no easy fix.

Further, no one person can hold the entire system in their head. In the paper “Moving off the Map: How Knowledge of Organizational Operations Empowers and Alienates,” Ruthanne Huising tel s the story of an organization that did an in-depth process mapping exercise across the entire company

Everyone was working on the part of the organization that they were familiar with, assuming that another set of people were attending to the larger picture

No one is controlling the system. The best you can do is learn to read the signals and find the associated levers so you can start shifting the system in a desired direction

Signals

A signal is an observable characteristic of the system. It tel s you something about the way the system works. Some signals are quantitative, like cycle time, lead time,† velocity, and a count of work items in progress (WIP). Others are more qualitative, like morale. Signals are everywhere.*

Although you are surrounded by signals, it can be difficult to determine what the signals are trying to say

The challenge with reading signals is that they are so very easy to misread (like confusing “buggy” for “less valued”). In this case, the problem was that the decision makers were using defect rates as a proxy for value.

Levers

A lever is a choice that causes a shift in the system.

There is an important implication here: If you make a choice—change something—and the system does not change at al , whatever choice you made was not a lever within the system.

The conversations were just that: conversations. All talk, no outcomes. They were not the lever she thought they were.

Broadly speaking, the levers at your disposal fall into two categories: policy and culture.

Policy levers are interventions that require formal authority. Examples include things like enacting and enforcing rules, changing job descriptions, assigning staff, rolling out new processes, bringing in external consultants, reorganizing teams, defining the branch-ing strategy, or allocating budget

By contrast, culture cannot be changed through mandates. Culture is inherently social and contagious. It spreads through contact, and everyone affects it simply by showing up. Culture levers include any action that

amplifies or encourages behavior, like recognition, gratitude, and peer pressure

Policy and culture are intertwined

If policy and culture are at odds, culture wins. That’s why organizational transformations are so difficult. Culture acts as a sort of organizational immune system.

while you need organizational authority to take policy actions, you only need to exercise influence to shift culture.

A Systems Thinking Toolkit

The key to understanding the behavior of an adaptive system lies in examining the relationships between its parts.

Rohan, Preston, and Theresa are trapped in patterns they can’t see— yet. As the story progresses, we’ll see that they’re measuring the wrong things, pulling the wrong levers, and misreading the signals all around them.

In Part One, you’ll learn about common illusions that prevent you from seeing what’s really happening in the system, and how to dispel them.

We introduce the CREATE framework to help you identify many common signals based on categories of effects like capacity, risk, execution, adaptability, trust, and economics

Part Two shows you how to use causal diagrams to see the interrelated nature of the system and begin to influence it

Part Three gives you a set of tools to make the most of those decision points

Finally, Part Four is about how to make the system resilient so it requires fewer interventions. A key factor in building resilience is becoming a learning organization

Summary

Software development organizations are adaptive sociotechnical systems. They are inherently nonlinear, so linear reasoning techniques that rely on seeing outcomes as a straightforward chain of cause and effect don’t work.

IDENTIFYING NONLINEARITY

Throughout this book, we will offer you exercises you can complete in about half an hour. These are intended to help you apply the concepts and techniques as they are introduced in your context and with your experiences

As these exercises build upon each other, we recommend you do them in order. On to your first exercise.

Part One: Reading Signals

You are surrounded by signals. Some of those signals are already central to your software delivery process. For example, your organization surely has a way to see the status of a given release or initiative. Maybe there’s a dashboard of relevant metrics.

Other signals are subtle and elusive. They don’t show up on dashboards or status reports, at least not directly

Given this abundance of data, you might wonder: Why can’t organizations just track everything and make better decisions? The answer is that not all signals deserve equal weight, and some can lead you directly to the wrong conclusions. In Accelerate: Building and Scaling High-Performing Technology Organizations, Dr. Nicole Forsgren, Jez Humble, and Gene Kim point out that many common metrics, like lines of code, velocity, and utilization, are overwhelmingly focused on outputs rather than outcomes

The problem with proxy measures is they can create and perpetuate illusions.

Chapter 1 Illusions

It can be so tempting to embrace illusions rather than face reality.

“The customer needs this change, so we’ll push it out; it isn’t that risky.” It’s one thing to be optimistic. It’s quite another to cling to illusions instead of seeking empirical evidence.

Worse, the bigger an illusion and the longer it remains, the harder the fall to reality. Or, as consultant, author, and teacher Jerry Weinberg wrote in his book Secrets of Consulting, “It may look like a crisis, but it’s only the end of an illusion.

We see three recurring patterns of illusion: the illusion of predictability, the illusion of progress, and the illusion of control

1. The illusion of predictability.

despite all the attempts to improve estimates, it remains impossible to predict the future with any precision

2. The illusion of progress.

Progress is only accurately measured in terms of outcomes, but so many of the mechanisms for gauging progress, like story points completed, are proxies.

3. The illusion of control.

*so many approaches to managing projects are wrapped up in * an attempt to control outcomes. Iron- ically, those attempts to control things can actually make things even less pre- dictable.

Let’s rejoin Rohan and Theresa to see how these illusions play out in practice.

Delayed. Again.

*“What happened?” Rohan asked. “You said you were confident we were ready to ship.” Theresa wanted to go home and wallow in self-pity with a cup of tea and a mind-numbing reality show. Instead, she faced Rohan. * “I’m sorry,” she said. “I truly thought we would make it until just now. The remaining tickets are so smal . Going into this meeting, I thought Mark would be willing to make an exception since we’re so close to done.”

Do you even care?” “I care,” Theresa protested. “We’re all working our butts off. Look at the velocity numbers—we’re moving faster than we ever have. But there’s also more work to do, and we’re still paying down technical debt.” “Moving faster?” Rohan snorted. “Last time I looked at the tracker, Engineering had only completed two-thirds of the features

Each unfinished feature was stuck waiting on something. One was theoretically done but couldn’t be tested until they got a test environment refresh. Another spanned three teams: The backend was nearly complete, the app work started, but the App Team was waiting on the Integration Team to add a new authorization level, and they were waiting on the Core Team, who wouldn’t get to it until they finished work blocked by something else. It was an intertwined chain of team interdependencies.

The Core Team’s numbers looked great—so how were they so far behind? A sinking feeling grew in her belly. Had she been looking at the wrong things?

Team velocity showed how many story points each team was completing, but stories weren’t shippable features. A different dashboard showed features completed by week. Theresa opened it and saw a completely different picture. Where team velocity had been climbing steadily, the number of features completed per week stut-tered and lurched.

Theresa thought about the chain of interdependencies again. Features involved multiple teams. She had pushed the Engineering managers to increase team velocity, assuming that if teams moved faster they could ship releases sooner. But velocity wasn’t the real issue. She ’d been measuring the wrong thing.

“How exactly did you allow work to get ‘stuck’?” Gus growled, using air quotes around the word.

Do you even know what your teams are doing?”

*Sounds to me like you have no idea what’s going on.” Theresa steeled herself. “I have seventy-three people spread across five teams. I don’t track every single thing they’re working on, but I do watch the metrics dashboard.” I trust the managers to take care of the details, she thought, like I wish you would trust me. * Gus pointed at her as he spoke. “Watching metrics isn’t enough. You can’t afford to be so disconnected from the actual work.”

The Problem of Illusions

Theresa’s release delay wasn’t caused by lazy engineers or inept execution. It was the inevitable result of three interlocking illusions working together to hide the real problems

These three illusions commonly reinforce each other. Let’s examine each one in depth.

The Illusion of Predictability

It’s completely reasonable for stakeholders to want to know when to expect a given feature to land

There could be half a dozen groups or more outside

of the software delivery teams that need to plan ahead for a feature release

But predicting the exact date when something will happen is incredibly difficult, especially the further out you plan. The illusion of predictability takes root when estimates are treated as immovable commitments.

Estimates Change

Why is it so hard to predict software projects? Michael Wolfe, serial entre-preneur and startup adviser, answered that question on Quora using an imaginary trip as a metaphor. His answer begins: Let’s take a hike on the coast from San Francisco to Los Angeles to visit our friends in Newport Beach

Four hundred miles, forty miles per day, ten days. What could go wrong? What follows is a rollicking diary-style recounting of a pair of friends on the worst adventure ever, encountering difficult terrain, angry sea lions, long detours, sleep deprivation, sore muscles, blisters, and il - ness

First, our fictional trip planner wildly overestimated the pace they could set.

Second, they made incredibly naive assumptions about the length of the journey based solely on a simplified map. How long is the coast

exactly? It’s incalculable. Mandelbrot, father of frac- tals, said as much in his clas- sic 1967 essay “How Long Is the Coast of Britain?”

*There isn’t even a trail that covers the entire coastal * route between San Francisco and Newport Beach

Finally, the protagonist in the essay had no idea how many complications they would encounter along the way. Their original estimate didn’t consider illness, detours, or rough terrain.

Software projects suffer from the same three challenges

Go and See

The solution to the problem of estimation is not to throw your hands in the air and say, “You’ll get it when you get it.” Instead, bring stakeholders along on the journey. The key is to not just share status but also context.

What have you learned, what has been surprising, and what new decisions have you had to make? Dr. Erik Hollnagel, adjunct Professor of Human-Machine Interaction at the University of Linköping (Sweden), popularized the concept of “work as imagined versus work as done.”4 When stakeholders can’t see how work is done, they use their imagination. Too often they imagine conceptually simple things are easy.

Show them what it really takes to do the work—not just pretty status charts—and you’ll build trust and empathy.

Variability plays a significant role in why estimates fail, something we’l explore in more detail in Chapter 3. For now, it is enough to recognize that creating visibility helps everyone understand not only the current state but the natural fluctuations that make precise prediction impossible.

The Illusion of Progress

Unfortunately, it’s all too common to measure progress in terms of effort rather than outcomes: number of code commits

Sometimes the target moves: The market shifts or the underlying tech stack is highly volatile. Other times the target remains the same, but we fail to understand how very far we are from it.

One way to avoid the illusion of progress is to deliver integrated work more frequently. In Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation, Jez Humble and David Farley say, “If it hurts, do it more frequently, and bring the pain forward.”

The Illusion of Control

Control-oriented practices like reviews or quality gates can serve to clarify intentions, facilitate trade-offs, provide feedback, or reduce risk, but they cannot guarantee outcomes.

Complex systems don’t work that way. There are no guarantees. In fact, in a complex system, attempts to control the outcome tend to backfire. Tight timelines intended to control a schedule can result in shortcuts that lead to quality problems and late-breaking surprises, ironically causing schedule overruns.

Pulling the Wrong Lever

Chapter 2 Seeing

Chapter 3 Variability

Part Two: Finding Levers

Chapter 4 Unintended Consequences

A change in one part of the system will inevitably cause changes elsewhere. Pul ing levers in isolation without understanding the system will backfire

If you don’t understand how the system works, you’re more likely to have levers backfire on you. But you can’t study it forever and not act

Sea Change

For now, I will serve as acting CTO while we look for a replacement

I know you and your team were busy keeping the system up yesterday, but did you have time to dig into what triggered the outage?”

*“It was the first Monday of the month. That matters because clients have automated jobs to pull monthly reports. Starting at 4 a.m. our time, we saw a spike in traffic followed by an increase in CPU * and memory utilization on every one of the six production cluster’s orchestrator nodes.

Production is running the same old code as last month

Wait a minute,” Mark said. “This isn’t the same code.” He looked at Theresa. “Ashley patched the central control plane during the MegaCorp outage.” Theresa’s eyes widened. “Of course,” she said. “But Ashley’s patch improved performance. It should have decreased latency not increased…” S

The patch. Remember how it increased concurrency…” Mark’s eyes widened in comprehension. “Oh. You mean the increase in concurrency meant the start-of-month bump in traffic put more load on the worker nodes…”

It’s not just tech debt, Rohan! Our whole KTLO burden is too high.”

it’s all the things we do to keep production humming.” She shot a pointed glare at Rohan, then gestured to Mark. “It’s partnering with Ops on rollouts across all six of our production environments and on implementing observability, creating dashboards, and performing administration tasks that require an understanding of the code base.

partner with Customer Success to onboard new customers, handle data deletion requests, and write diagnostic tools.

What do we need to do this week to make sure production doesn’t go down again?”

Now let’s talk about what’s coming up next, once the release is out and you’ve ensured production is rock solid. MegaCorp isn’t only upset about the recent outages. They’re also unhappy about our slow progress on the roadmap

We’ve already discussed the fact that software development is a nonlinear system. As a result, it’s difficult to ascertain the connection between cause and effect

Unintended consequences go one step further. They’re the reason things that seemed like a great idea at the time backfire

Predictability and Variability

Overreacting to Variation Creates Illusion Or imagine a confrontation with a stakeholder like Rohan who says, “You need more of a sense of urgency!” Now the team reacts to being blocked on a given ticket by starting the next ticket. Soon they have dozens of tickets in flight

This situation compounds over time. As they feel variability and start more work, they plan further and further out—expecting that work to get blocked, expecting delivery to take longer than desired. More work is added in parallel to compensate

In the late 1990s, Elisabeth went to work for a Silicon Valley startup. The company had grown quickly and, as is typical for startups, had shifted their focus multiple times. Unsurprisingly, the product had quality problems. The company decided to invest heavily in quality initiatives

handed a hefty budget to build out a testing team

Instead, quality continued to get worse

The more the organization invested in independent testing, the less the developers tested. The independent testers were finding more bugs because there were more bugs to find. Further, the more bugs there were to find, the more the organization had to make difficult decisions about which bugs to fix. Everyone was spending more time in bug triage meetings

The story of better testing, worse quality highlights another aspect of systems: local versus global optimization. Pioneering systems thinker Russell L. Ackoff said, “In any system, when one improves the performance of the parts taken separately, the performance of the whole does not necessarily improve and frequently gets worse

In his 1989 paper “The Germ Theory of Management, ” Myron Tribus, an American organizational theorist, referred to variability as “the virus of systems.”3 Similar to Ackoff, Tribus goes on to explain how unintended consequences can arise from creating goals to optimize parts of a system in response to variability. In his Perversity Principle, Tribus said: If you try to improve the performance of a system of people, machines, and procedures by setting numerical goals for the improvement of individual parts of the system, you will pay a price where you least expect to

The problem with quality is that it is in the eye of the beholder. The things that one member of the development team thinks will improve quality may not be an improvement from the customer’s perspective

the developers thought battery capacity was critically important, but customers wanted greater suction power

Further, the developers had a far more complex mental model than customers

Rework and Maintenance—Lost Capacity

In our story, Limitless Systems is struggling to execute

Preston correctly states that it’s not a capacity problem, but how can they frame the problem so they can address it?

John Seddon discussed the concept of value demand versus failure demand in his book Freedom from Command and Control. He built the model when working with support team

He defined value demand as “the demand customers make for things they want, things that are of value to them,”

failure demand as work “caused by a failure to do something or do something right for the customer.

Extending that model to software development, value demand is effort involved in delivering what the customer wants

that suggests that failure demand is everything else

If the strict dichotomy does not feel right for your organization, you can use a third category as an adjustment to your total capacity.

The point is not to be precise in categorizing the work but rather to have a shared understanding of where the time goes and how you can influence the primary drivers of failure demand

Sublime irony

Levers backfire when they represent a solution that is insufficiently connected to the actual underlying problem

Consider capacity and execution. When things are moving slowly, it’s tempting to reach for a capacity lever. Hire more people. Outsource work. But adding more capacity can make things go even slower if the underlying problem is related to execution rather than capacity.

Hiring more people can’t fix a problem with product-market fit. That just increases the burn rate and makes the economic situation worse.

The trick isn’t to perfect your ability to predict all possible interactions of every lever. That’s not possible. Instead, the goal is to learn how

to watch for signs of backfiring and avoid pulling the same lever over and over, hoping that next time it won’t backfire. It all comes back to reading signals.

Surprises Are Signals

Summary

Use the CREATE framework to help you think about all the areas to keep an eye on.

FINDING HIDDEN CONNECTIONS

Chapter 5 Cause and Effect

Act where the pain is caused, not where it is observed.

there are no problems that stem from a single root cause and no successes that belong only to heroes.

That’s why it’s so difficult to connect cause and effect, especially when the effect isn’t instantaneous

This chapter introduces causal models as a way of visualizing the various influences on your system so you can reason about the system as a whole, and you can see a broader range of places you can intervene to improve outcomes.

Dueling Priorities

“Carla and I were just talking about the backlog of open security tickets,” Preston answered. “Yes. I was explaining to Preston that I’ve had to grant exceptions to our security policies because keeping external dependencies up to date hasn’t been prioritized,” Carla explained.

*“To be fair, the issue isn’t just conflicting product priorities,” Theresa said. “The more we change, the more likely we are to introduce new bugs. We have external dependencies, but we also have our own crosscutting dependencies where the architecture is tightly coupled. That makes changes riskier.” * “The longer we wait to update those external dependencies, the more likely we are to have a serious security breach,” Carla countered.

Which is more important? The roadmap or the external dependency updates?” Preston looked serious. “Theresa, I don’t know if you understand how important this is. We’re going after more big companies like MegaCorp, and part of their evaluation criteria includes our security policies. This isn’t just a hygiene exercise. It’s critical to the business.”

“What if your group didn’t have to do the updates?” Carla asked.

“Remember that contracting company that did some app development for us last year?” Carla asked.

Theresa hesitated. “I have concerns,” she admitted

I don’t think anyone outside the organization could do them safely

*“I understand your concerns,” Preston said. “But we have to try something different. If we don’t solve this problem, we’re putting the entire business at risk.” Theresa frowned. “I thought you just said yesterday that adding more capacity wouldn’t solve the problem.” * “What I actually said was that adding more engineering capacity wouldn’t accelerate feature delivery,” said Preston

Even if we get caught up now, what are we going to do next year?

I’ve been wanting to automate bumping external dependency versions so it’s easier to stay more up to date, but we haven’t had the time or the staffing to even think about it. Could we feed two birds with one scone?”

Reframing Blame

The problem with blaming is that not only is it unlikely to paint an accurate picture of the situation, but it also leads to a less psychologically safe environment where people hide mistakes, leading to worse outcomes

Deming and the Red Bead Experiment

Their job was to produce white beads by dipping a specially constructed paddle with divots into a container of mixed white and red beads and moving it around until there was a bead in each divot. When they extracted the paddle, it usually held a mix of white and red.

Deming would chastise the workers for their defects (the red beads), then address the remaining participants to decide what to do about this shoddy workmanship. He offered solutions such as more training, motivational posters and slogans, visual reminders that red beads are bad, supervision of “low performers,” and firing the worst performers. People laughed at the absurdity of the solutions. None of those things mattered.

He had a 94-6 rule. He said, “I should estimate that in my experience most troubles and most possibilities for improvement add up to the proportions something like this: 94% belongs to the system (responsibility of management); 6% special.”3 This lines up with the ideas of variability we discussed in Chapter 3.

The system creates the majority of variability and issues. It’s leadership’s responsibility to understand the system, what is causing the variability and issues, and address the system as a whole.

Causal models help you see the system influences at work and, in doing so, find the appropriate levers or interventions so you can change the system

Visualizing Influences

The diagrams throughout this section are examples of causal models.

Causal models are a foundational systems thinking tool that helps you connect the dots between attributes of the system and their influences

Our style of causal models is inspired by a combination of Donel a Meadows’s and Gerald Weinberg’s diagramming styles. We use clouds to represent qualitative attributes (e.g., “perceived risk”), circles to represent quantitative attributes (e.g., “count of red beads”), and lines with arrows to show that one attribute influences another

When attributes move together (i.e., when one goes up, so does the other), we label the connector with a plus sign (+)

Let’s model the factors influencing the growing backlog of out-of-date, external dependencies. In the story, Theresa put off updating because all changes are risky, and there is too much other work to do.

See Figure 5.1.

We can expand this even further, adding in more influences, as in Figure 5.2.

With the more comprehensive causal model, you can see even more opportunities to intervene. We could continue to expand and refine this causal model, but let’s pause here to talk about creating a causal model step-by-step.

How to Create a Causal Model

The steps to create a causal model are straightforward: 1. Begin with a focus. 2. Identify influences and their relationships. Repeat this step to find the second-order influences on the first influences you identified. 3. Emphasize essential elements. 4. Identify possible levers.

modeling is remarkably difficult

Although you could create causal models on your own, this is an excellent exercise to do col aboratively with a group. Remember how Theresa and Carla both saw risk, but from different sources

Step 1: Begin with a Focus

start with a purpose—a problem you want to solve or something puzzling that you want to understand better. The trick at this stage is to figure out what attribute of the system is most closely connected to the problem or puzzle

how you will know if things are getting

better or worse

does faster delivery mean changes are delivered sooner? You would probably look at cycle time. Or that more changes are in each release? You ’d want to look at ticket counts. Or that it takes fewer work-hours to deliver? You ’d want to look at average hours per ticket or per story point. How you frame the focus of your model will have a big effect

Make sure to frame your focus as an attribute of the system

1. It can increase or decrease

2. It fits into the CREATE framework. This imperfect but useful sieve ensures that you’re focused on something concrete that is connected to an outcome that matters, rather than on an abstract concept.

Once you have identified the focus, write it in a cloud or circle (depending on whether it is a qualitative or quantitative attribute) in the center of your diagram.

You saw an example model earlier related to the number of out-of-date dependencies. Let’s build another example step-by-step related to the missed release date

We could either frame

that as “releases per year” or “time to release.” Both are equally valid. For our purposes in this example, we’ll choose “time to release.” That’s quantifiable, so we’ll put it in a circle

Step 2: Identify Influences and Their Relationships

Let go of problem-solving while modeling so you can be more open to seeing signals and influences you hadn’t noticed before.

Figure 5.4: Expanding Causal Model, Showing Influences

Work your way from the central problem to its influencing attributes, then farther out to the things that influence the influences

if the Engineering managers—Ashley, Andre, Kai, Samir, and Nora—were col aborating on this model, they would add in the factors contributing to their frustrations

Figure 5.5: Expanded Causal Model, Showing Second-Order

You could map all the way to the far reaches of your

universe, capturing attributes that describe the ecosystem of your technology, the competitive landscape your company operates in, and even macroeconomic conditions. They’re all valid influences. However, not every influence has the same weight or deserves the same consideration. You need to figure out when to stop adding to the model

look for the edges of your influence. Prominent modernist architect Eero Saarinen credits his father, Eliel Saarinen, Finland’s top architect, for teaching him to “Always design a thing by considering it in its next larger context—a chair in a room, a room in a house, a house in an environment, an environment in a city plan.”4 As you’re capturing the web of causal relationships, consider the next larger context each attribute exists within, but not beyond that. When you find you’re consistently identifying attributes beyond your sphere of influence, you’ve captured enough

Step 3: Emphasize Essential Elements

You’ve already focused the model only on those things within your sphere of influence. There are two more things you can do:

Figure 5.6: Causal Model with Stronger Influences Bolded and Attributes Annotated with Dimensions of the CREATE Framework

Step 4: Identify Possible Levers

Figure 5.7: Causal Model with a Decision Point

Your Model Is Wrong (and That’s OK)

Statistician George Box is widely credited with saying “All models are wrong; some are useful.” It turns out his actual original quote adds useful nuance. What he really said is, “Since all models are wrong, the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.”

Your model is not a picture of reality but rather your hypothesis about how reality works. Every unproven connection in your model is a hypothesis that is potentially worth exploring. (We cover using causal models to create hypotheses more in Chapter 11.)

If you are having difficulty finding levers in your causal model or if your model just feels unhelpful, consider the possibility that you may have oversimplified the model

The power of modeling is not in the model itself but rather in the discussions that the model supports and arriving at a shared understanding of reality

Dashboards Are Not Enough

When to Create a Causal Model

Causal models are useful any time you are trying to understand your system or find points of leverage to change it

col aborative modeling is even more powerful

Causal models are less useful for straightforward tactical decisions

Summary

Every system is perfectly designed to get the results it gets, as Professor Paul Batalden, the 2010 Deming Medal award winner said

CREATE YOUR MODEL

In this exercise, you’ll create your first causal model

Chapter 6 Feedback Loops

Systems shift and change over time because the output of the system feeds back into the system. Fast feedback supports timely decisions

when you begin to explore different kinds of feedback, this simple concept becomes much more complicated.

1. When an observation about the results of an action becomes an input to a decision about the next action.

2. When there is a cycle in a causal model

Feedback makes a system adaptive, but not all feedback is equally valuable, and sometimes something that looks like feedback isn’t.

Operation Snowball

It’s bad enough that we have problems with our test environments and flaky tests in the regression suite. But the smoke tests aren’t flaky. When they fail, they’re usually right that there’s a problem. Unless it’s an environmental thing. But it’s impossible to tell exactly what’s wrong without a lot of digging

What’s the difference between the regression tests and the smoke tests?” Nora asked.

*The regression suite puts the core engine through its paces,” Samir said. “It covers all kinds of error and stress conditions, but when it fails it’s just as likely to be a problem with the test suite as with the core.” * Ashley added, “And that’s why we don’t really trust them when they fail.

The smoke tests are supposed to be a smaller set of happy-path tests to give us a quick health check.” “That’s what they’re supposed to be,” Kai added. “But they go end to end through the entire system, including the apps, and it’s grown over the years.

They run in the pipeline whenever anything changes in the apps, API, integrations, or core engine,” Andre added. “It can only run against a fully deployed environment. No local execution

*Kai chuckled. “Too bad technical debt isn’t like financial debt. Then we could get a consolidation loan or declare bankruptcy.” * Andre looked thoughtful. “I wonder if the Snowball Method1 would work.

It’s a personal finance thing. Dave Ramsey, the finance guy, talks about prioritizing paying down debts from the smallest balance to the largest instead of paying off higher interest debts first. As you pay off each debt, you add its payment amount to your payments on the next largest debt. It’s all about building momentum, not taking the most optimized path.”

But we just don’t have time to clean up the smoke tests right now, and it’s not the most important thing to do, anyway

“Oooh! I love a mess!” Nora quipped.

What, are you the Marie Kondo of test suites?”

The test suite kicked off. “It takes a long time to run,” Kai said apologetically. “We won’t have results back for another fifteen minutes.” “Wel , then let’s use that time to see what we can tidy while we wait,” Nora suggested, “starting with the test duplication we just found.”

Test case after test case, they pared the smoke tests down to the essential use cases.

When the pipeline finished running the tests, Kai was surprised to see that his one small fix had cut the failures in half

a complex section of code that sprawled across a two-thousand-line file with deep nested indentations. “Ah. I see.” Nora nodded. “But I have an idea.” “Oh?” “Kent Beck says ‘for each desired change, make the change easy (warning: this may be hard), then make the easy change.’2 What if we

begin by refactoring this existing code and wrap that in simple unit tests as we go?”

They extracted complex logic into simple methods supported by unit tests. An hour later, he shook his head. “I never would have believed it,” he said. “But I see the problem now, and it’s a simple fix. Does this always work?”

By the end of the afternoon, the tests were passing, and the suite itself had half as many tests while slightly increasing the code coverage. By stripping out all the accumulated incidental details and duplication, they’d been able to see gaps in the test coverage more clearly, so they added in a handful of missing test conditions. Better yet, the total runtime had gone from fifteen minutes to five

Along the way, they improved test names and clarified failure messages, so it was much easier to see what had broken when the tests failed

I’ve just lived with that mess for the two years I’ve been here, and we cleaned it up in an afternoon. We could have done this so much sooner and saved so much time.

Do a Thing, See What Happens

They made changes to the tests and code, then the pipeline ran the

tests and gave them results. It’s the simplest kind of feedback there is: Do a thing and see what happens.

Dr. Deming popularized the Plan-Do-Study-Act cycle (PDSA, also sometimes called the Plan-Do-Check-Act or PDCA). He consistently credited Walter Shewhart with the idea, though Shewhart’s stages were specification, production, and inspection

There are other similar loops. Air Force Colonel John Boyd used his experiences as a fighter pilot in the Korean War to develop the OODA loop: observe, orient, decide, act. The OODA loop is about decision-making in critical situations, but it also centers on the idea of figuring out what to do, doing it, and using the observations from the resulting context to inform the next decision

Then there’s the build-measure-learn loop that Eric Ries developed as the core of his Lean Startup methodology. The key to every one of these loops is using observations from an action to inform the next action.

See Figure 6.1 on page 118 for examples of these feedback loops

The problem is that while it’s simple to talk about feedback, it’s much harder to interpret the information you receive. You have to understand the connection between the actions and the results.

Despite or Because

Chapter 7 Reinforcing and Balancing Loops

Part Three: Pulling Levers

Chapter 8 Context

Chapter 9 U-Curves

Chapter 10 Strategy Requires Choice

The more options you have, the more you’re able to see the path ahead and craft an actual strategy.

Strategy doesn’t require reams of analysis and a fancy board room. It requires context, understanding of relationships, and choice. In the absence of choice, you cannot have strategy. In the last chapter, we showed you how to find options leveraging U-curves. This chapter goes deep into how options create strategy and other ways to find options.

Auld Lang Syne

*She was distracted, fumbling with her phone, and almost ran into a man standing by the pickup counter. He looked up from his phone startled. “Theresa?” * It was Gus. Well, this is awkward, she thought. “Hi, Gus,” she said, trying to keep her face neutral. She was surprised to discover she was still holding a grudge for the way he treated her. “How are you? What are you up to?”

Tell me more about why you didn’t want to consider a shift in strategy.” Theresa wondered what Gus knew that she didn’t.

We ’d spent all that time working hard to make MegaCorp successful. I knew they were frustrated, but I thought we were inches away from pul - ing the relationship out of the fire. If we could do that, we ’d be on a path to sell to other companies of their size. Rohan had a good roadmap, your teams were making at least some headway on delivering faster, and Monique built a team tailor-made to serve enterprise customers.

it seemed foolish to walk away from that investment.” “And then our biggest logo churned.”

“You really got fired for disagreeing with Preston?” Theresa was incredulous.

Zoe was pushing on both him and the board about the difficulties Sales was having trying to sell the existing product to enterprises. In turn, Preston was pushing me to focus less on big bets with uncertain pay-offs. He bought into Zoe’s ideas about selling more, smaller deals. I believed addressing the smaller customers was a giant distraction we couldn’t afford.

“It wasn’t a single disagreement. It was a long-standing alignment problem. I made it worse because I ignored Preston’s input and refused to let Rohan change the roadmap. Turns out Preston and Zoe were right, and I was wrong.”

As I’m sure you can imagine, we’re scrambling to adapt to the new strategy,” Theresa said carefully. “Got any advice?”

Smaller customers? Half your work is probably going to be simplifying and streamlining the code to support fewer options with less configuration.” Theresa nodded. “Fewer options, but also more self-serve. That, and our current debate is the frequency of releases. Rohan wants to ship smaller releases more frequently.”

Then she couldn’t help adding, “I’m curious. The configurability added so much complexity. That made the code so much more difficult to change and test. Was it worth it?” Gus looked thoughtful again. “Like I said, it seemed obvious to me that we were going to need that level of configurability. Our strategy was built around enterprise customers, so everything followed suit from there.”

She wondered if she ’d made decisions because they seemed “obvious” at the time. She realized that any choice was all about trade-offs. Each decision created possibilities but also had costs and consequences

Strategy

Chutes and Ladders is a classic board game where you roll a die, move your pawn, and do what the board tel s you to do

*This simple game can teach us something profound about strategy. Think about it for a moment: What’s a winning Chutes and Ladders strategy? * Trick question, right? That’s the point.

In the absence of choice, there can be no strategy. This is why one of the most important and foundational thinking tools is choice. More specifically, learning to do the following: • Expand your options beyond the most obvious ones. • Avoid prematurely committing to a path. • Make choices that increase your future choices, rather than boxing you into a corner

Playing a Winning Game: Creating Your Options Map

Let’s picture the choices that the leadership team at Limitless Systems is making as a game board

The company pivoted to a different path, one that Rohan was contemplating all the way back at the beginning of our story. You could visualize these different paths to success like a game board as shown in Figure 10.1.

We call this an options map. It’s a way of visualizing possible paths from a starting point to an intended outcome. Options maps aren’t exactly a roadmap, although you might use an options map to inform a product roadmapping process. They’re not a decision tree, but you can use them to help you make decisions. Like causal models, an option map supports col aborative decision-making.

Unlike causal models, an option map doesn’t have a standard symbology any more than game boards do.

crossroads where a decision is necessary, which parts of the territory will be treacherous, where there are existing paved roads, and where you’ll have to blaze new paths.

Figure 10.1: Limitless Systems Options Map

Let’s walk through each step

Begin with the End in Mind

The first step in creating your options map is defining the possible end scenarios. One of those end scenarios will be your win condition.

we will focus primarily on defining your wins. As you think about your win condition, be clear about the problem you are solving. Think in terms of outcomes not output.

If you define winning in terms of a code coverage target like 80% statement coverage, you might reach your win condition and yet not see any real benefit. These are examples of producing an output for the sake of the output. There are so many more: Agile transformations that achieve agile transformation. Cloud migrations that achieve cloud migrations. Reorgs that achieve reorgs.

You can avoid the output trap by establishing a win condition that speaks to the intended benefit. If the intention behind increasing code coverage is to reduce defects or increase confidence before shipping, say that

Starting Context

Next, put a starting space on your options map to signify your starting context. You could just put the word “Start” with a bubble like in Figure 10.1. However, you might want to add more information if you are focusing on a narrow aspect of your current context

The Messy Middle

Once you have your start and end conditions, it’s time to fill in the middle steps. This is where things get interesting

You’ll add steps to your options map like spaces on a game board. Each space represents a shift in context. When you arrive there, something is true that wasn’t true before. Your goal with the messy middle is to identify a range of options so you’re in a better position to develop a strategy

Consider code freezes, or a period of time when changes cannot be merged or deployed. This is a common approach to mitigating risk. Retail-ers typically avoid changes during the holiday season, for example. Are code freezes the only option? Wel , no. For that matter, code freezes just shift risk around: That first deployment following a code freeze will be big and hairy and is much more likely to fail spectacularly. Other options could include smaller, faster releases; investment of leading indicators of problems; reducing blast radius of changes; increasing recoverability; and many, many more. Each of those could be a square on your options map

You could work backward from the end conditions or forward from the starting context or work from the middle out

Let’s build an options map for an example from our story. Recall the debate that Rohan and Mark had about the frequency of deployments

We’ll start the options map by choosing a win condition. Let’s aim for Rohan’s goal of daily deployments. However, we want to be very specific about what it means to win. It’s not enough to do daily deployments. It has to be safe.

The starting context is the current rate of deployments. At this point in our story, that’s roughly quarterly

Theresa and the team created a whiteboard full of stickies of all the steps to deploy. They would be great fodder for inspiration. However, let’s say that for the purposes of this example the biggest factors are scheduled downtime, manual steps, environment configuration divergence, and change failure rate

These drivers suggest interventions to consider

You could sequence these steps in a variety of ways and still end up at the win condition

If you are thoughtful about how you lay out possible steps, you might find valuable interim win conditions. For example, there are two interim wins marked with a star in Figure 10.4.

Creating More Options

How do you know if you’ve identified enough options? If you only have one option, you’re playing a game of chutes and ladders: You can’t form a strategy. Two options run the risk of being a false dichotomy (and that’s even worse when the second option is only there to make the first option look better). But if you keep exploring to find more options, you end up in analysis paralysis

The right number of options for your context is a function of two factors:

1. The expected cost of making the wrong choice. The more options you consider, the higher the probability of finding the best possible choice.
2. The cost of identifying and evaluating options

If that is starting to sound a lot like a U-curve, it’s because it is

let’s look at approaches to identifying more options

Lean on Causal Models

Relax a Constraint

One example: unnecessary consistency. Organizations often apply the same rules across the board, ignoring differences in local context

For example: • A team working on a new greenfield product was forced to fit into the same long-term planning cadence as a team with an existing legacy product

Revisit Assumptions

Whenever you catch yourself thinking “We can’t do that because…” examine what comes next carefully.

Create Consistency

Sometimes you can create possibilities by making inconsistent things more consistent. This is what good standards do

Sequencing and Timing Levers

As you fill in the messy middle of your options map, you can create even more options through different sequences of steps. You can also choose to pull certain levers sooner or hold off.

Last Responsible Moment

*If you pull a lever too fast or without understanding the system, you risk tampering and cutting off options. Pull a lever too slow and you risk fal - ing further behind or a problem gets worse. * So, when do you pull a lever? At the most responsible time. The most responsible time to decide is when the value from having options is less than the costs of maintaining those options.

framed by Mary and Tom Poppendieck as “…the last responsible moment, that is, the moment at which failing to make a decision eliminates an important alternative.

Pausation as a Strategy

Sometimes the most powerful strategic move is the decision not to act—at least, not yet

an intentional pause, a nonaction. We call this pausation. It is similar to the concept of slowif-ication from Wiring the Winning Organization by Gene Kim and Steven Spear

Pausation might be one of the most difficult things for engineering leaders to do. We see a problem; we want to act to fix it. But what if acting exacerbates the problem? What if the problem is self-correcting?

Walter Shewhart identified two types of mistakes

1. Reacting to an outcome as if it came from a special cause variation when it really came from a common cause 2. Treating an outcome as if it came from common cause variation when it actually came from a special cause

Trying to fix perceived problems that are part of normal variability is also a poor decision. Dr. Deming would call this tampering. He created his Funnel Experiment to demonstrate that tampering makes results worse.

Common examples of tampering include:

Making changes without using Shewhart charts and understanding influences. We find very few organizations that make changes based upon studying their system and understanding what normal is in their context

Creating random policy based on surveys

Clustering Levers

Imagine you used your causal model and determined that the uncomfortably high variability in your cycle time was largely due to accidental coupling of code over the years

you found your lever, but that lever may require a few other levers to be pulled in concert.

To be able to reduce the accidental coupling, you need a strategic architectural plan. That architectural plan may require a rethink of how some of the software is developed, perhaps requiring some upskilling in microservices or implementing idempotent services. This will require a new way of testing and perhaps some new observability to be implemented. These levers will need to be clustered together.

The feedback loops do not need to be large because you’re clustering levers together. As always, use your causal models to look for ways to decrease the feedback latency.

Sequencing Levers

The order in which you pull levers will also have an effect, not only on overall impact of the decisions but also potentially in the fidelity and quality of feedback loops. As you map your options, consider if there’s a sequence of lever pul s that could unlock an earlier interim win or a move that could unblock other possible moves.

Let’s look at the deployment frequency challenge that Limitless Systems is facing. The options map in Figure 10.4 shows six steps

They’re grouped in three themes: deployment fully automated, zero-downtime deployment, and rol backs. You can see from the map that there are many choices for sequencing.

If Theresa and team are strategic about the order they traverse the map, they’ll gain interim wins: • Fully automated deployments will save time. • Zero-downtime deployment will allow them to deploy without a maintenance window. • Rol backs will reduce the risk of a failed deployment. Which of those wins do you think would benefit them most? Which would have the biggest impact on frequency of releases? This is just a thought exercise. There is no one right answer. You would need to know a great deal more about their technical context to weigh the trade-offs accurately

Backed into a Corner

Gus’s decisions backed Limitless Systems into a corner. By relentlessly pursuing the enterprise market and pushing Rohan to squeeze more features out of engineering, Gus set the stage for an overly complex and fragile product that generated high failure demand. By the time MegaCorp defected to their competitor, pivoting was their only real option to get out of the situation.

Sometimes it isn’t that there are no choices but that the choices are too expensive. At one company, a product manager wanted to do more product experimentation and A/B testing, but the architecture and current product design made such experimentation extremely costly. A/B testing was an option, but one that was extremely difficult to justify. What can you do if you find yourself backed into a corner with no good options? Use an options map to navigate to a new context with more options

Summary

All the system thinking tools you have learned so far, including U-curves, causal models, and the CREATE framework, come into play as you build your options map.

BUILD YOUR OPTIONS MAP

Let’s explore the strategy for an initiative you have been working on using an options map. Your current context is the start. Create your options map

Chapter 11 Hypotheses in Change

Figure 11.4: A Causal Model Showing More Contributing Factors to Cloud Costs

Part Four: Building a Thriving Organization

Chapter 12 The Learning Flywheel

Chapter 13 Self-Correcting Systems

Chapter 14 A New Context