(2003-06-30) Iterative and Incremental Development: A Brief History

On Iterative and Incremental Development: A Brief History. We chose a chronology of IID projects and approaches rather than a deep comparative analysis.

PRE-1970

IID grew from the 1930s work of Walter Shewhart,1 a quality expert at Bell Labs who proposed a series of short “plan-do-study-act” (PDSA) cycles for quality improvement. Starting in the 1940s, quality guru W. Edwards Deming began vigorously promoting PDSA, which he later described in 1982 in Out of the Crisis.2 Tom Gilb3 and Richard Zultner4 also explored PDSA application to software development in later works.

The X-15 hypersonic jet was a milestone 1950s project applying IID,5 and the practice was considered a major contribution to the X-15’s success. Although the X-15 was not a software project, it is noteworthy because some personnel--and hence, IID experience--seeded NASA’s early 1960s Project Mercury, which did apply IID in software

Project Mercury ran with very short (half-day) iterations that were time boxed. The development team conducted a technical review of all changes, and, interestingly, applied the Extreme Programming practice of test-first development, planning and writing tests before each micro-increment. They also practiced top-down development with stubs

The recollections of Jerry Weinberg, who worked on the project, provide a window into some practices during this period. (see Agile Impressions)

We were doing incremental development as early as 1957, in Los Angeles

When much of the same team was reassembled in Washington, DC in 1958 to develop Project Mercury, we had our own machine and the new Share Operating System, whose symbolic modification and assembly allowed us to build the system incrementally, which we did, with great success. Project Mercury was the seed bed out of which grew the IBM Federal Systems Division. Thus, that division started with a history and tradition of incremental development.

The earliest reference we found that specifically focused on describing and recommending iterative development was a 1968 report from Brian Randell and F.W. Zurcher at the IBM T.J. Watson Research Center

THE SEVENTIES

In his well-known 1970 article, “Managing the Development of Large Software Systems,” Winston Royce shared his opinions on what would become known as the waterfall model, expressed within the constraints of government contracting at that time.9 Many--incorrectly--view Royce’s paper as the paragon of single-pass waterfall. In reality, he recommended an approach somewhat different than what has devolved into today’s waterfall concept, with its strict sequence of requirements analysis, design, and development phases. Indeed, Royce’s recommendation was to do it twice

Royce further suggested that a 30-month project might have a 10-month pilot model and justified its necessity when the project contains novel elements and unknown factors (hardly a unique case). Thus, we see hints of iterative development, feedback, and adaptation in Royce’s article

What did Royce think about the waterfall versus IID when he learned of the latter approach? In a personal communication, Walker Royce, his son and a contributor to popular IID methods in the 1990s, said this of his father and the paper: He was always a proponent of iterative, incremental, evolutionary development. His paper described the waterfall as the simplest description, but that it would not work for all but the most straightforward projects

Early practice of more modern IID (feedback-driven refinement with customer involvement and clearly delineated iterations) came under the leadership of Mike Dyer, Bob McHenry, and Don O’Neill and many others during their tenure at IBM FSD. The division’s story is fascinating because of the extent and success of its IID use on large, lifecritical US Department of Defense (DoD) space and avionics systems during this time.

The first major documented IBM FSD application of IID that we know of was in 1972. This was no toy application, but a high-visibility life-critical system of more than 1 million lines of code--the command and control system for the first US Trident submarine

In 1975, Vic Basili and Joe Turner published a paper about iterative enhancement that clearly described classic IID:14 The basic idea behind iterative enhancement is to develop a software system incrementally, allowing the developer to take advantage of what was being learned during the development of earlier, incremental, deliverable versions of the system.

In 1976, Tom Gilb published Software Metrics (coining the term), in which he discussed his IID practice--evolutionary project management--and introduced the terms “evolution” and “evolutionary” to the process lexicon. This is the earliest book we could find that had a clear IID discussion and promotion, especially of evolutionary delivery

Gilb is one of the earliest and most active IID practitioners and promoters. He began the practice in the early 1960s and went on to establish several IID milestones. His material was probably the first with a clear flavor of agile, light, and adaptive iteration with quick results, similar to that of newer IID methods.

And perhaps reflecting several years of seeing IID in action at FSD, Mills asked, “...why do enterprises tolerate the frustrations and difficulties of such [waterfall] development?”

Although unknown to most software professionals, another early and striking example of a major IID success is the very heart of NASA’s space shuttle software--the primary avionics software system, which FSD built from 1977 to 1980. The team applied IID in a series of 17 iterations over 31 months, averaging around eight weeks per iteration

The first IID discussion in the popular press that we could find was in 1978, when Tom Gilb began

publishing a column in the UK’s Computer Weekly

THE EIGHTIES

In 1980, Jerry Weinberg wrote about IID in “Adaptive Programming: The New Religion,” published in Australasian Computerworld. Summarizing the article, he said, “The fundamental idea was to build in small increments, with feedback cycles involving the customer for each.” A year later, Tom Gilb wrote in more detail about evolutionary development.

In another mid-1980s questioning of the sequential life cycle, Gilb wrote “Evolutionary Delivery versus the ‘Waterfall Model.’” In this paper, Gilb promoted a more aggressive strategy than other IID discussions of the time, recommending frequent (such as every few weeks) delivery of useful results to stakeholders.

A 1985 landmark in IID publications was Barry Boehm’s “A Spiral Model of Software Development and Enhancement”

In 1986, Fred Brooks, a prominent software-engineering thought leader of the 1970s and 1980s, published the classic “No Silver Bullet” extolling the advantages of IID

Commenting on adopting a waterfall process, Brooks wrote: Much of present-day software acquisition procedure rests upon the assumption that one can specify a satisfactory system in advance, get bids for its construction, have it built, and install it. I think this assumption is fundamentally wrong, and that many software acquisition problems spring from that fallacy.

By the late 1980s, the DoD was experiencing significant failure in acquiring software based on the strict, document-driven, single-pass waterfall model that DoD-Std-2167 required

in a conversation nearly a decade later, the principal creator of DoD-Std-2167 expressed regret for creating the strict waterfall-based standard. He said that at the time he knew of the single-pass document-driven waterfall model, and others he questioned advised it was excellent, as did the literature he examined, but he had not heard of iterative development. In hindsight, he said he would have made a strong recommendation for IID rather than the waterfall model

In 1988, Gilb published Principles of Software Engineering Management, the first book with substantial chapters dedicated to IID discussion and promotion

1990 TO THE PRESENT

Hundreds of books and papers were promoting IID as their main or secondary theme

In the 1970s and 1980s, some IID projects still incorporated a preliminary major specification stage, although their teams developed them in iterations with minor feedback. In the 1990s, in contrast, methods tended to avoid this model, preferring less early specification work and a stronger evolutionary analysis approach.

The DoD was still experiencing many failures with “waterfall-mentality” projects. To correct this and to reemphasize the need to replace the waterfall model with IID, the Defense Science Board Task Force on Acquiring Defense Software Commercially, chaired by Paul Kaminski, issued a report in June 1994 that stated simply, “DoD must manage programs using iterative development. Apply evolutionary development with rapid deployment of initial functional capability.” Consequently, in December 1994, Mil-Std-498 replaced 2167A.

Meanwhile, in the commercial realm, Jeff Sutherland and Ken Schwaber at Easel Corp. had started to apply what would become known as the Scrum method, which employed time-boxed 30-day iterations

In the mid-1990s, many contributors within Rational Corp. (including Kruchten and Walker Royce) and its clients created the Rational Unified Process (RUP), now a popular IID method. A 1995 milestone was the public promotion of the daily build and smoke test, a widely influential IID practice institutionalized by Microsoft that featured a oneday micro-iteration.

In 1996, Kent Beck joined the Chrysler C3 (Chrysler Comprehensive Compensation) payroll project. It was in this context that the full set of XP practices matured, with some collaboration by Ron Jeffries and inspiration from earlier 1980s work at Tektronix with Ward Cunningham.

In 1998, the Standish Group issued its widely cited “CHAOS: Charting the Seas of Information Technology,” a report that analyzed 23,000 projects to determine failure factors. The top reasons for project failure, according to the report, were associated with waterfall practices. It also concluded that IID practices tended to ameliorate the failures.

In February 2001, a group of 17 process experts--representing DSDM, XP, Scrum, FDD, and others--interested in promoting modern, simple IID methods and principles met in Utah to discuss common ground. From this meeting came the Agile Alliance.

And in 2002, Alistair Cockburn, one of the participants, published the first book under the new appellation.

H.L. Mencken said, “For every complex problem, there is a solution that is simple, neat, and wrong.”

Software development is a very young field, and it is thus no surprise that the simplified single-pass and document-driven waterfall model of “requirements, design, implementation” held sway during the first attempts to create the ideal development process.