On this episode of “When Books Could Code”, we’re talking about the book that probably had the largest single impact effect on my career, Extreme Programming Explained: Embrace Change by Kent Beck, published in October 1999. I read it in… probably March, 2000. There was a second edition in 2005, we’ll cover that in a later post.

This version of the post that you are reading is about my fifth pass at getting this one out. My draft got up to 7000 words, so we are splitting it into parts. In this part, we’re going to talk about the book itself and the problem that it is trying to solve.

Just to put this out here — Kent Beck is the only person that I didn’t have on the Tech Done Right podcast because I was too big a fan. There’s a non-zero chance that Beck will read this, and that’s mildly terrifying. Kent, if that’s you — hi! Whatever I say here, this book changed my life, I genuinely believe in what it tries to do and I likely would have burned out on programming without your book.

A Story

Let me try and describe my head space when I first read this book, I think my reaction will make more sense.

It’s the year 2000. I was about a year into my first job out of grad school as the only professional software developer at a very small web consultancy. They had been a documentary film company that had wandered into creating web sites at exactly the time when there was a huge wave of money pouring into the web, and they hired me because they thought my graduate degree meant I knew something about software. I knew it didn’t but I didn’t tell them that.

In early 2000 we started a new project for a major Fortune 500 company, the first time that several of their subsidiaries would collaborate on a common website. We had a tight 90 day deadline. Not because they needed it in 90 days, but because — and I swear this is true — they wanted to prove to themselves that they could move at “web speed”.

I won’t go into all the ins and outs except to say that nobody on our side had ever worked on a project in need of this much project management (least of all me). It featured me snapping at a client that “they weren’t going to be the one up at midnight” as a response to some request or other. (The owner of my company had me apologize). It featured our CEO tearing up the entire UI five days before the project was due, not because the client was complaining, but because he just didn’t care for it. He also loved nothing more than having us make live changes to the staging site on conference calls with the client. (The staging server lived under my desk.) Don’t even get me started on the chain-smoking Oracle contractor we hired…

It was a long 90 days, made even worse by the fact that we never seemed to know what we were doing. I was frustrated because I was never able to leverage the few things I did actually know about software in a way that made the project easier.

In the background, I was starting to hear tell about something called “extreme programming.” It was the greatest innovation in programming since the invention of the zero, or possibly an excuse for cowboy coders to run amok. I was intrigued. (I already knew who Kent Beck was from Smalltalk Best Practice Patterns and also was aware of his JUnit writing and also that he was part of the original C2 wiki crew.)

Anyway, after a few all-hands-on-deck long weeks, we did ship the project on time. Only for the client to a) get mad it wasn’t available instantly because they didn’t understand how DNS propagation worked, and b) just sit on it for weeks, because they didn’t need it for anything immediate, they were just trying to prove they could move fast.

Right after that project, I read XP Explained. I was, you might say, primed for it. We had something like a retrospective on the project, where I insisted that we not treat it as a success just because we shipped. I read excerpts from this book, which prompted some teasing from the boss because of course I would bring a book report to a project meeting. (I have been me for a very long time.) Turned out we didn’t really have the capacity at that place to do anything like XP, and I wound up spending the next several years searching for a place that did. And also trying to make the places I worked into places that could work like this, to mixed success.

The Book

In partial rebuke to some stuff that I wrote last time, the forward for this book comes from the Gang of Four’s own Erich Gamma. Gamma’s forward ends with this:

[Beck’s] views and approaches always challenge the way I approach software development. There is no doubt that XP challenged some traditional big M approaches; this book will let you decide whether you want to embrace XP or not.

I think the “M” in “big M” is meant to be “methodology”, but I invite you to spend a few seconds thinking about funnier things it could be. “Meatloaf”, “Mustache”, and “Midichlorian” are three that come to mind for me.

Call me sensitive… but that’s not exactly a full-throated endorsement, is it? It reads one notch short of “in conclusion, XP is a land of contrasts.”

Beck’s actual introduction in the Preface is much better, and does a fantastic job of contextualizing the entire book.

XP is a lightweight methodology for small-to-medium sized teams developing software in the face of vague or rapidly changing requirements. This book is intended to help you decide if XP is for you.

Simple. Clear. Defines a boundary. It’s worth noting after decades of comments over whether Agile is for large teams that the specific initial claim here is that XP is not for large teams.

We then get a very clear statement of a core principle:

XP takes commonsense principles and practices to extreme levels.

• If code reviews are good, we’ll review code all the time
• If testing is good, we’ll test all the time

And so on…

This is to my mind, one of the two core elements of XP. It’s not about the specific practices, as such. It’s about the idea that if there is something in your process that is beneficial but you only do it occasionally, try doing it continuously and in smaller slices. It will likely work better. That, and continual improvement (which is not quite described in this version of the book), are the most important ideas I took from XP.

Beck argues that the specific practices in the book are great because they are based on things that are generally useful for most teams, so the book is a guidebook on how to make them extreme, and therefore better. There’s a little bit of tension here — the book wants to claim that the original 12 practices work best as a group because they are self-reinforcing, but also that teams should find their own practices and it isn’t a one size fits all. Resolving this tension is a big part of the book’s second edition.

XP, Beck says, makes two sets of promises. To programmers, “they will be able to work on things that really matter, every day.” To managers, “they will get the most possible value out of every programming week.”

The Problem

The first part of the book is about the problem that Beck is trying to solve.

Beck uses this section to frame software development. “The basic problem of software development is risk”. XP “addresses risk at all levels of the development process. XP is also very productive, produces high-quality software, and is a lot of fun to execute.” This is interesting to me because in my head, I’ve always remembered this book as identifying the cost of change as the basic problem. Risk and cost of change are similar, but not quite the same…

A very common “fact” bandied around about this time was the that a majority of software projects fail. It was never quite clear to me where that fact came from or even if it was true. I don’t feel like I hear about this stat much these days, which is partially because it was always a little bit bogus, and partially because web development enabled a large section of the software industry to re-define “success”. But the idea that most projects fail, I think, is a big part of the background against which this book was written.

There’s a short chapter about the economics of software development, which, interestingly, marks “interest rates” as one of the key factors in the business of software — spending later and earning sooner is good because you spend less interest on loans and more interest on income. This idea drives some of what Beck later writes about prioritization. Notably, this book was written before the zero interest rate economy of the post 2008 world, which changed the math quite a bit.

The goals of XP in this context are:

• To take smaller steps in order to minimize downside risk by increasing the chance you will catch a problem earlier.
• To ship working software sooner so that you can start earning or saving money.
• To plan for continual change and reduce the cost of later change, rather than building an immutable design up-front.

Chapter 5 is where we get to cost of change, “One of the universal assumptions of software engineering is that the cost of changing a program rises exponentially over time.” I’ll note that in the book Leprechauns of Software Engineering, Laurent Bossavit argues that this was never actually true. Or at least that it depends on what you mean by “over time”. Anyway, Beck lays out the case here that XP will keep the cost of change steady over time “this is the technical promise of XP”, Beck says, and now it’s clearer to me why I remember that being the main argument.

After that we get the somewhat famous driving analogy, where XP is compared to a driver who is always prepared to make small adjustments rather than letting the car drift for blocks at a time.

We also get the first look at the four values: communication, simplicity, feedback, and courage, which will eventually kind of morph into the Agile Manifesto.

And another core principle of XP:

It is betting that it is better to do a simple thing today and pay a little more tomorrow to change it if it needs it, than to do a more complicated thing today that may never be used anyway.

I’ll note that a) I still very much believe that is true and b) there’s kind of an excluded middle there.

We also have five basic principles: rapid feedback, assume simplicity, incremental change, embracing change, quality work. And four activities — coding, testing, listening, and designing. “Feedback” here refers mostly to “how fast can we tell if what we did worked” and not “I’d like a socially approved way to say uncharitable things about my co-workers”.

What’s New

While I think the solutions proposed by XP got most of the attention, Beck’s framing of the problem in software development was also novel and the framing is what makes his solutions seem feasible.

There was a lot of momentum in the 90s over the idea that you could solve all software’s problems with better requirements and design up front. You can see part of that movement in Design Patterns, and also looking at anything written about UML at that time, plus it was the message of a lot of the 90s hype about objects.

This book was written against the backdrop of two ideas that were in very common currency:

• A lot of software projects fail, and the new design ideas weren’t necessarily helping that.
• The cost of change feels like it goes up, and whether that’s because of the phase of the software or just because the software keeps getting more complex is maybe not relevant.

And if those are the background facts you are assuming, then the idea of “well, what if we didn’t do X” is worth entertaining. So, if we don’t do Big Design, what can we do to lower risk? Well, we can test. Okay, but what do we do to lower the cost of testing? Well, we write the tests first. Okay but what do we do to keep that from getting out of control? Okay we pair program…

And so on, that’s where XP comes from. Next time, we’ll talk about the proposed solutions, or at least the first wave of proposed solutions.