Assigning the Blame

Finish the product development with a feedback loop. Published in Embedded Systems Programming, January 1997.

By Jack Ganssle

The last step in most projects is the one we dread the most - assigning the blame. Who is responsible for the late delivery? Why didn't we meet the specification document? Who let costs spiral out of control?

 The developers, that's who. When management sheds blame like a Macintosh repels water we wonder why we got into such an unforgiving profession.

 Something happened in this country in the last couple of decades, something scary for the future. We've become intolerant of failure. In 1965 a horrible fire consumed the Apollo spacecraft and three astronauts. An investigation found, and corrected, numerous problems. There was never a serious question about carrying on

 In the 80s, when the Challenger blew up commentators asked what NASA was doing to ensure such a tragedy would normal">never happen again. Huh? Sitting on 6 million pounds of explosive and you want a guarantee that the system was foolproof? Even my car is not totally reliable. There are no guarantees, yet society seems to expect miracles from us, the technology gurus.

 Consider the Superconducting Supercollider. If scientists could promise a practical result, or perhaps only promise finally resolving the issue of the Higgs particle, then maybe the SSC would be something more than an abandoned hole in the ground. Fear of failure sent the politicians fleeing. Yes, it was very, very expensive. I was angered, though, by the national lack of understanding that, in science, failure is an element of success. We learn by trying a lot of things; with luck, a few pan out. From each defeat we have the possibility of crawling towards success.

 As developers, we've got to learn to manage both failure and success. Our companies are demanding more from us every day. Downsizing and increasingly frenetic time-to-market pressures mean that Joe Engineer must take advantage of every opportunity to learn.

 Yet there is no Embedded University. We're mostly educated via OJT, a haphazard and inefficient way of learning. Few of us are privileged to work with a mentor of stature, so the best we can do is to examine the results of everything we do, with a critical, unbiased eye towards improving our skills, and improving the processes used to develop our products.

 Does this scenario sound familiar? A small team starts a project with great hopes and enthusiasm. Along the way problems crop up. Sales changes the features. Management reduces the product's cost. Schedules slip when compiler bugs appear. Code grows bigger than expected. Real time response isn't adequate, so the engineers start burning the midnight oil, making heroic changes to get the system out, but schedules slip more, tempers flare, and when the product finally ships no one is speaking to each other.

 A week later the developers are embroiled in another product, again starting with high hopes, and again doomed to encounter the same rather small yet common set of problems that cause late delivery.

 Sliding into middle age one has the chance to observe patterns in one's life, patterns we seem to repeat over and over. One friend wisely says "doing the same things over and over, and expecting different results each time, is clearly insane."

 Yet most engineering efforts exhibit this insanity. Careening from project to project, perhaps learning a little along the way but repeating the same tired old patterns, is clearly dysfunctional.

Engineering Managers

 In most organizations the engineering managers are held accountable for getting the products out in the scheduled time, at a budgeted cost, with a minimal number of bugs. These are noble, important goals.

 How often, though, are the managers encouraged - no, required - to improve the process of designing products?

 The Total Quality movement in many companies seems to have bypassed engineering altogether. Every other department is held to the cold light of scrutiny, and the processes tuned to minimize wasted effort. Engineering has a mystique of dealing with unpredictable technologies and workers immune to normal management controls. Why can't R&D be improved just like production and accounting? 

Now, new technologies are a constant in this business. These technologies bring risks, risks that are tough to identify, let alone quantify. We'll always be victims of unpredictable problems.

 Worse, software is very difficult to estimate. Few of us have the luxury to completely and clearly specify a project before starting. Even fewer don't suffer from creeping featurism as the project crawls toward completion.

 Unfortunately, most engineering departments use these problems as excuses for continually missing goals and deadlines. The mantra "Engineering is an art, not a science" weaves a spell that the process of development doesn't lend itself to improvement.

 Phooey.

 Engineering management is about removing obstacles to success. Mentoring the developers. Acquiring needed resources.

 It's also about closing feedback loops. Finding and removing dysfunctional patterns of operation. Discovering new, better ways to get the work done.

 Doing things the same old way is a prescription for getting the same old results.

Postmortems

 It's infuriating that typical projects fizzle out in a last minute crunch of bug fixes, followed by the immediate start up of a new development effort. Nothing could be dumber.

Did you learn anything doing the project? Did your co-workers? Is there any chance some bit of wisdom could be extracted from its successes and failures - a bit of wisdom that may save your butt in the future? Why do we careen right into the next project, hoping to avoid disaster by sheer hard work, instead of taking a moment to take a deep breath, gather our wits, and understand what we've learned? normal">must allocate time for a careful postmortem analysis of each and every project. Once the pressure of the ship date is gone, all of the team members should work towards extracting every bit of learning from the development effort.

 Usually we casually pick up some wisdom even without a formal postmortem. This is the basis for "experience", a virtue acquired by making mistakes. I'll never forget shoehorning an RTOS into an almost complete system over a decade ago. Putting it in after 20,000 lines of code were written hurt so badly I swore I'd never start a system like that again without installing an RTOS as the first software component. This bit of wisdom came in exactly the same way kids learn not to touch a hot stove: pain. I believe we can do better than learning by acquiring scars.

 A formal postmortem analysis has one goal: squeeze every bit of learning from the just completed project. Wring it dry, extracting information to compress the acquisition of "experience" as much as possible.

 The postmortem is not a forum for assigning blame.  When we started conducting these the engineers immediately became paranoid, thinking that this was the chance for management to "get" them, in writing, in a venue visible to all employees.

 If blame must be given (in general a very bad idea), then do it privately and constructively. Non-constructive criticism is a waste of time, to be used only when firing the offending employee (if then).

Similarly, the postmortem shouldn't be used as a staging area for the engineers' complaints against management. When there are valid concerns (for example, schedule slippages due to changing specs), then these should be coldly, accurately documented in a form that's useful to all involved. No whining allowed.

 No, a successful postmortem is a non-emotional, non-confrontational, reasoned, thoughtful process. It works when all participants buy into the idea that improvement is important and possible.

 I feel a successful postmortem results in a written document that will be preserved with other engineering materials, perhaps in a drawing system. The document is important, as it's a formal analysis of ways of doing engineering better. Just as a contract is a written version of an informal understanding, the postmortem report codifies the information..

 A great postmortem results in a report that's eminently readable, which even people not involved with the project can understand. File these together and give them to all new hires to give them "virtual experience."  I've yet to find one that met this objective but keep hoping.

 Start the report with a clear description of the project's goals. Sometimes in the heat of development we lose sight of why we're doing the engineering. For instance, as I write this my company is doing a system that has numerous technical specifications!. but the reason for spending the money, for allocating the resources, come down to speed, flexibility, and cost, in that order. Though the project must meet the specifications (a minimum requirement for all development efforts), all engineering decisions will involve tradeoffs between these three parameters. These three goals give the engineers a way to make the tradeoffs. 

I feel so strongly about keeping an eye on the important issues and not getting mired in details that I posted a sign in the lab listing these parameters, as well as quantitative acceptable results.

 Next, list the project's results. Was it on time? Did the team meet all specs? Was the customer happy? Get the facts down, with as much detail as possible. Remember, that for the purposes of the analysis, absolute honesty is crucial. A truthful assessment is the start of any self-examination, as it is of this project-assessment.

 The rest of the document is a critical look at every part of the project. Did the specifications change often? How often, and what was the real impact on the project? Were the tools up to snuff? What other toolchains could you have used, and why didn't you? Did real time problems cause trouble? Did you badly estimate the scope of the system! and if so, why?

 Never forget to look at the skills of all of the players. Did a new language no one really understand create problems? Perhaps new hires just didn't understand the company's technology.

 The goal is not to find failure, but to find answers. Successes are every bit as important to understand, so you can capitalize on them next time.

 No one person is smart enough to find solutions to all problems. The document should be input to a brainstorming meeting where your colleagues hash out better ways to perform next time. Feed these ideas, where appropriate, back into the document.

 The only bad postmortem is one that's not honest and thoughtful. At my company we assess ourselves without beating each other up - no matter how badly things went. I am, however, intolerant of flippant, whiny or unreflective postmortems. If a team member is unable or unwilling to look for ways to improve the organization, especially in this non-threatening context, then that person is simply not suited to a career in this fast-changing industry. At least not with me.

 Now, there's nothing wrong with whining (I prefer to call it "venting"). Everyone deserves a chance to say their peace, even when sometimes it's totally irrational. We're all humans, after all, and are far from the Vulcan ideal - even engineers. I think one role of the supervisor is to provide a forum for this venting. If you let people come into your office and spout off, with both of you knowing it's only a rant, you'll all be healthier and happier. It just has no place in the postmortem. 

Data

 A postmortem without specific quantifiable data is a waste of time. "Well, we ran somewhat late and were over budget" is useless information. "We finished early and saved a ton of money" is just as bad. You can't take action, or learn things, without knowing the specifics of the situation.

 But our memories are notoriously unreliable. During a six month project lots of things happen, good and bad. Many dates might be missed and many met. By the time you're analyzing the results of the project, there's no way you'll remember - accurately - even a few of these.

 There's no substitute for setting detailed action items and recording the results. When starting a new development effort identify who will do what when. Set up a database of these action items and religiously record the results.

 I highly recommend an article by T. J. Rodgers, president of Cypress Semiconductor Corporation, in the July-August 1990 Harvard Business Review entitled "No Excuses Management" (page 84). In it he describes how Cypress monitors performance of all projects. While his systems are borderline anal-retentive, the concept is not. Get people to commit to goals. Have them monitor themselves, and post results to a common database, where supervisors can track performance.

 Preserve the data, so during the postmortem you'll have the accurate needed for producing useful recommendations. 

Too many people feel that college is the end of education. It's just the start. We've all got to struggle forever to learn more and to improve. Reading, studying, seminars, trade shows are all important ingredients. Equally important is self- and organization examination, looking for good things to emulate and bad things to fix.

Free Embedded Muse newsletter - twice/monthly, hard-hitting technical info that goes to 48,000+ engineers. Click here to sign up or enter your email here: