Memo To My Boss

For hints, tricks and ideas about better ways to build embedded systems, subscribe to The Embedded Muse, a free biweekly e-newsletter. No hype, just down to earth embedded talk. 23,000 other engineers subscribe. It takes just a few seconds (all we need is your email address, which is shared with absolutely no one) to subscribe to the Embedded Muse.

MEMO

To:       Bob Smith, CEO
From:   Jake Schmidt
Attch:   resig.doc 

Dear Bob,

I wanted to respond to your memo of the 16^th. Circulated as it was to seemingly half the company I feel we developers are now operating under an insurmountable stigma. Yes, we all know the product shipped late. Very late. You want to know why the schedule was missed so badly, and want an action plan to assure this problem will never reoccur.

I was just one of many developers on the project. Others  on the team will probably respond with apologetic platitudes. However, you'll note my resignation is attached so I feel no pressure to paper over the very real problems with politically-correct but worthless suggestions.

Let me assure you, that, contrary to your strongly-stated opinion, we were not "spending half our time surfing the net for porn instead of cranking code", nor was the team composed of "laggards and slackers."  Therein lies the first problem: engineering was treated with contempt and disrespect throughout the task. Each of us are college-educated, highly trained engineers with years of experience. Yet we're viewed as cogs in the system, replaceable elements whose sole mission is to respond to marketing's latest harebrained wish-list.

The timecards show we averaged 70 hours a week valiantly trying to tame the beast. So maybe we did check personal email once in a while. Big deal. Yet suspicion reigned: managers, including you, prowled the lab looking for evidence of slacking. When you didn't find us playing computer solitaire you restricted net access to avoid "extraneous" emails and blocked all non-corporate web sites. Professionals cannot work in an environment lacking trust.

W. Edwards Deming, the great quality guru, long ago examined factors influencing motivation. He found professionals get most of their drive from "intrinsic" motivating factors, those that come from within ourselves. Things like feeling part of the team. The desire for the organization's success. Wanting to deliver a great product. Intolerance of poor quality.

He described "extrinsic" motivators as those imposed from on-high, usually without any sort of buy-in from the professionals involved. Artificial measurements ranked high on his list of extrinsic motivators. Deming showed that extrinsic motivators invariably drive out the intrinsic ones. In other words, people shrug in frustration and work to meet the numbers, rather than try to make things right.

A prime example: your imposition of lines of code per day metrics. You wanted lots and lots of software, and we gave it to you. Oh, man, we cranked code! Most of it sucked, of course, and the resulting clean-up is still incomplete.

Joe's suicide and the 45% turnover all stemmed from the lack of control and the frustration we felt. Which leads to the next point.

Turnover was a huge problem, but almost worse was the Company's unwillingness to train replacements. The shortage of decent embedded people meant we relied on headhunters and huge signing bonuses (we called these "newbie bait"). Hiring costs exceeded $50k per position, which is actually not out of line in this industry. Yet each new person was immediately demotivated by being tossed into the fray without even a second's worth of training.

We largely succeeded in getting the best and the brightest. At 8:00 each reported for the first day of work; by 9 they were expected to be productive. None knew our software standards (more later) or our process (more later) or our technology. The resulting anger and resentment formed a beautiful feedback loop back into the turnover problem.

Our $50k-per-person expenditures were for naught, wasted.

Like a house if cards, though, the turnover and lack of training were yet just another bit of fuel in the impending disaster. Your obtuse decision to create the code in C++ helped us build the bloated monster no one comprehends. I'll quote Joe's suicide e-note: "What kind of an idiot makes fundamental technical decisions based on one Business Week article?"

Yes, the magazine correctly reported that object oriented programming holds the promise of aiding in the creation of reuseable software components, and, yes, that is indeed a very good thing. And C++ is certainly an appropriate language for OOP.

Remember Billy, the new college grad? The one whose fiance arrived late one night waving a steak knife, demanding he gets some at-home time? She's better, by the way, and should be free in a few months. Well, Billy was the only one here with any OO experience. With the tight schedule we were somehow supposed to master OO, which is a whole new approach to design and coding, as we created the project. That's simply impossible.

People learn new design approaches like OO by building small projects and tossing them away. Instead, we were told to turn our first experience with this very different concept into a shipping product. Where's the chance for the mistakes and missteps that are a natural part of learning?

Joe had requested a month of C++ training for all of the developers before we started. You axed that, citing schedule pressure. Now we're a year late and the system still doesn't work. Dare I suggest that the training month would have leveraged the schedule in wonderful ways?

So there's another dysfunctional feedback loop: your lack of respect for us causes you to disregard our expert recommendations (e.g., for training) which makes us late! and lowers your respect for us.

We developers were also always puzzled by the surreal management structure which seemed to make all of us report to those 23 year old marketing whizzes in their fancy suits who kept making astonishing promises to the customers. (I know the DOD is an important customer, but are we still telling them we'll cram Linux into an 8051?) Every whim from these geniuses resulted in a mandatory new feature. It wasn't unusual to see them standing over the shoulders of a developer suggesting - nay, demanding - changes this instant. We were forced to implement these in an ad hoc manner, with no corresponding schedule changes.

Sales wasn't much better. Why is it they phrase all requests with the disclaimer "I'll never be able to sell that piece of junk unless you add!"?

All of us acknowledge that changes are a part of life. We know that it's awfully hard to fully specify a complex system, so some amount of system evolution is inevitable. But there's always a cost: adding features makes the project later. Period. Since change is unavoidable, important, and expensive we've got to manage it. I suggest  instituting standard change control procedures. This means that each request is approved by the project manager after we've looked at its impact, both on schedule/cost and system implications. This would minimize embarrassing moments, like when marketing browbeat the apps guys into adding an email client! yet the system never had, nor ever will have, an Internet connection.

As it was the list of features grew faster than a lottery winner's circle of friends. So fast, in fact, that no one to this day knows what all of the requirements really are! A more disciplined approach is to manage the features and prioritize their implementation. Instead of having hundreds of partially working "cool must-haves" we'd get the essentials all working properly, and only then move on to those less important. We could have shipped a bug-free subset of the product long ago, adding upgrades - correct, working upgrades - regularly.

Lest you think this is merely an anti-management polemic, I'd suggest another cause of trouble was too little oversight. Twenty developers reported to dearly departed Joe, without intermediate supervisors. No one can manage that many people effectively.

Joe, for all of  his virtues, was at heart and in practice an EE who'd rather be building analog circuits than running a big firmware project. Our previous embedded versions of the system succeeded because they were small. Dramatic overtime is enough to get 10,000 lines of C running; this is simply not the case when dealing with 250k lines. I suspect development by heroics appeals to management because the engineers seem so obviously dedicated, but the process just does not scale.

Yet Joe knew how to listen. We convinced him of the importance of creating some sort of development process. It was pretty lightweight, actually -  a clear coding standard,  rigorous code inspections, and a version control system.

Remember the pre-VCS days? Like when we shipped five different but defective versions of the code to that one customer? No one knew how to get a correct build. Or the time those two ZZ-Top look-alikes lost it, smashed their keyboards on the PCs, and then rode their Hogs right through the cubicle walls, destroying the server? A VCS would have made backups easier, so all that data wouldn't have been lost. I can't believe you hired them back as consultants.

One success was that, once implemented, our VCS worked well.

We tried to institute firmware standards, to insure code was clean and consistent. It failed due to lack of accountability, a critical element of managing anything. In software engineering one important way to insure things are done right is to review work products. Like the code. In the absence of these reviews, there is no accountability other than "well, it seems to work." So when Joe was busy dealing with the problems of 20 developers he was unable to herd his flock towards doing things right all of the time. Developers, driven to panic by growing product size and your pronouncements of doom if we slipped, dropped the firmware standards and the reviews.

I'd be the last to suggest nepotism, but when you replaced Joe after his sad demise with your ex-auto-mechanic cousin Vinny, all hopes of instituting a development process crashed. Vinny's favorite saying was "yo, this cain't be any harder than rebuilding the carb on a 67 Impala. What's wrong with youse guys?" Then he'd stand behind the nervous engineer swinging a crescent wrench. In a desperate bid to look busy, of course the developers dropped any pretense at maintaining the process. Vinny wanted to see us crank code.

We needed a coach, someone to help us do the right things every day, rather than a recently paroled felon whose admittedly profound jailhouse skills didn't port well to the lab.

So, with neither standards nor inspections the system grew creaky and ponderous. Bugs surfaced with alarming frequency, the rate accelerating as time went on. An intriguing studies by Caper Jones showed that, in a survey of 4000 projects, the number one cause of slipped dates was poor quality. Bugs. Sound familiar?

The company's attitude to debugging tools appalls me. Telling us "instead of spending $10,000 on debuggers, just stop making bugs" was naive. Study after study shows that even with the best development processes, we're going to have to hammer out some bugs. Even decent code inspections - had we used them - would eliminate only 70% of the defects. Traditional debugging is a part of life.

Our choice is simple: buy decent tools or bloat the schedule. A lot. Remember the $50k per engineer hiring costs? Tools are cheap by comparison.

As you seem to have forgotten, the delivery date was capriciously assigned by you. Rather than developing one from a list of requirements followed by a careful design, you told us the ship date was to be June 15. To our shame we didn't walk out en masse, but spent three days nervously moving little triangles around in the project management software to make a schedule that looked believable, all the while knowing each milestone was a sham, the totality a blatant lie.

And so, I close with the suggestion that you replace Vinny with a strong, highly ethical, and experienced manager of software engineering. Someone able to stand up to impossible demands from above, and impose and nurture a coherent development ethic from below.

With disgust,

Jake

MEMO

To:       Jake Schmidt
From:   Bob Smith
Cc:       Vinny DeAlonzo

Dear Jake,

With marketing making demands for the new version of the product and sales complaining they can't sell the old one, I'm a bit too busy to read your most recent memo, though I'm sure it has much useful information. I'll get to it soon, no doubt.

However, I've promised the customer we can deliver another new upgrade in three months. Please go ahead and create an appropriate schedule and get started. We'll work out the features and specs later.

Business Week suggests that Java will let us run on many different platforms. So write it all in Java. The magazine said something about garbage collection, whatever that is, but I figure if you don't make any garbage, there won't be any problems. Why can't you engineers think of these solutions? Plan on porting this code to that old 8051 design after you get Linux running on it.

I've been thinking about how to keep this new project on-schedule. From now on, all of you developers will eat lunch at your desks and keep working. That, and some more overtime, should do the trick.

Sincerely,

Bob