Memo To My Boss
||For novel ideas about building embedded systems (both hardware and firmware), join the 25,000+ engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype, no vendor PR. It takes just a few seconds (just enter your email, which is shared with absolutely no one) to subscribe.
By Jack Ganssle
Published in Embedded Systems Programming, November, 2001
Bob Smith, CEO
From: Jake Schmidt
I wanted to respond to your memo of the 16th.
Circulated as it was to seemingly half the company I feel we developers are now
operating under an insurmountable stigma. Yes, we all know the product shipped
late. Very late. You want to know why the schedule was missed so badly, and want
an action plan to assure this problem will never reoccur.
I was just one of many developers on the project. Others
on the team will probably respond with apologetic platitudes. However,
you'll note my resignation is attached so I feel no pressure to paper over the
very real problems with politically-correct but worthless suggestions.
Let me assure you, that, contrary to your strongly-stated
opinion, we were not "spending half our time surfing the net for porn instead
of cranking code", nor was the team composed of "laggards and slackers."
Therein lies the first problem: engineering was treated with contempt and
disrespect throughout the task. Each of us are college-educated, highly trained
engineers with years of experience. Yet we're viewed as cogs in the system,
replaceable elements whose sole mission is to respond to marketing's latest
The timecards show we averaged 70 hours a week valiantly
trying to tame the beast. So maybe we did check personal email once in a while.
Big deal. Yet suspicion reigned: managers, including you, prowled the lab
looking for evidence of slacking. When you didn't find us playing computer
solitaire you restricted net access to avoid "extraneous" emails and blocked
all non-corporate web sites. Professionals cannot work in an environment
W. Edwards Deming, the great quality guru, long ago
examined factors influencing motivation. He found professionals get most of
their drive from "intrinsic" motivating factors, those that come from within
ourselves. Things like feeling part of the team. The desire for the
organization's success. Wanting to deliver a great product. Intolerance of
He described "extrinsic" motivators as those imposed
from on-high, usually without any sort of buy-in from the professionals
involved. Artificial measurements ranked high on his list of extrinsic
motivators. Deming showed that extrinsic motivators invariably drive out the
intrinsic ones. In other words, people shrug in frustration and work to meet the
numbers, rather than try to make things right.
A prime example: your imposition of lines of code per day
metrics. You wanted lots and lots of software, and we gave it to you. Oh, man,
we cranked code! Most of it sucked, of course, and the resulting clean-up is
Joe's suicide and the 45% turnover all stemmed from the
lack of control and the frustration we felt. Which leads to the next point.
Turnover was a huge problem, but almost worse was the
Company's unwillingness to train replacements. The shortage of decent embedded
people meant we relied on headhunters and huge signing bonuses (we called these
"newbie bait"). Hiring costs exceeded $50k per position, which is actually
not out of line in this industry. Yet each new person was immediately
demotivated by being tossed into the fray without even a second's worth of
We largely succeeded in getting the best and the brightest.
At 8:00 each reported for the first day of work; by 9 they were expected to be
productive. None knew our software standards (more later) or our process (more
later) or our technology. The resulting anger and resentment formed a beautiful
feedback loop back into the turnover problem.
Our $50k-per-person expenditures were for naught, wasted.
Like a house if cards, though, the turnover and lack of
training were yet just another bit of fuel in the impending disaster. Your
obtuse decision to create the code in C++ helped us build the bloated monster no
one comprehends. I'll quote Joe's suicide e-note: "What kind of an idiot
makes fundamental technical decisions based on one Business Week article?"
Yes, the magazine correctly reported that object oriented
programming holds the promise of aiding in the creation of reuseable software
components, and, yes, that is indeed a very good thing. And C++ is certainly an
appropriate language for OOP.
Remember Billy, the new college grad? The one whose fiance arrived late one night waving a steak knife, demanding he gets some at-home
time? She's better, by the way, and should be free in a few months. Well,
Billy was the only one here with any OO experience. With the tight schedule we
were somehow supposed to master OO, which is a whole new approach to design and
coding, as we created the project. That's simply impossible.
People learn new design approaches like OO by building
small projects and tossing them away. Instead, we were told to turn our first
experience with this very different concept into a shipping product. Where's
the chance for the mistakes and missteps that are a natural part of learning?
Joe had requested a month of C++ training for all of the
developers before we started. You axed that, citing schedule pressure. Now
we're a year late and the system still doesn't work. Dare I suggest that the
training month would have leveraged the schedule in wonderful ways?
So there's another dysfunctional feedback loop: your lack
of respect for us causes you to disregard our expert recommendations (e.g., for
training) which makes us late and lowers your respect for us.
We developers were also always puzzled by the surreal
management structure which seemed to make all of us report to those 23 year old
marketing whizzes in their fancy suits who kept making astonishing promises to
the customers. (I know the DOD is an important customer, but are we still
telling them we'll cram Linux into an 8051?) Every whim from these geniuses
resulted in a mandatory new feature. It wasn't unusual to see them standing
over the shoulders of a developer suggesting - nay, demanding - changes
this instant. We were forced to implement these in an ad hoc manner, with no
corresponding schedule changes.
Sales wasn't much better. Why is it they phrase all
requests with the disclaimer "I'll never be able to sell that piece of junk
unless you add..."?
All of us acknowledge that changes are a part of life. We
know that it's awfully hard to fully specify a complex system, so some amount
of system evolution is inevitable. But there's always a cost: adding features
makes the project later. Period. Since change is unavoidable, important, and
expensive we've got to manage it. I suggest
instituting standard change control procedures. This means that each
request is approved by the project manager after we've looked at its impact,
both on schedule/cost and system implications. This would minimize embarrassing
moments, like when marketing browbeat the apps guys into adding an email
client, yet the system never had, nor ever will have, an Internet connection.
As it was the list of features grew faster than a lottery
winner's circle of friends. So fast, in fact, that no one to this day knows
what all of the requirements really are! A more disciplined approach is to
manage the features and prioritize their implementation. Instead of having
hundreds of partially working "cool must-haves" we'd get the essentials
all working properly, and only then move on to those less important. We could
have shipped a bug-free subset of the product long ago, adding upgrades -
correct, working upgrades - regularly.
Lest you think this is merely an anti-management polemic,
I'd suggest another cause of trouble was too little oversight. Twenty
developers reported to dearly departed Joe, without intermediate supervisors. No
one can manage that many people effectively.
Joe, for all of his
virtues, was at heart and in practice an EE who'd rather be building analog
circuits than running a big firmware project. Our previous embedded versions of
the system succeeded because they were small. Dramatic overtime is enough to get
10,000 lines of C running; this is simply not the case when dealing with 250k
lines. I suspect development by heroics appeals to management because the
engineers seem so obviously dedicated, but the process just does not scale.
Yet Joe knew how to listen. We convinced him of the
importance of creating some sort of development process. It was pretty
lightweight, actually - a clear
coding standard, rigorous code
inspections, and a version control system.
Remember the pre-VCS days? Like when we shipped five
different but defective versions of the code to that one customer? No one knew
how to get a correct build. Or the time those two ZZ-Top look-alikes lost it,
smashed their keyboards on the PCs, and then rode their Hogs right through the
cubicle walls, destroying the server? A VCS would have made backups easier, so
all that data wouldn't have been lost. I can't believe you hired them back
One success was that, once implemented, our VCS worked
We tried to institute firmware standards, to insure code
was clean and consistent. It failed due to lack of accountability, a critical
element of managing anything. In software engineering one important way to
insure things are done right is to review work products. Like the code. In the
absence of these reviews, there is no accountability other than "well, it
seems to work." So when Joe was busy dealing with the problems of 20
developers he was unable to herd his flock towards doing things right all of the
time. Developers, driven to panic by growing product size and your
pronouncements of doom if we slipped, dropped the firmware standards and the
I'd be the last to suggest nepotism, but when you
replaced Joe after his sad demise with your ex-auto-mechanic cousin Vinny, all
hopes of instituting a development process crashed. Vinny's favorite saying
was "yo, this cain't be any harder than rebuilding the carb on a 67 Impala.
What's wrong with youse guys?" Then he'd stand behind the nervous engineer
swinging a crescent wrench. In a desperate bid to look busy, of course the
developers dropped any pretense at maintaining the process. Vinny wanted to see
us crank code.
We needed a coach, someone to help us do the right things
every day, rather than a recently paroled felon whose admittedly profound
jailhouse skills didn't port well to the lab.
So, with neither standards nor inspections the system grew
creaky and ponderous. Bugs surfaced with alarming frequency, the rate
accelerating as time went on. An intriguing studies by Caper Jones showed that,
in a survey of 4000 projects, the number one cause of slipped dates was poor
quality. Bugs. Sound familiar?
The company's attitude to debugging tools appalls me.
Telling us "instead of spending $10,000 on debuggers, just stop making bugs"
was naive. Study after study shows that even with the best development
processes, we're going to have to hammer out some bugs. Even decent code
inspections - had we used them - would eliminate only 70% of the defects.
Traditional debugging is a part of life.
Our choice is simple: buy decent tools or bloat the
schedule. A lot. Remember the $50k per engineer hiring costs? Tools are cheap by
As you seem to have forgotten, the delivery date was
capriciously assigned by you. Rather than developing one from a list of
requirements followed by a careful design, you told us the ship date was to be
June 15. To our shame we didn't walk out en masse, but spent three days
nervously moving little triangles around in the project management software to
make a schedule that looked believable, all the while knowing each milestone was
a sham, the totality a blatant lie.
And so, I close with the suggestion that you replace Vinny
with a strong, highly ethical, and experienced manager of software engineering.
Someone able to stand up to impossible demands from above, and impose and
nurture a coherent development ethic from below.
From: Bob Smith
With marketing making demands for the new version of the
product and sales complaining they can't sell the old one, I'm a bit too
busy to read your most recent memo, though I'm sure it has much useful
information. I'll get to it soon, no doubt.
However, I've promised the customer we can deliver
another new upgrade in three months. Please go ahead and create an appropriate
schedule and get started. We'll work out the features and specs later.
Business Week suggests that Java will let us run on many
different platforms. So write it all in Java. The magazine said something about
garbage collection, whatever that is, but I figure if you don't make any
garbage, there won't be any problems. Why can't you engineers think of these
solutions? Plan on porting this code to that old 8051 design after you get Linux
running on it.
I've been thinking about how to keep this new project
on-schedule. From now on, all of you developers will eat lunch at your desks and
keep working. That, and some more overtime, should do the trick.