For novel ideas about building embedded systems (both hardware and firmware), join the 35,000 engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype and no vendor PR. Click here to subscribe.
By Jack Ganssle
Published January, 2003 in Embedded Systems Programming.
You'll deliver that system you should be working on now - rather than reading a magazine - late. Almost certainly. Managers fume while developers regularly employ heroics in futile attempts to bring projects back on schedule. Despite massive overtime and exhausted engineers, late projects continue to be the norm. We repeatedly promise to meet schedules! yet never do.
Nearly all embedded systems ship with defects, some minor, too many with major flaws that show up only after shipment, resulting in unhappy or injured customers. We make implicit promises to deliver "perfect" code, yet that goal, too, remains elusive.
I looked at a system last Spring where management was shocked to discover the delivered product had less than half the required features implemented. That's not uncommon; we continue to deliver products without all of the promised functionality.
I believe part of the problem with late, buggy and feature-laden firmware stems from our casual attitude to promise-making. "Yeah, sure, it'll be done next week," when we really know that months of work remain to get the thing done right.
We either promise the impossible or acquiesce to promises made on our behalf by management. Both situations are dysfunctional behaviors that don't lead to our goal of a great product finished on time. Both are a form of lying, an activity that poisons relationships and engineering teams.
So here are a few rules of promise-making. Neglect them at your peril!
1. Keep Them
A justice of the peace recently told me she advises marriage-bound couples to minimize their vows. "Don't promise too much," she advised, "marriage is hard; promise just a little - only what's really important. Then shut up and then keep those commitments." Interesting advice; over the years I've heard too many sweeping pledges that crumble in the barrage of real life.
In marriage and in development, never make a promise that you cannot keep, no matter how much pressure you feel, no matter how critical the promise may seem to the company or a customer.
Robert Service, in the delightful poem The Cremation of Sam MGee, wrote "a promise made is a debt unpaid". Good advice for both artic explorers and embedded developers.
Salesmen love to promise the world. Engineering always groans at the latest impossible feature that somehow got shoved into the spec to satisfy a customer's perceived need. These promises morph simple gadgets into insanely complex devices that can't be made and that probably no one really wants anyway. If you spend time in the field with one of these snake-oil pushers it's pretty obvious how this happens. The sales people feel enormous pressures: from the company president to flog more product, from the bank account to earn commissions, and from their own gigantic egos that equate sales success with a sense of personal worth.
Faced with a balking customer they cave to these forces and swear anything to get a signed contract. Somehow engineering will make it all fine in the future. That's tomorrow's problem, but today I got the sale!
If you have kids you know how they'll hide a report card to put off the day of reckoning. Did it myself when young. Like the sales folks, they turn today's problem into tomorrow's.
We act exactly the same when we make an impossible promise. Are you figuring on lots of OT, quite a bit of hope, and more than usual luck to make the delivery on-schedule? Experience shows you'll likely fail, and in the meantime turn into a stress puppy, alienate your family, and infuriate the boss.
If there's an upside to these impossible promises I can't find it.
2. Put it in Writing
When you do make a promise, put it in writing. Now. Immediately. Wait, and you'll forget to jot it down. Someone will come back weeks in the future holding you to a statement that, by then, you're not sure if you made or not! or at least not in that way.
Many companies have memo pads with the warning "avoid verbal orders!" In law and in life a verbal contract - a promise - is just as binding as one wrapped in a hundred pages of flowery legal prose. When I hear someone say they can't commit to a date (even for a casual dinner engagement or help with a charitable project) without checking their calendar and other commitments, I know I'm dealing with a true professional. Sure, an immediate "yes" would be nice, but a delayed but thoughtful "yes" likely means that person will indeed come through.
When the boss asks you over the water cooler for a rough guess at a schedule, be wary. Don't bind yourself unless you're quite sure you can meet that commitment. And then write it down. If you're truly just tossing ideas around, make it clear that until he sees it in a memo it's just speculation.
The boss/employee relationship is really mixed up in most companies. Developers have this sense that the boss is some sort of demigod to be feared and maybe even worshiped. "Yes, massa" seems to be the expected response to any request from the upper levels.
But it ain't so. Bosses and employees, especially smart ones like us, should have a dynamic two-way peer relationship. The boss manages us, and, if we're wise, we manage him (or her). Practice daily boss management; help him be more successful as he's doing the same for you. There are a lot of aspects to boss management, but in the context of this discussion realize that you must train your supervisor to expect certain behaviors from you.
Through constant and unflagging repetition show your boss how you handle promises. You make them in writing. Period. When he claims you committed to X but there's no paperwork to back it up, he's wrong. Period.
Some wags suggest hanging a sign on your desk that says
I'll present my Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. The early registration discount ends January 20.
This means you must not slip up - ever. Put every promise in writing. Show the boss how this practice is good for the both of you: you never forget a guarantee, and he gets a clear and concrete vision of what you are doing. There's no uncertainty and no confusion, both enemies of effective management.
A poll at a recent Embedded Systems Conference suggested that 46% of embedded people get either no spec or one that's pathetically devoid of detail. That means we're undertaking huge development efforts with no clear view of the goal. We're making promises we don't even understand. Nothing is in writing. Is this rational or even ethical behavior?
3. Give Margins of Error
Add a deviation range to all written promises. How much error will this undertaking likely experience? Might forces outside your control alter things?
Bosses expect a date cast in stone. That's not unreasonable if there's so much padding in the schedule that the only uncertainty is how to kill all of the extra time. But an aggressive date without any margin for error cannot be met. 50 years of software engineering has taught us the folly of such promises. Yet despite this history bosses still expect firm, fixed dates. Meanwhile we miss more ship dates than we hit.
We must understand that traditional business practices do mandate such scheduling. A firm fixed price contract requires spending resources exactly to the plan. A satellite launch date set by planetary geometry can't be renegotiated. When the date is truly and absolutely fixed then we must have some other flexibility in the development process to give us the room to maneuver when (not if) problems arise.
First, there must be a comprehensive and unchanging spec. Yet we know these are rare. Most modern agile development methods are predicated on the observation that the spec changes all of the time. We need a more adaptive strategy than that offered by Big Up Front Design. But if the schedule is truly fixed aim for as much spec as you can get.
Second, realize that development is a dance, a stochastic process rather than a smooth cruise to a delivered system. If we can't avoid time-consuming unexpected problems, be prepared to go to plan B! or C or D. For example, keep the system so simple there's no fear of slipping. Or be willing to ship on-time by removing cool but non-mission-critical features. Try not to short-change project tools or resources. If possible use contractors working on subsystems in parallel with the main effort. But find and exploit other options.
Third, when the date is set more by marketing concerns than cosmology, give promises encapsulated in bounds. "We'll deliver in 5 months plus 3 minus 1" is a lot more honest than the usual "sure, 5 months, whatever."
I suspect the next great area in software engineering will be resolving the disconnect between modern development methods and traditional management techniques. Spiral development, Evolutionary methods, XP, SCRUM, and a whole host of approaches all deliver incrementally. Make some subset of the system work, then iterate, re-evaluating the schedule at each delivery. That eminently sensible approach recognizes the realities of software and meets the needs of programmers, but leaves management quite at sea. The bosses have good reasons for requiring dates. Our methods are woefully inadequate at meeting this need.
Did you promise zero defects? That's unreasonable, if only because no one knows how to measure such a standard of perfection. Better: "passes our entire test suite without problems". That bounds the problem, eliminating open-ended and impossible promises.
4. Eliminate Assumptions
If you find someone assuming you made a promise, immediately correct that assumption. Refer them back to your written promises.
Promises are like rumors. They grow with the retelling. Engineering tells marketing 6 months, marketing suggests to sales 5, and the customer finally hears 4. Management decides this will be the best desktop OS ever, sales says it'll be almost defect-free, and the dealers hear about a Linux-killer that can't be hacked.
A developer wrote that he was fired because the customer was appalled at the lack of features provided with his company's initial release of a new product. An early department-wide meeting that included sales and engineering resulted in slashing those features. Somehow sales forgot to tell the customer. With neither meeting minutes nor follow-up memos engineering took the fall and this engineer lost his job. Appalling but true.
A technically-unsavy friend recently bought radar for his sailboat. He called me over to look at the "malfunctioning" unit. Everything looked fine to me, till Tom mentioned that the salesman promised that he'd see ships on the screen. "All I see are those splotches; I thought there would be pictures of ships", he complained. The sales guy promised it would show ships, but this customer assumed something very different.
Be ruthless in destroying unfounded assumptions.
5. Fess Up
Stuff happens; life is tough, there are no guarantees. Do everything in your power to meet your promises, but when that becomes impossible admit it early.
When I ran engineering groups the employee behaviors that were most infuriating were to give me bad news late, and to give bad news without options.
Things do go wrong, problems surface, and unexpected difficulties are inevitable in life. Another part of boss management is keeping him informed of progress and roadblocks. When it's clear disaster lurks, bring the boss into the picture. Wait too long and you'll all be faced with fewer recovery choices.
Understand that in most cases your supervisor knows less than you do about the subject at hand. That doesn't make him a dimwit. Proper roles are for you to do the work and him to clear a path for you. Don't expect him to ride to the rescue; it's up to you to look for options. Try to present both the problem and a range of proposed solutions. "This thing won't run as fast as we hoped, but if we trim these features, or eliminate wait states, or add a co-processor, we can recover." Even if none of the solutions are particularly palatable, discussing them opens the doors for effective brainstorming.
Presenting disaster unaccompanied by alternatives invites emotional responses of frustration, anger, and hopelessness.
Promises are the coin of our business; we trade promises of future product completion for ongoing salary and benefits. Make them lightly and you will pay a bitter price.
Have you ever been frustrated by a neighbor who doesn't follow through on promises made about coaching little league or similar activities? The rule of thumb is to find a busy person when you need something done. Busy people know how to say "no" when they can't manage a commitment, and how to insure they meet their promises. It seems to me that busy people are effective because they've practiced proactive promise-making, and know how to follow through on those commitments.
. That's a noble ideal for us all to try for.