Lies, Damn Lies, and Schedules

Scheduling is totally dysfunctional. Here's why.

For hints, tricks and ideas about better ways to build embedded systems, subscribe to The Embedded Muse, a free biweekly e-newsletter. No hype, just down to earth embedded talk. 23,000 other engineers subscribe. It takes just a few seconds (all we need is your email address, which is shared with absolutely no one) to subscribe to the Embedded Muse.

You're wrong. Worse, you're lying.

 The vast majority of embedded projects run late, sometimes due to truly uncontrollable factors, but all too often for venial reasons. Most fail before the first line of code hits a file, before the first IC is dropped onto the CAD screen, before even a clear definition of the project exists.

 We fail at delivering embedded systems on-time when the schedule is propped up by prayer and hopeful wishes. That oh-so-familiar sick feeling in the pit of the stomach when looking at the latest schedule is a sure sign that we know prompt delivery is impossible. We kid ourselves that with enough long weekends and development heroics somehow, just maybe, we'll bail the project out. Statistics show, though, that despite the insane hours the projects are generally doomed to run late.

 All too often we bring this on ourselves. Remember all those times as a teenager, agreeing to be home by 10PM when you just know the party won't even get started till that time? We deferred the pain now for midnight trials! or worse, hoped the old folks would be asleep and wouldn't notice our late arrival. 

Nothing has changed. When we agree to a schedule  imposed from on-high, one we just know is impossible, we're repeating the same patterns of our childhood, and we know, deep in our souls, that the same result is inevitable.

 We're lying to ourselves and to the boss.

 I had the fascinating opportunity at this year's Embedded Systems Conference to argue this point with 150 developers, many of whom felt trapped into a dysfunctional cycle of being held to delivery times that made no sense. A few of the participants had the luxury of schedules that were respected by the boss, and that didn't become subject to capricious revisionism. Most of the rest felt coerced into timetables not of their own devising that were selected for marketing reasons rather than grounded in reality. 

Yet I maintain that yielding to these pressures is nothing short of lying. To yourself and to the boss. We do it in frustration, feeling there's no other option, as, after all, "he is the boss". But surely there must be a more honest way of going about this business we're in!

Self-Deceit

Let's admit - or at least examine - the fact that a lot of scheduling issues come from our own inadequate estimating skills.

 There are several kinds of estimation problems. The first comes from the those truly unexpected events or unknown factors that are way beyond our control. The CPU is suddenly pulled from the market. Maybe there's still some research left in our R&D project (like, the algorithm just does not work accurately enough, and we've got to invent another approach). Perhaps a new technology we're using, or one new to us (DSP, fuzzy, etc.), just turns out to be more rife with problems than we expected.

 These are risk factors. There's a lot we can do to mitigate risks, and to allow for a certain amount of possible (probable) problems rising from the "unexpected", as the unexpected is always to be expected to some extent. A wise project manager does indeed look at all aspects of each project and tries to assess where things could go wrong. Clearly, any new technology, from processors to languages to tools to algorithms, are suspect areas. Anything new to the development team, even if it's an old proven technology (RTOSes, for example) demands a learning curve that steals time from the project. Borderline science (like, "can we really measure moisture content using Tarot cards?") will almost certainly create schedule havoc - maybe a lot. It's pretty easy to identify these factors and to add some sort of cushion for their effects.

 Beware, though, of the project loaded with risk factors. New CPU not quite yet ready, brand new tool chain that should be working "real soon now", algorithms that might be right (we hope!), and under-trained staff. Too many risk variables spells disaster.

 Sometimes, though, the perversity of nature is the only constant we can depend on, and things go haywire in surprising ways. The chief developer dies in a car accident. Thieves steal all of our computers (don't laugh - it happened to me). A lawsuit makes a critical technology suddenly unavailable. Few bosses will permit schedule padding to deal with these possibilities; so slippage is therefore unavoidable. At least, though, there's no doubt that the problem comes for extraordinary reasons.

 Of course, if your firm is one where the extraordinary is ordinary, then find and fix the pattern of failure! or leave.

 A second kind of estimation problem comes from poorly educated managers, or worse, from those who expect miracles from subordinates. In the 50 years people have been programming electronic computers we've learned one fact above all: without a clear project specification any schedule estimate is nothing more than a stab in the dark. Yet every day dozens of projects start with little more definition than "well, build a new instrument kind of like the last one, with more features, cheaper, and smaller." Any estimate made to a vague spec is totally without value.

 The corollary is that given the clear spec, we need time - sometimes lots of time - to develop an accurate schedule. It ain't easy to translate a spec into a design, and then to realistically size the project. You simply cannot do justice to an estimate in two days, yet that's often all we get.

 Further, managers must accept schedule estimates made by their people. Sure, there's plenty of room for negotiation: reduce features, add resources, or permit more bugs (gasp!). Yet, in a survey of developers I conducted at the Conference, all too many replied that their schedule estimates were arbitrarily changed by management to reflect a desired end date, with no corresponding adjustments made to the project's scope.

 The result is almost comical to watch, in a perverse way. Developers drown themselves in project management software, mousing milestone triangles back and forth to meet an arbitrary date cast in stone by management. The final printout may look encouraging but generally gets the total lack of respect it deserves from the people doing the actual work. The schedule is then nothing more than dishonesty codified as policy. Scary stuff.

 Finally, there's an insidious sort of dishonest estimation too many of us engage in. It's easy to blame the boss for schedule debacles, yet in too many cases we bear plenty of responsibility. We get lazy, and don't invest the same amount of thought, time and energy into scheduling that we give to debugging. "Yeah, that section's kind of like something I did once before" is, at best, just a start of estimation. You cannot derive time, cost or size from such a vague statement! yet too many of us do. "Gee, that looks pretty easy - say a week" is a variant on this theme.

 Fact: scheduling is harder than coding and debugging. It requires plenty of deep thought, and plenty of time. 

Yet, even with the best of intentions it's easy to succumb to management's desire for a date and project cost today, now, when we know that more time is needed.

 Doing anything less than a thoughtful, thorough job of estimation is a form of self deceit, that rapidly turns into an institutionalized lie. "We'll ship December 1" we chant, while the estimators know just how flimsy the framework of that belief is. Marketing prepares glossy brochures, tech pubs write a manual, production orders parts. December 1 rolls around, and, surprise! January, February and March go by in a blur. Eventually the product goes out the door, with everyone exhausted and angry. All too much of this stems from a lousy job done in the first week of the project when we didn't do a reasonable job of estimating it's complexity.

 It's time to stop the madness! 

Honest is Not Enough

Even the best shops, those that practice the most careful forms of estimation all too often come up with schedules that are still grounded in little more than hope.

 We learn in school to practice top-down decomposition. Design the system, break each block into smaller chunks, and iterate until no part of the code is more than a page or two long. Then, and only then, can you understand its complexity. We generally then take a reasonable guess: "this module will be 50 lines of code". Instead of lines of code, some companies use function points or other units of measure.

 Cool. Swell. Do this and you will still almost certainly fail.

 Few developers understand that knowing code size - even if it were 100% accurate - is only half of the data absolutely required to produce any kind of schedule.

 It's amazing that somehow we all manage to solve the following equation:

     development time = (program size in units of X) * (time per X)

 when X is totally unknown.

 If you estimate modules in terms of lines of code (LOC), then you must know - exactly - the cost per line of code. Ditto for function points or any other unit of measure. Guesses are not useful.

 When I sing this song to developers the response is always "yeah, sure but I don't have LOC data! what do I do about the project I'm on today?" There's only one answer: sorry, pal - you're outa luck. IBM's LOC/month number is useless to you, as is one from the FAA, DOD or any other organization. In the commercial world we all hold our code to different standards, which greatly skews productivity in any particular measure.

 You simply must measure how fast you generate embedded code. Every single day, for the rest of your life. It's like being on a diet - even when everything's perfect, you've shed those 20 extra pounds, you'll forever be monitoring your weight to stay in the desired range. Start collecting the data today, do it forever, and over time you'll find a model of your productivity that will greatly improve your estimation accuracy. Don't do it, and every estimate you make will be, in effect, a lie, a wild, meaningless guess.

The Arbitrary Boss

When management distorts or destroys your careful estimate, or beats you into agreeing to one that cannot possibly happen, failure is certain. Period. Yet this practice is the norm. 

Bosses complain that the engineers pad their estimates so much that there simply must be fat. They feel justified in whacking off a month or two. Or, there are those who feel an aggressive schedule inspires harder work, possibly true, but only when "aggressive" is not confused with "impossible".

 My feeling that if there's no mutual trust between workers and management, the employment situation is dysfunctional and should be terminated. Professionals - us! - are paid for doing the work and for making reasonable technical recommendations. We may be wrong sometimes, but a healthy work environment recognizes the strengths and weakness of each professional. If your boss thinks you're an idiot, or refuses to trust yes">  your judgment, search the employment ads.

 Too many bosses have little or no experience in managing software projects. The news they get is invariably bad - the project will take six months longer than hoped - yet it generally comes with no options, no decisions that he can make to achieve the sort of balance between product and delivery.

 It's critical we learn to manage our bosses. When presenting bad news, be sure you give options. "We can deliver on time but without these features, or 6 months late with everything, or on-time but with lots of bugs!" An intelligent analysis of choices, presented clearly, will help get your message across.

 We need to develop trust with our superiors by educating them about development issues, by being right (meeting our own predictions), and by communicating clearly.

 We've got to avid our knee-jerk reaction to any change request by quoting a long, arbitrary, time impact.

 Just as it takes time and many projects to get the data you need to be an accurate estimator, educating the boss and creating trust can be a very slow process. So slow, in fact, that you must remember that sooner or later the boss will die or move on! and you'll be in charge. Then remember. Treat your people with trust and respect, and teach them what you've learned about scheduling.

 Evolution is a great thing. Perhaps the firmware industry will mature as new generations of people learn to do things correctly, and then slowly replace the dinosaurs now all too often at the top.

Conclusion

"Lying" is a strong word, one I've used to shock rather than incriminate. One friend prefers to call it "truth management". We do need to rethink the estimation process, and I feel that absolute honesty is the only standard worth using. Scheduling and estimation is a form of promise-making, and our ability to make promises we'll keep is a measure of who we are. It's a two way street, from management to developers and the reverse. 

An old phrase coined about the clerks and small fry of the Third Reich captures the puzzling ability of humans to make small compromises that add up to ultimate horror: "the banality of evil." Individuals cheat on taxes and, in a small way, thus cheat their neighbors. Maybe some scam an unemployment check or two. As individual developers we too often participate in the small truth managements that generate a disaster, admittedly not a huge one, but one that is avoidable and that detracts from peoples' lives.

 Remember: "We can dispense with reality and promise delivery, and wish the project complete on time."

 Or, we can do it right.