Follow @jack_ganssle

The logo for The Embedded Muse For novel ideas about building embedded systems (both hardware and firmware), join the 25,000+ engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype, no vendor PR. It takes just a few seconds (just enter your email, which is shared with absolutely no one) to subscribe.

By Jack Ganssle

The Great Divide

I often visit Washington, DC's Holocaust Museum (http://www.ushmm.org). It's a bleak must-see stop when taking out-of-town friends on tours of the city. Even self-involved teenagers emerge sobered, momentarily distracted from their cell swarming by the hideous stories they've just heard. Few adults come out without reddened eyes. Books convey much of the horror, but the museum both teaches the truth and gives visitors a visceral feel for the concentration camps 60 years ago.

The museum offers far more than just the exhibits. Daytime and evening discussion groups, talks by survivors, and workshops help - somewhat - unveil the causes of the genocide. Though the grotesque evil perpetrated by Himmler et al defies comprehension, it's perhaps best to perceive these people as monsters, non-human creatures who blighted Homo sapiens for a time. Unhappily the cry "never again" seems less accurate than "too often again," as events in Rwanda, Cambodia and so many other places proclaim. It seems there is another species clothed in human-like form which lives amongst us, ready to commit unspeakable acts with little provocation.

Through the discussion groups and talks I try to understand the thinking of the little people, not the monsters, the clerks who stamped forms dooming entire villages. Or the guards herding Jews into freight cars, packed inhumanly tightly, bound for Auschwitz and Treblinka. Thousands, perhaps tens of thousands of soldiers and civilians were the machinery that enabled the monsters' fury. These folks, most of whom returned to normal life in post-war Germany, were an essential part of the Holocaust.

But were they evil? Were they morally as culpable as those who dropped tablets of Zyklon B into the gas chambers?

In my younger days of moral absolutism the answer was an easy "yes!" But the spectrum of behavior in Germany was very wide. Most of the country contributed in some way to the war effort, and thus indirectly to the Holocaust. In every country today taxpayers fund and enable the government's actions. If a citizen of some outlaw regime continues to pay taxes, is that person as culpable as the benighted leader?

Though Enron's leaders will submit to their own Nuremburg, surely lower-ranking middle managers had some inkling of the financial shenanigans. They'll go unprosecuted. Is this fair? Is their behavior evil?

These ethical dilemmas aren't obscure problems buried in the sands of time, but a constant part of life. We're frequently asked to take actions that may, perhaps only in some small way, compromise our beliefs. When purchasing got a better deal on 10% resistors, and we really need 5% in an application because the system just won't be reliable otherwise. do we let it slide? Even when getting the right parts will delay shipping, so the company posts a loss for the quarter?

Scheduling may be one of those areas where we're often forced to promise things we can't deliver. To lie, to breach one's ethics. Such a breach always has consequences - the product is late, customers irate, and there may be significant financial penalties as well. Like most of us, I try hard to behave ethically, but worry that when we acquiesce to management's impossible demands, or when we submit a plan we know is impossible, that we're in some small way emulating the behavior of those German clerks. Of course there's no parallel to the scope of the evil; but is the decision process similar?

I'm drawn to Methodist theologian Stanley Hauerwas' suggestion that we evaluate our ethical choices by asking "if I decide X, what kind of person does that make me?" So if I lie or cheat on the little things, say creating a schedule, I'm a liar and a cheat. And likely will behave the same way when wrestling with the big decisions.

Scheduling Madness

No part of any project is as contentious or as infuriating as scheduling - both for the engineers and for management. No one ever believes a software schedule, though people may delude themselves in planning using the flawed dates and numbers. "Let's see. since the project will be done at 3:14 March 21, we better buy hire lots of sales people the week before." Yet we continue to create reams of PERT and GANTT charts and consume terabytes of Microsoft Project.

Item three of the IEEE's Code of Ethics reads: "[We promise] to be honest and realistic in stating claims or estimates based on available data." A noble promise, indeed, one that we as professionals should - must - live up to, if we're ethical people.

Alas, use the word "schedule" and developers' eyes roll. Most feel it's impossible to create an accurate delivery estimate, so they guess and then double the figure. And are late regardless.

For most projects it is possible to accurately estimate a schedule. See my article from October of last year (The Middle Way, http://www.embedded.com/showArticle.jhtml?articleID=49901892) for more information. It's not easy; it requires some time and dedication, yet the process of Wideband Delphi produces both an accurate timeline and a better, more refined, specification. Since poor specs insure project disaster the double benefits of Delphi are too compelling to ignore. Yet few of us employ any sort of formal process to estimate project size. Sometimes we're prohibited by management. Often it's because we haven't learned the art of estimation. Or we're lazy. In my opinion, the latter two smack of unprofessionalism at best or even poor ethics or outright fraud.

In more than a few organizations it's quite impossible to do any sort of formal scheduling since the boss kindly provides the due date. Or he alters the date we've laboriously generated. Dysfunctional? You bet, at least when he acts capriciously. But we developers have to understand pressures the boss faces. We must lift our heads out of the lab, stop staring into the depths of a computer screen, and get some insight into why our dates just aren't acceptable. There are a lot of reasons, and some are intractable.

NASA engineers and their contractors have been cited for many spacecraft failures that stem from buggy code. In nearly every case the investigation board complains about unrealistically-short schedules. An Aerospace Corporation study titled "Small Satellite Costs" shows dramatically a correlation between short schedules and failed missions. So we could save billions by giving developers time to get things right - right?

Maybe not. There's a launch window to Mars about every two years. Miss it and the mission goes into a 24 month wait state. You just can't negotiate with planetary geometry. So today it's not uncommon to "ship" with the code not quite done, using the long coast phase to work out some of the firmware's problems. That seems a reasonable trade-off, though this and other practices have still lead to numerous failures.

What about show dates? Every industry has trade shows, and the next event is never more than a year away. We developers loudly protest a delivery date set just to get the new gadget to the show, with good reason.

But the boss is right, too. The show is important - sometimes vitally so. It's the first chance to demonstrate the technology to potential customers and investors. The press is out in force at these events, and free PR is one of the most valuable commodities the show provides. Slip the date and your company is broadcasting silence into what is probably a hotly competitive field.

I've written about dealing with show dates and finding a compromise that satisfies management's legitimate needs with the reality of engineering. See my February, 2001 column (http://embedded.com/story/OEG20010222S0055) for ideas.

To management a fast schedule is important. Engineers are some of the most highly compensated people in any company. We burn money at a frightening rate while our efforts produce nothing that earns revenue during the development cycle. We're overhead, that cursed (to MBAs, at least) drain on profitability they'll do anything to trim. While this fact doesn't change the realities of engineering, the pressure on the boss to crank more stuff out faster is overwhelming. Today, sadly, few venture capital firms will fund startups that don't have an offshoring plan. Cut the overhead by using low-cost workers and crank up margins for the investors.

In a healthy organization the boss needs a real date. One we won't miss or renegotiate. Engineering is but part of the work needed to get a product out the door. Tech writers create a manual. Production may need to re-tool the assembly line. Advertising can't (or shouldn't) start till product is ready to be sold. If all of the elements are ready, except there's no product, the business runs inefficiently.

Todays hyper-competitive business climate permits no inefficiencies. Ironically we engineers who invented fax machines, telecom and the Internet have built the very foundation of our failure. Everything happens so fast that a small slippage compounds into disaster. Slip this morning and the Chapter 7 lawyers are milling by evening.

Finally, someone has to manage the money and work a balance between spending (on ads, parts and the like) and income. When will we ship? Only then does a revenue stream develop to offset these costs. In the meantime should we draw down the line of credit? Secure another round of financing? When engineering is late we destroy the delicate cash flow projection; more money has to come from somewhere.

As a young engineer I worked for a company that was always a step away from bankruptcy. Creative accounting kept the business afloat. When it went public an auditor said "I've seen every trick this company has pulled before. but never all by one outfit." Schedules all came from the accounting department. "If the product isn't shipping by such and such date, we're going out of business." Needless to say, rampant uncompensated overtime kept the company afloat.

I lost two boats in the Atlantic; have been thrown overboard in mid-ocean, alone, a thousand miles from land; brought a plane down with failed landing gear; and have raised teenagers. People ask if I was scared during those adventures. I can honestly say the scariest thing I've ever done is making payroll week after week, year after year. One friend ran a company that for 67 weeks in a row didn't have enough money in the bank for payroll each Friday morning, yet this sorcerer somehow managed to conjure up enough cash to dispense checks by afternoon. He grew old before his time.

Making payroll changes something in a manager. A slippage, an excuse, a promise unkept that jeopardizes employee paychecks is awfully hard to tolerate. When the company's owner has to get a personal loan or take a big whack on a dozen credit cards to cover wages because engineering is late - again - it's not terribly surprising that he's morose and irascible. Summoning the weekly payroll miracle is as exhilarating as a roller coaster ride, yet as painful as chemotherapy when there's no product to ship.

Of course it's silly to abuse engineering because the company is short of cash. But feelings run strong, and we human creatures, even Spock-like engineers, react strongly to those emotions.

I'm frustrated by the great divide between the development team and the boss or the CEO. Each side feels abused by the other, mostly over scheduling. We engineers post Dilbert cartoons around and sneakily slide occasionally appropriate ones under the boss's door. Though Scott Adams does indeed elucidate some painful truths in a very funny way, his image of the noble engineer versus the pointy-haired boss helps no one and widens the gulf that divides us.

Wise developers must understand the powerful pressures on the boss. And management simply cannot issue capricious deadlines, and must be willing to work with a spirit of compromise with the engineering team.

Conclusion

Decades ago the boss stopped by my office and helpfully gave me the end date for large new development effort. I protested; he calmly told me that, fine, if we couldn't deliver on time I was to lay off three engineers immediately. Sputtering, I agreed to the end date, hoping that perhaps with enough divine intervention we might not be terribly late. That was a promise I knew I couldn't fulfill, and so was clearly wrong. But the alternative seemed worse. To this day I wonder what the most ethical choice would have been.

Are we required to fall on our sword when faced with this or similar dilemmas? There's some balance there, between a fiduciary responsibility to the family and to demanding the boss works with an honest schedule. And between an honest schedule and the company's sometimes very real need for the impossible.

But the lines are blurry. There's no Holocaust Museum or discussion group to work out the ethics of these scheduling questions. I welcome your thoughts and insight.

Here's a challenge. When you're the boss, and the CEO tells you to cut 6 months off the schedule because there's just not enough cashflow to support the long effort, what action will you take?