A Matter of Style

How do we REALLY institute change?

For hints, tricks and ideas about better ways to build embedded systems, subscribe to The Embedded Muse, a free biweekly e-newsletter. No hype, just down to earth embedded talk. 23,000 other engineers subscribe. It takes just a few seconds (all we need is your email address, which is shared with absolutely no one) to subscribe to the Embedded Muse.

There's nothing a manager likes to hear better than "It's done." Unhappily, in the embedded world I think "it's done" is about as honest as "sure, I'll respect you in the morning."

 "It's done" means something different to everyone. The ultra-smart engineer might mean he has a pretty good idea how to create the product. A firmware-builder generally means the code has been completely written! more or less. Tested? Hey, that's just a detail!

 At the risk of outraging you, gentle reader, I look at "it's done" from an absolutist's perspective. "It's Done" means it's ready to ship to the customer. Period. If you manage an engineering group you're being evaluated on your ability to get quality products to market. As a developer you've got to understand and work towards your boss's or boss's boss's needs. Anything less and you're a meaningless cog in a big wheel, someone all to easy to dispense with during the next down-turn.

 Our goal is not creating code or building widgets. It's getting a product to market. "It's done" means the product is ready to go.

 "It's done, well except for that last bit of optimization" is nothing more than a lie, as that last little task is probably one of a dozen last little tasks, like testing the optimization, or updating the documentation, or going through the release procedure. Yet we know our supervisors want to hear "it's done" - sometimes, when a project is running months late, it's all they dream of - so we do a bit of truth management to get them off our back, to make them happier, or to relieve some of the pressure on us.

 Less than total, objective honesty to those in charge is the act of a teenager, who defers getting in trouble today for much bigger problems tomorrow.

 Yet, we're all guilty of this. I hear it everyday from developers who convince themselves that they are so close to completion that they are all but done, or who feel that one more night of heroic efforts will prove them close enough to being correct. We computer people are the worst sort of egotistical intellectuals; we believe, no we normal">know, that we're so good that we'll bail ourselves out of the disaster of the hour by a few all nighters.

 Experience proves otherwise.

 Projects get late; panic mode starts; and sure enough, the system gets delivered eventually. All of our promises and plans are for naught, though, as a thousand technical, managerial, and business problems thwart our best efforts. Maybe I'm turning into a pessimist, but it seems the one thing we can rely on is the perversity of nature. Something normal">always goes wrong.

 A wise boss hearing "it's done" will question you deeply to understand the exact level of "done". A novice - or one working in the desperation of excessive hope - may believe "done" means ready to ship, and start making commitments no one will live up to.

Don't Do Better

It's up to each of us to break this dysfunctional cycle. But promising ourselves to just "do better" is stupid and ineffective.

 With a bit of age (but less than anticipated maturity) it's interesting to look back, and to see how most of us form personalities very early in life, personalities with strengths and weaknesses that largely stay intact over the course of decades.

 The embedded community is composed of mostly smart, well educated people, many of whom believe in some sort of personal improvement. But, are we successful? How many of us live up to our New Year's Resolutions?

 Browse any bookstore. The shelves groan under self help books. How many people actually get helped, or at least helped to the point of being done with a particular problem? Go to the diet section - I think there are more diets being sold than the sum total of national excess pounds. People buy these books with the best of intentions, yet mostly fail at their goals.

 Our desires and plans for self improvement - at home or at the office - are one of the more noble human characteristics. The reality is that we fail - a lot. It seems the most common way to compensate is a promise made to ourselves to "try harder" or to "do better". It's rarely effective.

 Change works best when we change the way we do things. Forget the vague promises - invent a new way of accomplishing your goal. Planning on reducing your drinking? Getting regular exercise? Develop a process that insures you're meeting your goal. Log the results daily.

 The same goes for improving your abilities as a developer. Forget the vague promises to "read more books" or whatever. Invent a solution that has a better chance of succeeding. Even better - steal a solution that works from someone else.

 The moral is to accept that we humans are pretty bad at living up to our promises, and therefore to make grand, sweeping changes to the way we do things, in order to improve when we do.

 Check out "Quality is personal: a foundation for total quality management" by Harry Roberts, New York: Free Press, 1993, which is a great book about finding new ways to do better. It sort of turns the quality movement inside out, acknowledging our limitations yet providing a framework for improvement despite human frailties.

 Unfortunately in many circles the term "quality" has been Dilertized to the point of total cynicism. So many fads - management and others - get rammed down our throats that it's understandable we cringe at the these overused terms. Try to swallow your bile and read the book for useful concepts. I think improving every area of life is critically important, and pulled some neat ideas from it.

 To tie this aside back to the concept of overcoming the "it's done" weakness, I think the solution is to develop a policy --  and more importantly a   normal">system -- of total exposure. We need to recognize that "it's done" means different things to different people.

 Nothing beats a dynamic action list. It matters little if you use the latest project management software or a ruled paper pad; what's important is maintaining a list of things that must be done before you can utter those profound and soaked-in-meaning words "it's done" with firm resolve that the project is indeed ready.

 No list is effective unless it's updated in real time as the project changes or as you learn more. Planning is always incomplete, a continual source of acrimony as the scope of a project changes. No matter - add the changes to the list, check of things as they are completed, but use the list as your scorecard that shows exactly which parts of a project are "done", and what remains. Expose it constantly to the entire project team, so everyone knows the project's status.

 Undisciplined project teams always drop the management tools - like the action list - as soon as they get "too busy" to keep these tools up to date. Ignore your management tools and disaster is inevitable. 

The Wrap-up

Your action list must contain everything that is part of the project. End-of-project details are just as important as getting the code out, yet these are all too often neglected.

 Take documentation, for example. Somebody has to do that user's manual before the product is ready for the customer. If all of the code is complete, the hardware works, but there's no manual, well, the product for all intents and purposes does not exist. You can't ship it; the development is still in that "spending money, unable to create revenue" status that so endears us to upper management.

 Some documentation inevitably cannot be created till the product itself is pretty much ready to go, but much can - and should be - developed in parallel with the normal engineering.

 In fact, I think it makes sense to rethink much of this. Often a user's manual is a much better project specification than the raw specs themselves. Doesn't it make sense to write the manual first - before creating a line of code? Even better, the non-techie MBA types will get a better look at the final product through the customers' eyes when they read the manual.

 Another area all too often neglected is version control. Some outfits defer the issue of controlling software till a release is imminent. Again, this is inefficient and downright silly. Before starting a project implement a version control system (VCS). Once this was a difficult process; now it's almost trivial.

 Even a one-person shop needs a formal VCS. It's truly magical to be able to rebuild any version of a set of firmware, even a version years old. The VCS provides a sure way to answer those questions that pepper every bug discussion, like "when did this bug pop up?"

A number of vendors provide these. Most of my experience has been with Microsoft's Visual Source Safe, a Windows-hosted product that looks a lot like File Manager.

A VCS insulates your source code from all of the developers. Under Microsoft's product the code sort of disappears, rolled into a database of some sort. Scary stuff! You can get at the code only from the user interface provided by the tool, and cannot change code unless you have the rights to do so. It controls the number of people working on modules, and provides mechanisms to create a single correct module from one that has been (in error) simultaneously modified by two or more people.

Sure, you can cheat the controls on any VCS, as you can cheat your project activities list, but doing so is counterproductive. Maybe you'll get a few minutes of time savings up front, followed inevitably by hours or days of extra time paying for taking the shortcut.

So, save time by never bypassing the VCS. Check modules in and out as needed. Don't hoard checked-out modules "in case you need them". Use the system as intended, daily, so there's no VCS cleanup needed at the project's end.

The same goes for project backups. Develop a system before you start that insures everything is backed up in a reasonable manner at frequent intervals. Don't plan on an end-of-project backup and file cleanup.

A VCS can actually mask the real back-up problem. Normally you'd host the master source files on a single network drive, giving you one node that has all critically important data. The problem is that developers might check out a file and spend weeks making changes while it lives on their local drive. This means a single hard drive failure anywhere in the engineering group can result in the loss of critical data.

In my experience we can't rely on individuals to be religious about back-ups. Some are passionately concerned and will do whatever is needed to have every file stored in more than one place. Others are concerned but inconsistent in their efforts.

I admit to being anal-retentive about backups. A fire that destroys all of the equipment would be an incredible headache; one that took the data would destroy the business. Yet, no yes"> preaching about data duplication, and making draconian rules, has been singularly ineffective. There's a bell curve of response from people, which is not acceptable. An approach that does not rely on people's good intentions is clearly needed.

We keep all critical data (including the VCS database) on one server, and run a series of backups of that machine. All are automated, and other than changing the tape require no human intervention.

Tapes are great for archival storage, but are far from random access. Documents and other files that are not under VCS control change frequently, but the old versions tend to get lost