For novel ideas about building embedded systems (both hardware and firmware), join the 40,000+ engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype and no vendor PR. Click here to subscribe.

By Jack Ganssle


Published 8/10/2005

Old-timers remember the pre-Flash days, when firmware lived in EPROMs. Compile the code. Test it. Ship it. Bugs? The customer lived with them until we were able to Fedex new chips.

Before Flash, updates were horrendously expensive. The vendors shipped new EPROMs to the entire customer base; to people who were expected to dismantle a complex system, extract one or more socketed chips, and install new ones without bending leads or zapping fragile electronics with static. Today the reverse is true. Though the vendor might send an email to the customer base about bug fixes, the users must go to the company's web site and get new files themselves. Somehow the customer/provider relationship has changed. The responsibility for getting a fix installed belongs to the end-user.

I see an anti-parallel to the auto industry of the 70s. Detroit shipped cars full of annoying defects, counting on dealers and customers to resolve the problems. Then the Japanese demonstrated a new paradigm of providing perfect cars cheaply. Today PC users, and it seems more and more embedded systems consumers, are also expected to work their way through a long punch-list of defects.

In the 70s American automakers and consumers thought there was no practical solution to the problem of shoddy cars. Consumers were resigned to a series of dealer visits during the first few months of new ownership. If we today assume that big systems are inherently buggy, will some brilliant upstart prove us wrong? America is the world's largest exporter of software. Is that hegemony as tenuous as we found Detroit's to be?

I wonder if in the pre-Flash days the huge costs behind an upgrade made developers more careful in testing their code. Now that it's easy to do an upgrade, the temptation to ship early is powerful. Only a very strong team, with enlightened support from management, can resist the ship-urge.

NASA routinely sends spacecraft off on long voyages with incomplete code. Why not use a 2 year coast phase to cheat the schedule? Every brand-new PC's software is hopelessly outdated the minute a consumer opens the box. And more than a few embedded apps get shipped long before their time, buggy and incomplete.

UPS takes 5 days to ship a package across country. Does your company use that week to play with the schedule? Ship it, sure, but then use a heroic sprint to really finish the code and email updates?

I know of only one case where this approach was totally successful. We sent a big system to the Middle East by ocean-going ship, a 30 day trip we used to finish the software. But (fortunately) the ship sank, providing us a much-needed two extra months of development time. Rumors that the vessel's plating had been compromised by the software team were vigorously denied.

But you can't count on those kinds of providential events. Be suspicious if the boss adds "ship sinks" to week 32 of the Pert chart.