Follow @jack_ganssle

The logo for The Embedded Muse For novel ideas about building embedded systems (both hardware and firmware), join the 27,000+ engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype, no vendor PR. It takes just a few seconds (just enter your email, which is shared with absolutely no one) to subscribe.

By Jack Ganssle

It's Only a Software Change

Published 7/29/2003

Back in the 70s, when co-development meant fending off dinosaurs as we punched out our programs on paper tape, I worked on a system that used 16 - count `em - sixteen 1702A EPROMs as the program store, for a breathtaking total of 4k of memory. It's hard to believe that a $20 chip requiring three power supplies held only 256 bytes of code, but the microprocessor industry was in its infancy then.

This was a physically large system that analyzed the chemical makeup of coatings on nylon. The customer was anxious to get the device and our under-funded company was desperate to ship so we could put the receivable on the books. It went early, of course, before the firmware was complete, sent as cargo on a ship. The voyage to the Middle East required a month, during which time we figured we could finish the code and then air freight a final memory card to the customer.

The ship sank a week into the voyage. Insurance paid the our company and the resulting delays bought us the time we needed to get the code right.

We were lucky.. But unfortunately, you can't count on such disasters as a scheduling mechanism.

Eventually the program worked correctly and we committed it to masked-ROM to reduce production costs. The stockroom had thousands of sets of chips. The smallest software change obsoleted all of those parts. Though management never understood that firmware changes, feature enhancements, and bug fixes were hugely expensive in engineering time, they were deathly afraid of tossing out so much inventory. For the only time in my career I was in a situation where no one ever said "hey, it's only a software change."

Today better technology - like Flash memory - has removed the inventory constraint to code changes. It's only a software change - we download new code in seconds, can deliver fixes and enhancements via the Internet, and can even return to the glory days of self-modifying code. We can ship early, before the system works, via UPS Ground, an analog to the old slow-boat-to-China, and email code that works as the system arrives in the customer's anxious hands.

I'd never advocate getting rid of reprogrammable memory devices. They do offer astounding benefits. But there's an underlying cost ignored by virtually all managers. It's too damn easy to figure a minor enhancement is "only a software change."

Engineering is generally a fixed expense in most organizations; the cost to develop and maintain a product is rarely amortized over the number of units sold. Using reprogrammable parts means there's no cost to making changes, or at least no cost that's traceable to the product. We developers understand that the very simplest of feature changes might require weeks or months of expensive engineering, but the product manager gets the upgrade apparently for free. Why not endlessly needle us with requests for mods?

I hear engineering managers complain bitterly about the cost of Flash memory; in some products it's nearly the largest cost driver in the system. A scarier costs, in my opinion, is that it reinforces the myth of "it's only a software change."

It's a fact that even a single-bit change to a program may require weeks of validation, integration, testing, and productization. Managers just don't seem to get this. Masked ROM gave them a cost they could understand; reprogrammable chips don't.

The sense is that software changes are free. That implies engineering is free. QED. Yet they're dumping engineers in droves, because we're expensive assets. Where's the logic in all of this?