For novel ideas about building embedded systems (both hardware and firmware), join the 35,000 engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype and no vendor PR. Click here to subscribe.
By Jack Ganssle
A quiet "cool!" slipped from my lips as the room filled with smoke. I watched the engineer first turn pale, then lean heavily on this elbows. Head hung low, slowly swinging slowly back and forth, he seemed on the verge of a breakdown.
All of 16 years old, I had no understanding of the poor man's torment. With kids, a mortgage, and multiple car loans no doubt his family lived on the American-standard brink of bankruptcy. Now, due to a simple mistake, he had smoked a $40,000 system due for delivery in a week. Later I learned he offered his resignation to the boss, who wisely sent him back to the lab with orders to "fix that unit now!"
The company was in the business of selling prototypes (always glamorized by a moniker reflecting a more perfect product) to NASA. Everything was a one-off design; everything was delivered to the customer. The frenetic pace of Apollo brooked no delays.
My undercapitalized employer always spent tomorrow's money on today's problems. There was no spare cash to cover risks. In the case of the smoking ground support box we had no spare system, nor even spare boards. As is so often the case, business issues overrode the laws of physics and human missteps: the prototype simply had to work - period.
Years ago I carried this same dysfunctional approach to my own business. We prototyped products, of course, but did so leaving no room for failure. Schedules had no slack; spare parts were scarce, and people heroically overcame resource problems. In retrospect this seems silly, since by definition we create prototypes simply because we expect mistakes, problems, and, well. failure. But business imperatives have their own urgency.
Can you imagine being a civil engineer? Their creations - a bridge, a building, a major interchange - are all one-off designs that simply must work correctly the first time. We digital folks have the wonderful luxury of building and discarding trial systems.
Software, though, looks a lot like the civil engineer's bridge. Costs and time pressures means code prototypes are all too rare. We write the code and knock out most of the bugs. Version 1.0 is no more than a first draft, minus most of the problems.
Though many authors suggest developing version 1.0 of the software, then chucking it and doing it again, now correctly having learned much from the first goaround, I doubt that many of us will often have that opportunity. Schedules are just too frenetic, workforces too thin, and time-to-market pressures too intense. The old engineering adage "If the damn thing works at all, ship it", once only a joke, now seems to be the industry's mantra.
Besides - who wants to redo a project? Most of us love the challenge of making something work, but want to move on to bigger and better things, not repeat our earlier efforts.
Hardware now suffers from the same ills. Reprogrammable logic means the hardware is nothing more than software. Slap some smart chips on the board and build the first production run. You can (hopefully) tune the HDL to make the system work despite major logic errors.
We're paid to develop firmware that is correct - or at least correct enough - to form a final product, first time, every time. We're the high tech civil engineers, though at least we have the luxury of fixing mistakes in our creations before releasing the product to the cruel world of users.
Though we're supposed to build the system right the first time, we're caught in a struggle between the computer's need for perfect instructions, and marketing's less than clear product definitions. The B-schools are woefully deficient in teaching their students - the future product definers - about the harsh realities of working in today's technological environment. Vague hand waving and white-board sketches are not a product spec. They need to understand that developers must be unfailingly precise and complete in designing the code. Without a clear spec, the engineers, by default, will create the spec.
Most of us have heard the "but that's not what I wanted" response from management when we demo our latest creation. All too often the customer - management, your boss, or the end-user - doesn't really know what's they want until they see a working system.
The solution is a prototype of the system's software, running a minimal subset of the application's functionality. This is not a skeleton of the final code, waiting to be fleshed out after management puts in their two cents. I'm talking about truly disposable code.
I'll present my Better Firmware Faster seminar in Melbourne, Australia February 20. All are invited. More info here.
Any prototype will fail unless the goals are clearly spelled out.
The best prototype spec is one that models risk factors in the final product. Risk comes in far too many flavors: user interface (human interaction with the unit, response speed), development problems (tools, code speed, code size, people skill sets), "science" issues (algorithms, data reduction, sampling intervals), final system cost (some complex sum of engineering and manufacturing costs), time to market, and probably other items as well.
A prototype may not be the appropriate vehicle for dealing with all risk factors. For example, without building the real system it'll be tough to extrapolate code speed and size from any prototype.
The first ground rule is to define the result you're looking for. Is it to perfect a data reduction algorithm? To get consensus on a user interface? Focus with unerring intensity on just that result. Ignore all side issues. Build just enough code to get the desired result. Real systems need a spec that defines what the product does; a rapid prototype needs a spec that spells out what won't be in it.
More than anything you need a boss who shields you from creeping featurism. We know that a changing spec is the bane of real systems; surely it's even more of a problem in a quick-turn model system.
Then you'll need an understanding of what decisions will be made as a result of the prototype. If the user interface will be pretty much constant no matter what turns up in the modeling phase, jump into final product development. If you know the answer, don't ask the question.
Define the deadline. Get a prototype up and running at warp speed. Six months or a year of fiddling around on a model is simply too long. The raison d'ątre for the prototype is to identify problems and make changes. Get these decisions made early by producing something in days or weeks. Develop a schedule with many milestones where non-developers get a chance to look at the product and fiddle with it a bit.
The goal of a prototype is to resolve unknowns, not to build a final product. Yet when the boss sees something working he's inclined to ship. A hardware prototype looks ugly; there's a zillion green wires all over the PCB. Those green wires are also in the software version, but just aren't visible. "Working" doesn't mean maintainable, reliable or even tested, so it's wise to create a prototype that might work, but that isn't shippable. So I'm inclined to eschew C and use really high level "languages" like Visual Basic. Or Excel macros. The goal is to get something going now. Use every tool, no matter how much it offends your sensibilities, to accomplish that mission.
Spreadsheets are wonderful tools for evaluating the product's science. Unsure about the behavior of a data smoothing algorithm? Fiddling with a fuzzy logic design? Wondering how much precision to carry? Create a data set and put it in your trusty spreadsheet. Change the math in seconds; graph the results to see what happens. Too many developers write a ton of embedded code only to spend months tuning algorithms in the unforgiving environment of an 8051 with limited memory.
Though a spreadsheet masks the calculations' speed, you can indeed get some sort of final complexity estimate by examining the equations. If the algorithm looks terribly slow, work within the forgiving environment of the spreadsheet to develop a faster approach. We all know, though too often ignore, that the best performance enhancements come from tuning the algorithm, not the code.
Does your product have a GUI? Maybe a control panel? Lots of math? Consider using products like MATLAB or LabVIEW to create a look and feel that marketing can evaluate. Coupled to standard data acquisition boards and a bit of code you can produce models of many sorts of embedded systems in hours.
The cost of creating a model of your product is tiny compared to designing, building, and troubleshooting real hardware and software. Though there's no way to avoid building hardware at some point, count on adding months to a project when a new board design is required.
Another nice feature of doing a model of the product is the certainty of creating worthless code. You'll focus on the real issues - the ones identified in your prototyping goals - and not the problems of creating documented, portable, well structured software. The code will be no more than the means to the end. You'll toss the code as casually as the hardware folks toss prototype PC boards.
Though the PC is a great platform for modeling, do consider using current company products as prototype platforms. Often new products are derivatives of older ones. You may have a lot of extant hardware and software in a system on the shelf. Be creative and use every resource available to get the prototype up and running.
Toss out the standards manual. Use every trick in the book to get it done fast. Do code in small functions to get something testable quickly, and to minimize the possibility of making big mistakes.
All of us have worked with that creative genius who can build anything, who pounds out a thousand lines of code a day, but who can never seem to complete a project. Worse - the fast coder who spends eons debugging the megabyte of firmware he wrote on a Jolt driven all-nighter. Then there are the folks who produce working code devoid of documentation, who develop rashes or turn into Mr. Hyde when told to add comments.
We struggle with these folks, plead with them, send them to seminars, lead by example, all too often without success. Some of them are prima donnas who should probably get the ax. Others are really quite good, but simply lack the ability to deal with detail. which is essential since, in a released product, every lousy bit must be right.
These are the ideal prototype developers. Bugs aren't a big issue in a model, and documentation is less than important. The prototype lets them exercise their creative zeal, while it's limited scope means problems are not important. Toss twinkies and caffeine into their lair and stand back. You'll get your system fast and they'll be happy employees. Use the more disciplined team members to get the bugless real product to market.
The agile development approaches stress the importance of getting something working fast in order to provoke change. Show the customer a system to get lots of feedback early. There's a lot of value in employing these techniques in prototyping.
The NASA system I described smoked due to a bad power supply which took out over 100 ICs. The engineer, his resignation rejected, through heroic efforts and countless packs of cigarettes managed to salvage it, delivering it almost on time. Hours later he was deep into another high-pressure project with another impossible deadline and too few resources. He had kids to feed, so unquestioningly accepted the new assignment.
Watching this man's angst I started to see how pressure can shrivel a person's soul. Part of management is effectively using people's strengths while mitigating their weaknesses. Part of it is also giving the workers a break once in a while. No one crank out 70 hour weeks forever without cracking. Prototypes are always created under tremendous time pressures. But one must be realistic and recognize that humans have human limitations.