Shared Perceptions

For hints, tricks and ideas about better ways to build embedded systems, subscribe to The Embedded Muse, a free biweekly e-newsletter. No hype, just down to earth embedded talk. 23,000 other engineers subscribe. It takes just a few seconds (all we need is your email address, which is shared with absolutely no one) to subscribe to the Embedded Muse.

Shared Perceptions

This June, on my annual summer sailing vacation I stopped in Bermuda for a crew change and some R&R before heading south. Having put a number of sailing stories on the 'net it's not too unusual for someone to recognize "Voyager", my 32 foot ketch. So, tied up to the wharf in St. Georges I wasn't surprised to be hailed from a gangly fellow on the dock. He was headed to the White Horse Tavern and wanted to buy me a drink.

Wizened by the sun, with a week-old growth of beard and soiled tee shirt he looked more like one of us sailors than an engineer. But the high tech California lifestyle had been financially good to him, and he had retired to a life of beaches, salt air and rum.

We chatted over Dark and Stormies far into the night, greatly enriching the pub's proprietor.  A regular ESP reader, even here in this near-tropical idyll, we debated some of my fanatically held opinions so often expressed in this pages. I enjoy these sorts of sessions, always learning much even when we hold diametrically opposing opinions. Though some folks grow out of such passionate banter in Sophomore year, I find that a vigorous exchange, be it on politics or engineering, is educational, interesting and great fun.

Jamie told me he had spent much of his career working for a small contract design outfit. Their 25 engineers created some 50 new systems per year, for clients as varied as the Defense Department (where paperwork mattered more than product) to dot com startups requiring some embedded intelligence (where often Jamie's company provided a lot of business planning, since these operations usually had no idea how to make a viable enterprise).

I spent 15 years building an In-Circuit Emulator company, so had the wonderful chance to look over the shoulders of literally thousands of developers creating an astonishing variety of products. It was like vicariously participating in these projects, since anyone who sells debugging tools gets called in only when the project is at its absolute nadir.

I've observed that most of us work nearly alone, in small teams. We tend to learn mostly from our experiences, but those are necessarily limited by working only on a couple of products per year. If our company produces the same sort of product based on similar technology year after year, then our experience base has little chance to grow.

It's quite interesting to compare engineers from companies that produce products to those working for smaller consulting outfits. At the risk of making an overly patronizing generalization, many of the consultants have a wider range of "tools", of abilities, tricks and techniques to get a product to market. They might work on a dozen products in a year, each in very different fields. This broad scope of experience helps them make better systematic decisions (e.g., 8 vs 32 bits). They'll often use better tools, more high level design processes, for example, because over the course of so many projects they find these techniques work.

So Jamie and I shared many of the same experiences, both with technology and the people that build the systems.

Perhaps the most striking impression we've both brought from our careers is the astonishing level of heroics used to complete most projects. Every new product starts with optimistic folks making very quick progress in the design and coding stage. Early debugging, in unit test, progress is quick. Integration, of the software with the hardware or of tested software units into a more complete whole, is where schedules collapse and design flaws become glaringly apparent.

Developers almost universally panic, and then, through late nights and other heroics beat the system into submission. This is where features are discarded willy-nilly in an attempt to minimize schedule slippages. A recent study by Jerry Krasner showed that 57% of us reduce schedule slippages by last minute feature shedding.

I was once fiddling with business statistics and was shocked to discover that some 70% of our emulators were shipped FedEx overnight. Turns out that most of the customers had planned to use much more limited tools, but were stymied by problems far more difficult than imagined. Jamie, too, had found that front-end planning to deal with inevitable back-end surprises was critical to project success.

Though I'm a pretty happy guy, I consider unbridled optimism at the outset of a project to be a sign of future disaster. Before every ocean passage, like the one that brought me to meet Jamie, I tear the boat apart from stem to stern inspecting, testing, and insuring that the vessel is safe for sea. So too for an embedded project: I've learned to quench the enthusiasm and probe the deepest recesses of the plans, designs, and schedules for risks and flaws. Things always turn out more difficult than expected.

Too Many Choices

I asked Jamie if the consulting company had a standard set of tools and technology. In my own business I was loathe to let engineering switch compilers and processors, having found the cost of such change to be astronomical. A small company, with end-users who cared little what was in the box, we were reasonably successful and keeping a consistent technology for many years.

He harrumphed, and said that each customer mandated a certain processor and often even toolchain. Costs were thus much higher than they could have been. Switch even a very smart developer from the ColdFire to a PIC, and watch productivity collapse for the duration of the learning curve. Add in the need for perhaps tens of thousands of dollars for new tools - plus the time to master these always infernal tools - and it's easy to see the cost of capricious processor selection.

He admitted the difficulty of creating standard library-like components that work across processor families. Clearly device drivers share little common structure; even something as inherently simple as a timer has totally different I/O requirements between parts from Motorola and Intel.

Consultants often cannot share technology developed for different customers. Non-disclosure agreements, written by legal departments that don't have a clue about the costs of product development, frequently explicitly prohibit the sharing of any part of a product's source code.

Other consulting companies I've worked with have managed to educate some of their customers about the savings inherent in sharing some of the source. When clients sell to different industries there's usually little threat to a common code base. But this means the consultants must teach the rarely-receptive lawyers the benefits, a difficult task given the legal profession's aversion to risk of any sort. Real or perceived.

Jamie told me of their storage room, filled with costly development systems for projects shipped long ago. What an unproductive waste!

Motorola, Intel and others dealt very effectively with customers' needs for more horsepower by making CPU families that share a common architecture yet offer a wide mix of performance and on-board peripherals. As an ICE vendor I always admired the customers who took full advantage of this. A processor change meant that the compiler could stay the same, and the developers needed no retraining. All emulator vendors offered a simple and cheap pod changes that optimized the customers' investment.

Despite the costs inherent in changing processors, Jamie did say they managed to keep the costliest of all engineering resources constant: process. Their firmware standards and their ways of going about building a product stayed mostly unchanged regardless of client or technology. He admitted, as I've discovered in my own customers, that creating a development process, training the developers, and then helping them turn what is first perceived as a pain into wholly-embraced religion, is difficult and time consuming. But the benefits mean more predictable scheduling, more reliable products, and happier customers.

Inadequate Tools

Perhaps our most spirited debate concerned tools, most especially debuggers. My ex-ICE-vendor status puts me squarely in the "price is no object" camp. If spending $10k or more saves a significant amount of debugging time, then the cost is quickly recouped.

But! a continuing frustration of all debugger people is convincing customers of this "truth". All of us sincerely believe it, yet it's a hard sell. I remember a cost/benefit analysis prepared by the biggest ICE vendor many years ago that attempted to prove just how much money a big-buck emulator saved. Later, an executive at the company admitted to me that this report always fell on deaf ears.

It's pretty hard, after all, to ask management for $10k for a tool whose sole purpose is to find your mistakes. They're inclined to tell the developers to do it right, first. A noble goal, perhaps, but certainly elusive.

My non-scientific observation of hundreds of customers showed that half of the average project is spent in debugging. We're making a lot of mistakes, for sure.

Jamie's company was one of the few that tackled bugs from the outset, minimizing them via code inspections, careful design, and by using all of those great process ideas the pundits preach but few really employ.  With so many projects happening per year, they had had a chance to collect enough statistics that clearly showed the benefits of that approach. They just didn't need a lot of debugging horsepower. And with so many different products based on an equally diverse set of architectures, the cost of tools was staggering.

So I challenged him about features. Modern debugging tools, like BDMs, offer a bare set of resources, a set much reduced from those of an emulator. There's no doubt in my mind that for very good reasons BDMs are the present and the future of debuggers. CPU speeds are too high; SMT and BGA connections make too many unprobable. Yet it's odd that despite increasing size and complexity of our products, our debugging capabilities are reduced from the days of yore.

Jamie was adamant that features are of little value. That exotic cool thing used once a year to find a really tough bug, in his opinion, doesn't really exist. If you don't know how to use the feature, or maybe even forgot about it, then it's of no help.

Well, yeah. Sure. But shouldn't we be experts at our tools? Shouldn't we continually sharpen our skills?

But then I thought about my old customers. Few ever learned more than single stepping and setting breakpoints. The vast depth of features were almost always untapped.

Then he turned the tables on me: "just how many of Microsoft Word's feature do you really know? Or Excel?"

I use these programs for hours a day, yet haven't a clue about most of their capability. Maybe there's a lot of truth there. Me, I hate to load new software on my PC because the thought of reading yet another manual is nauseating.

And there's the dark secret of most products: count on bugs and problems in those features that are rarely used.

Business is Hard

The last impression we shared of this funny industry is our distorted notion of the ease of turning an idea into a product, and then into a profitable business. Jamie told tale after tale of customers backed by venture money, generally from unsophisticated investors more gullible than the average professional VC, who failed despite creating a working product.

There's a euphoria associated with starting a business, especially during the frenetic 90s. Some think that the unbridled optimism of those years was a unique feature of the short lived dot com zaniness. Not true. In 20 years of starting and running various companies I've seen a constant pattern of people - especially engineers - whose very cool idea distorts their vision of reality.

Other business owners confirm my observation: people come to us with ideas, constantly, in an effort to start some sort of partnership. Almost without exception (in my experience, at least) major flaws unrelated to the idea itself clearly doom the invention.

Engineers always seem to think that if the product costs $100 to manufacture, it can sell for $110. None anticipate the overwhelming costs of marketing and sales, of paying for overhead and the cost of capital. In my ICE days our manufacturing costs were about $1000 per unit, yet we had to charge around $8k to squeak out a minimal profit.

Others feel that placing an ad will make the world beat a path to their doorstep, which has been disproven a thousand times even before the Web showed us, yet again, this bald fact. Advertising is crucial but inadequate; a successful business learns to build marketing and sales through a complex and expensive set of interconnected approaches. Advertising is crucial but by itself totally ineffective.

And so our discussion and sometimes loud but always friendly debate roared into the night. I was delighted to find another embedded person in this remote island, and even more pleased by Jamie's spirited and thoughtful comments.

So, if you see Voyager in some port, give us a shout and let's trade ideas. Or, don't hesitate to let me know your feelings about this fascinating business at the Embedded conferences. There's nothing like a great debate.