A Value Proposition - Unfulfilled

Are we too busy to see the real value of things?

Published in Embedded Systems Programming

For hints, tricks and ideas about better ways to build embedded systems, subscribe to The Embedded Muse, a free biweekly e-newsletter. No hype, just down to earth embedded talk. 23,000 other engineers subscribe. It takes just a few seconds (all we need is your email address, which is shared with absolutely no one) to subscribe to the Embedded Muse.

Last March I wrote an article on embedded.com (http://embedded.com/showArticle.jhtml?articleID=18201946) stating that vendors of software tools have done a lousy job of selling us on their value proposition. You know the drill. Take UML, for instance, or static analyzers that examine source code and identify errors before debugging starts. The products look pretty good, and those alluring magazine ads create a bit of tool lust. The salesman isn't at his desk, of course, but a disembodied voice vows that "this call is very important to us." Two weeks pass with more calls diverted to voicemail before the company's representative finally rings back. His strong voice is assuring but strangely lacking in specifics. Sure, there's a heart-warming story or two about some apocryphal project saved by the tool. You're encouraged to read the product literature on the vendor's site, but these cite unreadable academic papers which gleaned insight on a 500 line program written in Smalltalk, developed by three undergrads whose real objective was a passing grade.

 For some reason too many vendors don't understand that their customers are engineers, people who make all choices in life, to our spouses' regret, based on quantitative data. Don't tell us your product will improve productivity - prove it. Give us the numbers. Back up your claims with data and a wealth of documented success stories from people willing to discuss their projects with us.

 The article created quite a furor. Many readers wrote to agree. Others suggested management is at fault, as too many won't spend money on software tools, or are afraid of changing technology. One developer said he's not even allowed to use a compiler! Another complained that his boss won't part with a lousy $239 for Gimpel's must-use Lint.

In an associated poll (http://embedded.com/pollArchive/?surveyno=26500001) only 36% of 260 respondents felt vendors do a pretty good job. That's a flunking grade by any measure. Another poll (http://embedded.com/pollArchive/?surveyno=12900001) showed that only 14% of readers can get their company to fork out more than $1000 for any tool that's not absolutely required, like Lint or a source code analyzer.

So the tool hucksters are not impressing their engineer customers, and haven't convinced management that spending money will save bigger bucks in the long term.

 No vendor responded to the article via the public forum though several wrote to me privately. All agreed that for reasons they understand and probably many more they've done a poor job of getting their message across. One in particular expressed frustration with customers for not "getting it," and dissatisfaction with their own marginally-effective marketing efforts. They asked for a meeting to discuss these issues at the West Coast Embedded Systems Conference. I showed up with a couple of pages of thoughts on the subject stuffed into a pocket. Perhaps it was the beer; somehow interest in the subject had waned. 

 I'm passionate about tools and feel strongly that most of us can benefit from broadening our arsenal of code development tricks and techniques. Yet there is no market for tools. Pundits peg the entire embedded tool industry (for everything from compilers to emulators to RTOSes) at a paltry $764 million, a third of which all goes to one company. The EE world is very different; Agilent alone sells $2.5 billion of test equipment into engineering labs every year. Ask your boss for a $1k software quality tool and watch the door slam in your face. EEs routinely get $50k logic analyzers. Yet software consumes something like 70% of the development cost for electronic products.

 But the problem of limited use for modern software tools and techniques is real, and boils down to three major issues: the nature of developers, the nature of the boss, and that of the vendor itself.

It's Us, Darn It

Few of us would go to the boss and say "my software sucks. I need lots of dough to get it under control." We have a primal fear that asking for software quality tools is akin to admitting incompetence. Version control systems, compilers, editors and the like are safe requests as they are proficiency-neutral. Debuggers, emulators and BDMs stand on shakier ground, but are expected purchases. Code quality tools fall into a whole different battleground. "What's the big deal? Just put the quality in from the get-go. Then you won't need these things!"

 Process terror also afflicts too many of us. We're afraid that new tools and procedures will stifle creativity. Developers view their next project as a blank canvas just waiting for the artist to express anything in any manner. Software would be better and delivered faster if only we had a paint-by-numbers approach.

 Many developers simply don't want smart tools. Want proof? Consider how many engineers ignore compiler-generated warning messages. Compilers, which know the language far better than most of us, emit warnings when there's something possibly fishy in the code. Why would anyone pooh-pooh such portents of possible peril?

 We're skeptical of the untried. A new tool or new process means taking a leap of faith in the middle of an already compressed schedule. Worse, we don't really trust anyone else's code, because we know how bad ours is. So developers view big software products as neatly-wrapped bug packages.

 Finally, in the embedded world most engineers have only a passing acquaintance with any sort of disciplined software process. Few participate in a software process improvement network. We're like alcoholics. That one nasty code patch leads to another, and then another. Pretty soon you're furtively hacking, cheating the VCS so no one sees your bad habits. Process improvement must have a social component, rather like AA, to keep us on the straight and narrow.

 For all of these reasons, when we're promoted to the exalted level of boss the entire cycle repeats.

Nah, it's the Boss

The boss, of course, answers to his superior, or to the stockholders. All demand fiscal discipline. It's hard to justify software quality tools when the costs might be high and the promised results nebulous at best. Anyone can see economic justification for not spending, but the wisdom of investing for future benefits eludes most people.

 It's easier to buy a logic analyzer or scope than a pile of bits. The test equipment has physical manifestation. There's a place to hang a property tag. A $5k piece of software looks like a nickel CD with a book. And generally not a very well-written book, at that. Accountants know how to depreciate the scope, but what are the rules for software?

 Seasoned bosses have learned through bitter experience that software is a problem. It feels inherently intractable. Schedules slip while engineers mumble incomprehensible explanations. Bugs surface with alarming frequency, and again the developers offer little solace. When the boss expects software to be the gateway to perdition he naturally views vendors as snake oil salesman.

 The top dog, who is never part of any sort of process improvement network, who probably reads nothing about software engineering, hasn't the faintest inkling that change is possible, that there are ways to soup up the firmware efforts. He figures thrashing about in the quicksand, just trying hard to keep your nose above the sand, is the natural way of things.

 Shipping always comes first, it seems. Schedule 'lles, 'ven quality. I'm reminded of that old cartoon of the raging medieval battle, arrows flying and swords being wielded with great gusto. The general complains to his aide that he's in the middle of a battle; he simply doesn't have time to see the salesman. Who is standing there, anxious to sell a cartload of machine guns.

 Finally, virtually no one buys anything that incurs upfront costs for long term benefits. That's why the savings rate is so low in the USA. Progress comes, it's felt, only from generating code. Planning? That's an obstacle to progress. Ditto for design, and code inspections, or running the source through Lint or other analysis tool. Just crank some more code, fast!

Maybe the Slimy Vendors?

The vendors are mired in the muck of their own filth as well.

 Every salesman over-promises and under-delivers. Supporters hail every tool, every methodology as offering the bright hope of emancipation from the morass of missed schedules and buggy products. Though many of these tools and methods do help a lot, none live up to the hype. So users are jaded.

 Vendor honesty is too often an oxymoron. Sales people must learn that practicing engineers despise hokum. We'll respect you more when you tell us where your products don't apply.  As it is, all developers fear the painful and expensive product evaluations. Does it work in my environment? How can we expose the limitations early?

 There are too many competing tools and methods. All promise miracles. Should we get started with eXtreme Programming, Test Driven Development, Feature Driven Development, or Shlaer-Mellor? Is UML the right tool? Where does RUP fit in? And what about those zany SEI people pushing CMM, CMMi, PSP and the TSP?

 As I mentioned earlier, the dearth of hard numbers and believable user stories makes most engineers roll their eyes when facing an earnest salesman. Marketing stories that read like marketing stories don't sell.

 Vendors have no effective channels of communications with customers. Ads are swell but rarely get much attention. Web sites work fine after a potential customer identifies a need. None of us want to see a salesman or a dentist; they're both viewed as evils to be avoided whenever possible. Magazine articles touting a specific product are rarely-read puff pieces.

Why not a Scope?

Hardware designers have little trouble securing funding for budget-busting items like scopes and logic analyzers for a number of reasons. First, a scope is a low-risk purchase. There are no illusions, little hype, but there is a general sense by everyone of what these things do.

 The datasheets are as specific as the formula for Viagra. A few pages spell out bandwidth, vertical resolution, sensitivity and number of channels in clinical detail. The devices sell themselves based on bald facts, hard numbers understood by vendors and customers alike.

 Finally, the entire electronics community finds these devices indispensable. Heck, I had a scope (a surplus vacuum tube version without triggering) in 8^th grade. There's no debate about either their efficacy or their essential role in the electronics lab.

 Can vendors of software tools morph their marketing into the scope model? I'd sure like to see a Lint datasheet that said "300 studies confirm the product finds 68% of C errors for 22% of the cost of debugging."

 To reduce risks vendors must find better ways to help prospective customers evaluate the products. Frankly, I'm sick of downloading eval copies of compilers and other tools. My hard disk is a tangled mess of registry hacks and incompletely uninstalled software. Either give us an utterly clean and trustworthy way to uninstall the products, or devise a web-hosted approach.

 Risk reduction also requires lots of user stories. Surfing over to iLogix's web site I find only 8 descriptions of how customers used the well-respected Rhapsody; most read like the promo material they are. Few numbers appear. At Programming Research's site I could find no user stories for their quite cool QA-C.

 Ultimately vendors need to stealthily infiltrate the heads of prospective customers. That happens best when friends and colleagues tout the virtues of a tool or method. Today's buzzword for this is viral marketing, a difficult and often slow process.

 And customers - us - must participate aggressively in the world of software engineering. This is an evolving field; the state of the art is much different than when we left college. Stagnate, do things the same way you did 10 years ago, and watch your career wither.

 Staying involved gives the vendors a better chance to get their message across. Horrors! Who wants to talk to vendors? Well, it better be all of us. Many of these products and techniques are excellent adjuncts that will improve your code and reduce development time.

 Ultimately, for us to compete effectively in this new world order, we'll have to exploit every trick, tool and technique available.