You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Dumbing Down Engineering Education - Redux
• We'll Ship a Fix
• Genghis Khan and Software Engineering
• Failure of the Week
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Anniversary time! 200 years ago this month Michael Faraday ran experiments that led to the electric motor. There's a good article about this in Spectrum. I've ordered the book cited there ("Faraday, Maxwell, and the Electromagnetic Field").

Lehman's second law: An evolving system increases its complexity unless work is done to reduce it.

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Done Herres responded to last issue's On System Descriptions:

Paul Miller commented:

In response to my rant about EULAs in the last Muse Chris Gates suggested this bit of perfection.

Sinelabore has kindly offered a copy of their SinelabeRT tool for this month's giveaway. I haven't tried it, but it does look interesting. From their website: "Generate production ready source code from UML state – and activity diagrams! Build robust event-driven embedded real-time software based on hierarchical state machines. Use your favorite modeling tool and generate code from it with an easy-to-use command line tool. Or use the built editor to create state machines within minutes. Automatic model checks warn from design flaws. Configure the generation process according to your needs. Simulate your model. Generate trace code automatically if needed. All major state diagram features like hierarchical states, regions, history, sub-machines … are supported." More info here.

Enter via this link.

In response to an article in the last Muse about dumbing-down engineering educations, Vlad Z writes:

For an alternative view, Greg P. writes:

Some other correspondents complained about college "elitism" in language I don't care to reprint here. I don't want to get into the whole "elite" debate that seems to be rending the country, generally in poorly-reasoned ways. However, in my opinion the best thing we can do for our children is to give them an education. And that starts very young. A few weeks ago a friend told me of a family he visited (with young kids) where there was not a single book in the home. I was appalled.

It's easy to encourage kids to read but I don't know how to inculcate a love of reading. With reading and a bit of curiosity it's possible to learn most anything. My parenting rule from the time the kids could talk was "I'll buy you any book you want." (Of course, there are the obvious limits). Limited TV and screen time plus family reading time worked for us.

Engineering textbooks can be dense and inscrutable. Having to struggle just in the reading part of trying to decipher a complex text pretty much dooms a student to failure.

"We'll ship a fix."

"We'll fix it in software."

These are two all-too-common statements we hear. Both are valid. Both make me shudder.

As I write this Windows is urgently telling me to reboot to load a new version. But I'm here, now, and want to get this done rather than wait half an hour for yet another Microsoft fix - and this despite nighttime auto updates being enabled.

One of my favorite programs is Filezilla, but it needs constant updates - 15 so far this year, 7 of those in January. It needs an update nearly every time I load it. No doubt these new versions are important, but they interrupt my workflow.

On the other hand there's Dreamweaver, a monster of a program that does a few things really well. But it's very slow and ridden with problems that never get fixed.

A couple of years ago I wrote about a meme going around that bugs are not a problem if you can fix them quickly. Quick fixes are nice. I hear many complaints about a well-known RTOS that has been around forever where fixes take months. That's a terrible way to treat customers. But bugs ARE a problem. It's the developers' duty to ship known good code, not software that will eventually evolve into something more-or-less acceptable.

Sure, we're not perfect so perfect software is nearly unobtainable. But did you know that only about 23% of companies manage to remove more than 90% of injected bugs from their code before shipping? Only 0.4% remove 99%. We've got to do better than that.

In-the-field fixes have saved many projects. The Mars Pathfinder mission suffered from nearly catastrophic priority inversion problems as it was enroute to the planet, saved by an uploaded fix. Mars Exploration Rover crashed when it tried to drill a rock, the problem traced to an inproperly-managed filesystem. It, too, was saved by a patch.

Field fixes are a fact of life in this business. But better processes that lead to correct code are just as important.

Regular readers know I'm a big fan of code inspections. Overwhelming evidence - and common sense - has shown that two heads are better than one at finding problems, and adding another team member or two yields even better code.

But inspections suck. Few developers really enjoy them. I sure don't, but we're professionals hired to build systems as efficiently and as perfectly as possible. A doctor might not enjoy giving prostate exams much, but to avoid such tests just out of personal distaste is both unprofessional and unacceptable. The same goes for the unsavory bits of our work.

Because of this common aversion it's not uncommon for a team that starts with all of the best intentions to slowly find reasons to avoid reviewing code; that nearly always eventually results in inspections becoming completely abandoned.

Because of this I feel inspections are one of the few areas where Genghis Khan software management is appropriate. A strong leader insists on their continued use. That person both supplies the resources needed and audits the process to ensure compliance.

But you don't want management on the inspection team (developers tend to avoid pointing out errors when the boss is around, out of respect for their colleagues). So how can one ensure the team is doing the right things?

The solution is root cause analysis, used statistically. From time to time the boss should identify a bug that was found and perhaps fixed, and have the team or even a member of the team figure out how the mistake made it into the code.

Was an inspection on the affected code ever done? How much time was devoted to it? Were other problems identified? (This implies collecting metrics, which is nearly painless when automated with tools). It's possible the function was unusually complex and the review did find many other problems. So, were complexity metrics taken?

Or - gasp - did the developers shortchange the inspection or skip it altogether?

Perhaps the bug slipped in later, post-inspection, due to poor change management procedures.

Bugs are part of the process, but proper bug management should be an equally important component. It's not possible to do a root-cause analysis of every, or even most, problems. But some level of it keeps the developers on the proper track and can identify flaws in the system that cause delays and quality problems.

C.A.R. Hoare said: "The real value of tests is not that they detect bugs in the code but that they detect inadequacies in the methods, concentration, and skills of those who design and produce the code." And that observation is also true for looking for root causes of at least some of the bugs.

Daniel Briggs sent this:

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.

Though I mentioned this part in Muse 381, the NewYorker (of all places) has a very accessible piece about the world's biggest chip. Think of a hunk of silicon the size of a dinner plate (a 46,225 mm² die) that needs 15 KW and which sports 1.2 trillion (yes, with a "t") transistors.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

In an announcement that has stunned the computer industry, Ken Thompson, Dennis Ritchie and Brian Kernighan admitted that the Unix operating system and C programming language created by them is an elaborate prank kept alive for over 20 years. Speaking at the recent UnixWorld Software Development Forum, Thompson revealed the following:

"In 1969, AT&T had just terminated their work with the GE/Honeywell/AT&T Multics project. Brian and I had started work with an early release of Pascal from Professor Niklaus Wirth's ETH labs in Switzerland and we were impressed with its elegant simplicity and power. Dennis had just finished reading 'Bored of the Rings', a National Lampoon parody of the Tolkien's 'Lord of the Rings' trilogy. As a lark, we decided to do parodies of the Multics environment and Pascal. Dennis and I were responsible for the operating environment. We looked at Multics and designed the new OS to be as complex and cryptic as possible to maximize casual users' frustration levels, calling it Unix as a parody of Multics, as well as other more risque allusions. We sold the terse command language to novitiates by telling them that it saved them typing.

Then Dennis and Brian worked on a warped version of Pascal, called 'A'. 'A' looked a lot like Pascal, but elevated the notion of the direct memory address (which Wirth had banished) to the central concept of the language. This was Dennis's contribution, and he in fact coined the term"pointer" as an innocuous sounding name for a truly malevolent construct.

Brian must be credited with the idea of having absolutely no standard I/O specification: this ensured that at least 50% of the typical commercial program would have to be re-coded when changing hardware platforms. Brian was also responsible for pitching this lack of I/O as a feature: it allowed us to describe the language as "truly portable".

When we found others were actually creating real programs with A, we removed compulsory type-checking on function arguments. Later, we added a notion we called "casting": this allowed the programmer to treat an integer as though it were a 50kb user-defined structure. When we found that some programmers were simply not using pointers, we eliminated the ability to pass structures to functions, enforcing their use in even the Simplest applications. We sold this, and many other features, as enhancements to the efficiency of the language. In this way, our prank evolved into B, BCPL, and finally C. We stopped when we got a clean compile on the following syntax:

for (;P("\n"),R-;P("|"))for(e=3DC;e-;P("_"+(*u++/8)%2))P("|"+(*u/4)%2);

At one time, we joked about selling this to the Soviets to set their computer science progress back 20 or more years.

Unfortunately, AT&T and other US corporations actually began using Unix and C. We decided we'd better keep mum, assuming it was just a passing phase. In fact, it's taken US companies over 20 years to develop enough expertise to generate useful applications using this 1960's technological parody. We are impressed with the tenacity of the general Unix and C programmer. In fact, Brian, Dennis and I have never ourselves attempted to write a commercial application in this environment.

We feel really guilty about the chaos, confusion and truly awesome programming projects that have resulted from our silly prank so long ago."

Dennis Ritchie said: "What really tore it (just when ADA was catching on), was that Bjarne Stroustrup caught onto our joke. He extended it to further parody, Smalltalk. Like us, he was caught by surprise when Nobody laughed. So he added multiple inheritance, virtual base classes, and later ... templates. All to no avail. So we now have compilers that can compile 100,000 lines per second, but need to process header files for 25 minutes before they get to the meat of "Hello, World".

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.