"Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 429, September 6, 2021
Copyright 2021 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Embedded Studio cross platform IDE

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Anniversary time! 200 years ago this month Michael Faraday ran experiments that led to the electric motor. There's a good article about this in Spectrum. I've ordered the book cited there ("Faraday, Maxwell, and the Electromagnetic Field").

Quotes and Thoughts

Lehman's second law: An evolving system increases its complexity unless work is done to reduce it.

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Done Herres responded to last issue's On System Descriptions:

FMEA's are frequently regarded as another form to hack out at the last minute.  When used properly, you have every major function at the top level, then the next level down of functionality, etc:  At that point, you not only document what could cause this not to meet specification, but WHY!  What thought process went into design forks is key.  Too often, at some late stage, some one suggests going in a different direction and no one can remember why we did not drive off of that cliff.

Paul Miller commented:

The System Description is a great idea. I often use a variant of that by writing an early test plan. I have found this to be a useful way to validate the requirements. Once clients understand the test plan, they see the contradictory and missing bits in the requirements. We go from there to a system description/ engineering specification. In one company I worked for, our ISO-9001 process required three approved documents before starting design work – requirements (e.g., product specification), engineering specification and test plan.

I write the early test plan as a "what we are going to do" rather than "how we are going to do it" document. For instance, a test instruction might read "Induce failure in the gizmo and verify failure is indicated on the control panel" rather than "disconnect the wire from JB4-TB5-3A and verify panel lamp PN5-PL14 is on."

A detailed test plan can be written for final product verification as needed.

In response to my rant about EULAs in the last Muse Chris Gates suggested this bit of perfection.

Freebies and Discounts

Sinelabore has kindly offered a copy of their SinelabeRT tool for this month's giveaway. I haven't tried it, but it does look interesting. From their website: "Generate production ready source code from UML state – and activity diagrams! Build robust event-driven embedded real-time software based on hierarchical state machines. Use your favorite modeling tool and generate code from it with an easy-to-use command line tool. Or use the built editor to create state machines within minutes. Automatic model checks warn from design flaws. Configure the generation process according to your needs. Simulate your model. Generate trace code automatically if needed. All major state diagram features like hierarchical states, regions, history, sub-machines … are supported." More info here.

Enter via this link.

Dumbing Down Engineering Education - Redux

In response to an article in the last Muse about dumbing-down engineering educations, Vlad Z writes:

> The first is an attempt to teach "coding" via some abbreviated
> schedule. Some of that is just HTML or CSS which doesn't require much
> erudition.

"Coding" is not software engineering.  I have dealt with people who do "coding"; it is very hard to explain a problem to them that manifests itself via bad coding.  Fortunately, they were all application programmers.

> The second is more concerning. Dumbing down the engineering curriculum
> by trimming math and science strikes me as a disservice to the student
> and to society in general. As a certified Old Fart I know that few
> engineering students really know what sort of work they will be doing.

As another Old Fart, I take that a bit further.  It's not just math. I have a non-EE, non-comp-sci degree (comp-sci degree did not exist yet), which covers broad subjects.  Thermodynamics, fluid dynamics, phys-chem, electric motors and generators, and a number of others.  The curriculum provided knowledge not only in specialty subject, but in adjacent areas of knowledge as well.  As an embedded software engineer, I worked in various areas, and understanding the subject matter helped solve software problems.  It even helped to change an approach to designing a solution once. 

Embedded software engineering deals with the real world, the analog one.  One needs to know what the system interacts with.

Dumbing down engineering for the sake of inclusion of those who cannot deal with college level math and sciences but satisfy various non-engineering criteria is not just disservice to the students, it's dangerous. Colleges doing remedial courses are doing the job of high schools, rather than their own.  IMHO, the engineering curriculum should be broadened, not dumbed down.

For an alternative view, Greg P. writes:

Ex-US Navy tech here. No college. Fell into firmware and love it. But it's just not that hard! My boss, he's happy if he can get anything that works. I've never even used algebra in this field.

Some other correspondents complained about college "elitism" in language I don't care to reprint here. I don't want to get into the whole "elite" debate that seems to be rending the country, generally in poorly-reasoned ways. However, in my opinion the best thing we can do for our children is to give them an education. And that starts very young. A few weeks ago a friend told me of a family he visited (with young kids) where there was not a single book in the home. I was appalled.

It's easy to encourage kids to read but I don't know how to inculcate a love of reading. With reading and a bit of curiosity it's possible to learn most anything. My parenting rule from the time the kids could talk was "I'll buy you any book you want." (Of course, there are the obvious limits). Limited TV and screen time plus family reading time worked for us.

Engineering textbooks can be dense and inscrutable. Having to struggle just in the reading part of trying to decipher a complex text pretty much dooms a student to failure.

We'll Ship a Fix

"We'll ship a fix."

"We'll fix it in software."

These are two all-too-common statements we hear. Both are valid. Both make me shudder.

As I write this Windows is urgently telling me to reboot to load a new version. But I'm here, now, and want to get this done rather than wait half an hour for yet another Microsoft fix - and this despite nighttime auto updates being enabled.

One of my favorite programs is Filezilla, but it needs constant updates - 15 so far this year, 7 of those in January. It needs an update nearly every time I load it. No doubt these new versions are important, but they interrupt my workflow.

On the other hand there's Dreamweaver, a monster of a program that does a few things really well. But it's very slow and ridden with problems that never get fixed.

A couple of years ago I wrote about a meme going around that bugs are not a problem if you can fix them quickly. Quick fixes are nice. I hear many complaints about a well-known RTOS that has been around forever where fixes take months. That's a terrible way to treat customers. But bugs ARE a problem. It's the developers' duty to ship known good code, not software that will eventually evolve into something more-or-less acceptable.

Sure, we're not perfect so perfect software is nearly unobtainable. But did you know that only about 23% of companies manage to remove more than 90% of injected bugs from their code before shipping? Only 0.4% remove 99%. We've got to do better than that.

In-the-field fixes have saved many projects. The Mars Pathfinder mission suffered from nearly catastrophic priority inversion problems as it was enroute to the planet, saved by an uploaded fix. Mars Exploration Rover crashed when it tried to drill a rock, the problem traced to an inproperly-managed filesystem. It, too, was saved by a patch.

Field fixes are a fact of life in this business. But better processes that lead to correct code are just as important.

Genghis Khan and Software Engineering

Regular readers know I'm a big fan of code inspections. Overwhelming evidence - and common sense - has shown that two heads are better than one at finding problems, and adding another team member or two yields even better code.

But inspections suck. Few developers really enjoy them. I sure don't, but we're professionals hired to build systems as efficiently and as perfectly as possible. A doctor might not enjoy giving prostate exams much, but to avoid such tests just out of personal distaste is both unprofessional and unacceptable. The same goes for the unsavory bits of our work.

Because of this common aversion it's not uncommon for a team that starts with all of the best intentions to slowly find reasons to avoid reviewing code; that nearly always eventually results in inspections becoming completely abandoned.

Because of this I feel inspections are one of the few areas where Genghis Khan software management is appropriate. A strong leader insists on their continued use. That person both supplies the resources needed and audits the process to ensure compliance.

But you don't want management on the inspection team (developers tend to avoid pointing out errors when the boss is around, out of respect for their colleagues). So how can one ensure the team is doing the right things?

The solution is root cause analysis, used statistically. From time to time the boss should identify a bug that was found and perhaps fixed, and have the team or even a member of the team figure out how the mistake made it into the code.

Was an inspection on the affected code ever done? How much time was devoted to it? Were other problems identified? (This implies collecting metrics, which is nearly painless when automated with tools). It's possible the function was unusually complex and the review did find many other problems. So, were complexity metrics taken?

Or - gasp - did the developers shortchange the inspection or skip it altogether?

Perhaps the bug slipped in later, post-inspection, due to poor change management procedures.

Bugs are part of the process, but proper bug management should be an equally important component. It's not possible to do a root-cause analysis of every, or even most, problems. But some level of it keeps the developers on the proper track and can identify flaws in the system that cause delays and quality problems.

C.A.R. Hoare said: "The real value of tests is not that they detect bugs in the code but that they detect inadequacies in the methods, concentration, and skills of those who design and produce the code." And that observation is also true for looking for root causes of at least some of the bugs.

Failure of the Week

Daniel Briggs sent this:


Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.

This Week's Cool Product

Though I mentioned this part in Muse 381, the NewYorker (of all places) has a very accessible piece about the world's biggest chip. Think of a hunk of silicon the size of a dinner plate (a 46,225 mm2 die) that needs 15 KW and which sports 1.2 trillion (yes, with a "t") transistors.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.


Joke For The Week

These jokes are archived here.

In an announcement that has stunned the computer industry, Ken Thompson, Dennis Ritchie and Brian Kernighan admitted that the Unix operating system and C programming language created by them is an elaborate prank kept alive for over 20 years. Speaking at the recent UnixWorld Software Development Forum, Thompson revealed the following:

"In 1969, AT&T had just terminated their work with the GE/Honeywell/AT&T Multics project. Brian and I had started work with an early release of Pascal from Professor Niklaus Wirth's ETH labs in Switzerland and we were impressed with its elegant simplicity and power. Dennis had just finished reading 'Bored of the Rings', a National Lampoon parody of the Tolkien's 'Lord of the Rings' trilogy. As a lark, we decided to do parodies of the Multics environment and Pascal. Dennis and I were responsible for the operating environment. We looked at Multics and designed the new OS to be as complex and cryptic as possible to maximize casual users' frustration levels, calling it Unix as a parody of Multics, as well as other more risque allusions. We sold the terse command language to novitiates by telling them that it saved them typing.

Then Dennis and Brian worked on a warped version of Pascal, called 'A'. 'A' looked a lot like Pascal, but elevated the notion of the direct memory address (which Wirth had banished) to the central concept of the language. This was Dennis's contribution, and he in fact coined the term"pointer" as an innocuous sounding name for a truly malevolent construct.

Brian must be credited with the idea of having absolutely no standard I/O specification: this ensured that at least 50% of the typical commercial program would have to be re-coded when changing hardware platforms. Brian was also responsible for pitching this lack of I/O as a feature: it allowed us to describe the language as "truly portable".

When we found others were actually creating real programs with A, we removed compulsory type-checking on function arguments. Later, we added a notion we called "casting": this allowed the programmer to treat an integer as though it were a 50kb user-defined structure. When we found that some programmers were simply not using pointers, we eliminated the ability to pass structures to functions, enforcing their use in even the Simplest applications. We sold this, and many other features, as enhancements to the efficiency of the language. In this way, our prank evolved into B, BCPL, and finally C. We stopped when we got a clean compile on the following syntax:

for (;P("\n"),R-;P("|"))for(e=3DC;e-;P("_"+(*u++/8)%2))P("|"+(*u/4)%2);

At one time, we joked about selling this to the Soviets to set their computer science progress back 20 or more years.

Unfortunately, AT&T and other US corporations actually began using Unix and C. We decided we'd better keep mum, assuming it was just a passing phase. In fact, it's taken US companies over 20 years to develop enough expertise to generate useful applications using this 1960's technological parody. We are impressed with the tenacity of the general Unix and C programmer. In fact, Brian, Dennis and I have never ourselves attempted to write a commercial application in this environment.

We feel really guilty about the chaos, confusion and truly awesome programming projects that have resulted from our silly prank so long ago."

Dennis Ritchie said: "What really tore it (just when ADA was catching on), was that Bjarne Stroustrup caught onto our joke. He extended it to further parody, Smalltalk. Like us, he was caught by surprise when Nobody laughed. So he added multiple inheritance, virtual base classes, and later ... templates. All to no avail. So we now have compilers that can compile 100,000 lines per second, but need to process header files for 25 minutes before they get to the meat of "Hello, World".

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.