Click here to go to the on-line version The Embedded Muse

The Embedded Muse Logo The Embedded Muse
Issue Number 492, June 17, 2024
Copyright 2024 The Ganssle Group

Editor: Jack Ganssle,

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Gang Programming Solution

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded" in the subject line your email will wend its weighty way to me.

Quotes and Thoughts

Scientists dream about doing great things. Engineers do them. James A. Michener

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Freebies and Discounts

James Grenning and Wingman Software are offering two pairs of seats for his newly released self-paced Test-Driven Development for Embedded C/C++ training course. James is the author of Test-Driven Development for embedded C and founder of Wingman Software. His courses have trained thousands of embedded software developers in TDD and SOLID design principles around the world.

The introductory price is currently $500 per learner, and the course's value could be immeasurable to your career. Each prize consists of two seats so you and a colleague can learn together.

The course has over a dozen exercises, ninety short videos presentations and demos, reference material as well as linked content. The best way to determine if TDD can be valuable to you, is to experience it. James' course is designed to guide you through an experience so you can decide if it can change your life. Find out more about the course here

James is also offering a consolation prize for all entries, $100 off the introductory price.

Enter via this link.

Is Software The Root of All Evil?

In the last issue, a reader who wishes to be anonymous wrote:

Recently, we've come into new corporate leadership in a large company. The new head of engineering has concluded, based on meetings with all engineering groups (mechanical, electrical, software, PM, manufacturing, etc.) that the software team is essentially the root of all evil in the company. Software is the reason that projects are delayed and run past schedule, and we "need to fix it". Now, we know that software/firmware is the bottom of the engineering pit here. Last to get requirements, but also first in line to catch all the blame. My question for the group here is, how common is this perception, and what have you done to combat this?

Lars Pötter responded:

A game I like to play is to take the blame. Stand up to management and say that you completely and fully accept that it is you, that is to be blamed for it. And then continue to agree that all these issues are serious and cost the company a fortune and that you really really want to fix this, but you just don't know how. You are sorry for what you did and want to avoid making these mistakes in the future. All you need is the input on what you could have done differently to avoid the mess. And that you are open to suggestions. So please everybody help us find ways to prevent these horribly things to happen in the future.

This is a shortcut, as the blame game is now over. And everybody can focus on the way forward. It is also real fun to watch the face of the bad boss that did not expect this to happen.

If people do not understand what happened then go into detail. Start with one issue and tell what you your part was. And openly ask what of that caused the issue. Or how you need to change your behavior to avoid the issue in the future. Most of the time the solution will be that someone else needs to do more thorough checks or provide better information,...

Maybe even have the firmware people come up with a strategy on how to move forward.

Most importantly: This probably is not a technical issue! Do not be afraid to suggest non technical solutions!

If the complaint was that the product launch was delayed due to serious issue found in testing. Then the obvious solution is to have testing for such issues earlier on. To be extra sure that it does not happen again, the needed test need to be defined as soon as possible in the project. Such a request is easier to get accepted than more precise requirements. Even though they are the same thing.

The head-of-whatever is usually happy if one presents buying some tools or change some processes as the solution to fixing the department he decided to be the issue. As long as he thinks that this is putting more stress on the culprits. And more testing sooner, to detect they made again, surely is more pressure, right?

But only ask for stuff that really works. Management is usually less understanding the second time round.

Don Herres wrote:

From my own experience as design engineer for a component supplier, software is the wizardry to fix the product when an unknown test from the end user fails or a feature is added.  A software revision (not a fix since it met the original spec) can take days whereas a hardware revision takes months.  How many of each depends on the severity of the change and how fast the supplier can implement changes, but the ratio seems to be correct.  Other teams may not readily admit the software group was the first to save the schedule from being completely shredded.

Here are Christoph Schmidt's thoughts:

When I read your thoughts about "Software - The Root of All Evil", another topic jumped into my mind. As software developers we're also often responsible to fix problems from other domains as software fixes are typically faster and more economical than changing the hardware.

We changed a component to reduce cost and the new one doesn't perform as expected? Change the software driver to make it work.

Device produces too much EMC noise? Let's see how we can change these PWM signals to reduce emissions.

Sensor doesn't read with the expected precision? Change that algorithm to somehow make a reasonable prediction based on some model.

These are all cases we had in the last couple of months in some of our projects. And for some reason this stuff tends to pop up one or two weeks before we need to ship the product to the customer. Like this some managers also get the impression that delays are mainly software faults, even if the problems didn't originate from a faulty software implementation

Crashes From Space!

A friend for whom I have enormous respect once told me we don't need watchdog timers. "Just write perfect code", he said. While I admire his quest for perfection, even completely defect-free code is simply not good enough.

Our devices face many insults, not the least an incoming rain of high-energy particles. The Oh-My-God particle in 1991 was a cosmic ray packing 3x1020 electron volts, about the energy of a baseball at 90 km/h. Thankfully those events are quite rare, but given the tiny geometry of modern semiconductors it's wise to pay attention to these threats.

A watchdog timer is a must. Some developers take additional precautions, like updating GPIOs and other outputs many times per second.

Memfault sells a tool that collects health and crash data from your products after they are deployed. They recently looked at unexpected resets in the devices using their technology. As you may know, we're nearing the top of an eleven-year sunspot cycle. The sun was unusually active in April and May causing dramatic auroras far to the south. Memfault's data shows resets spiked dramatically during these solar flares. For instance, look at the spike on May 11, which correlated with a major flare:

I had been imaging the sun, and on that same day took this picture of solar region 3664, responsible for that flare:

Sun on May 11 with inset of massive sunspot in solar region 3664. Taken with a Celestron 8 and solar filter from my backyard.

The solar cycle peaks next year. Figure on a lot of flares and coronal mass ejections (CMEs) firing energetic particles at our systems.

The Carrington Event in 1859 was an extreme CME that caused telegraph wires to spark, and even allowed unenergized lines to transmit messages. There was no electrical grid then; no phone lines; no satellites. Another CME of that magnitude is certain to occur, perhaps next week, perhaps centuries from now. A comparable event in 2012 narrowly missed the Earth. Few in the know expect anything other than disaster for our massively-wired planet if - when - such a storm hits us. We have built an incredible high-technology world, but it is surprisingly vulnerable to rare but unavoidable events. Hubris, alas, may be our downfall.

But even absent such a high-energy event, Memfault's data shows that "space rays" (to be overly dramatic) do pose a hazard to our systems.

Toothal Tribulations

At first I thought this was a joke... but it's not. Do you need a connected toothbrush? One that can play music? It seems Procter & Gamble feels we do. A $200 version of their Oral B electric toothbrush has Alexa built in. Alas, this must-have product has been discontinued. For the unfortunate who did shell out the big bucks, the Connect app needed to configure the thing ("configure" a toothbrush???) is no longer available, so the device is essentially bricked.

But not to worry: the company now has a version with, of course, AI! One can get a full-featured toothbrush for $0.99, but why not fork over $400 for a unit that, well, brushes your teeth?

I wonder if these devices need time to boot up. Or if a software update mid-brush suspends operation. Is there a customer support line? A 500 page manual?

Failure of the Week

Here's a failure of common sense:

This is a police report about a recent accident involving a Pennsylvanian politician. Note the GPS position, given to 13 digits after the decimal point. That's a resolution of a few nanometers.

Also note that this occurred "12.57 feet north of...". Wikipedia pegs GPS's accuracy at "30-500 cm", or, at best, one foot. Those two digits after the decimal point are noise... and meaningless in a police investigation. I suspect accuracy measured in tenths of a furlong would be plenty.

Roland Lemmers sent this:

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

These jokes are archived here.

You might be an engineer if...

   If your father sat 2 inches in front of your family's first color TV with a magnifying lens to see how they made the colors, and you grew up thinking that was normal

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.

Click here to unsubscribe from the Embedded Muse, or drop Jack an email.