Click here to go to the on-line version The Embedded Muse 484

The Embedded Muse Logo The Embedded Muse
Issue Number 484, February 19, 2024
Copyright 2024 The Ganssle Group

Editor: Jack Ganssle,

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Gang Programming Solution

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded" in the subject line your email will wend its weighty way to me.

Quotes and Thoughts

"Time pressure is probably the foremost reason behind the emergence of bulky software. The time pressure that designers endure discourages careful planning. It also discourages improving acceptable solutions; instead, it encourages quickly conceived software additions and corrections. Time pressure gradually corrupts an engineer’s standard of quality and perfection. It has a detrimental effect on people as well as products." Niklaus Wirth

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

  I'll be speaking at the upcoming Embedded Online Conference. Muse readers can use promo code GANSSLE24 to get a $200 discount if used by February 15, and a $100 discount if used after Feb. 15 and before March 31st.

Phil Koopman has an interesting article about the problems with test as a way to insure correctness. Recommended.

Filip Kubicz wrote:

I noticed that in the page with tools that you maintain, in diff tools section there is no mention of Meld.

Meld is the greatest diff tool that I have been using and saved me from a lot of trouble when resolving conflicts over the years. To integrate it with git (git mergetool) whenever I need to compare conflicting versions, it's enough to follow this instruction:

Several people wrote with the sad news that Niklaus Wirth has died. Chuck Chester sent this - apparently he was an embedded guy as well as a CS pioneer. One of his creations was ALGOL, which infuriated a lot of people, but it was the first structured language I used, and I found it more satisfying than the FORTRAN we were required to use at the University of Maryland.

Engald's GCHQ has released some never-seen-before pictures of the Colossus, a WWII code-breaking machine that is thought to be the first real electronic digital computer. There's even a fragment of the schematic. You won't find a single transistor in this baby, as those were still several years in the future when it was designed.

Dysfunctional Teams

A reader who wishes to be anonymous poses an interesting question:

One topic I’m eager to hear others’ opinions on is the importance of team. I work for a rather prestigious engineering company. The job security is good, projects are challenging, and plenty of room for corporate growth. As a young engineer (< 4 years of industry experience), these are attractive qualities in a company.

However, I'm having difficulties with the team I work with. My managers are good, the hardware engineers are good, but my software engineer colleagues are lacking. As we all know, working with difficult people is inevitable at every company. Some people couldn't care less about software quality, not pulling their weight, playing politics, etc... This isn't what I have an issue with, people will be people. I have my own weaknesses, bias, quirks, etc... We are human after all.

The problem I'm facing is that the team as a whole is dysfunctional. It's not one difficult individual, it's more than half of the software engineers. The oldest on the team is in their early thirties while everyone else (including myself) is in their mid to late twenties. There is no senior software engineer or manager per se. Majority of them lack maturity. If I had a dollar for every minute spent in a meeting where the team talked about everything but the subject at hand, I'd be able to retire (at 26!). I'm tired of the lack of care and focus. I look at the hardware engineers in my group. They are a cohesive, focused group. I would like to remain steadfast and outlast the problem members of the team, but I work in a fast paced, intense industry. I'm foreseeing 60-70 hour weeks ahead because of the lack of team proactiveness. I'm not willing to sacrifice my wellbeing for something that could be avoided. I try my best to do my part and influence others to do so, but I'm struggling to see it through. It's gotten to the point where I've given up on most of them.

Experienced engineers are in demand and I feel that I am at a point where I could be successful at a different company. But the location, projects, and career advancement opportunities are hard to beat.

This leads to my discussion point. Those that have had similar experiences or insight, how important is having a good team chemistry? Those that ended up leaving or staying, what was your experience?

Any advice?

More on Branchless Programming

Some comments on branchless programming garnered more dialog. Charles Manning wrote:

A response to Louis Bertrand's response to what I wrote about branchless programming.
Louis, first off I want to clarify that I am not proposing people use branchless programming methods. As you point out they make the code harder to read.

The real purpose of what I wrote was a musing (fitting for the Embedded Muse) as to whether branchless processing could be seen as a way to reduce the complexity due to removing branches. I think that even if the compiler was to produce branches under the hood then that would be the case, according to the definition of complexity. Certainly the obfuscation of branchless programming is unlikely to make  the code more reliable.

I have only used branchless programming in one circumstance in real live production code and that was in an ARM NEON SIMD coprocessor which was computing 8x16 bit values in parallel. Any branching would have totally wrecked the SIMD "flow", so branchless is often the way to go with SIMD and similar (vector processing etc). This code was written with intrinsics (ie effectively assembly language) so I knew exactly what was going to be generated.

As I wrote, this branchless processing is really only a potential benefit in long pipelines (even then branch predictors likely would take care of most of the damage). An ESP32 would seem to have  a 5 or 7 stage pipeline which is not very long, so I would doubt it could win much, if at all,  from branchless programming. On the flip side, an ESP32 probably doesn't have a very fancy branch predictor so who knows.

There are some devices which should be able to do branchless programming of that form quite well, such as RISC V which has the slt and similar instructions which set the destination register to 1 or 0 depending on the condition, making them ideal for branchless workflows.
For example the code snippet   greater = (a < b) * a + (a >= b) * b could potentially compile to

    slt  t0, a0,  a1     # t0 = 1 if a0 < a1, else 0
    xori t1, t0,  1      # t1 = 0 if a0 < a1, else 1
    mul  t0, t0, a0      # t0 =  t0 * a0
    mul  t1, t1, a1      # t1 = t1 * a1
    add  a0, t0, t1      # a0 holds  result  of t0 + t1

As you rightly point out, many compilers (eg. gcc) generate the slower branching code anyway. In what I saw, the RISC V code that was generated ended up being slower and larger: and still has two branches!

    li  a4, 0
    bge a0, a1, skip0
    mv  a4, a0
    li  a5, 0
    blt a0, a1, skip1
    mv  a5, a1
    add a0, a4, a5

That's worse than just going with  greater = (a < b) ? a : b which should compile to just two instructions, one of which is a branch:

  bge a0, a1, skip
  mv  a0, a1

LESSON LEARNED: Compilers are tested and developed around commonly seen code. Getting clever might just clever you into a hole!

Jonathan Harston commented:

Branchless testing? Does a subroutine return count?

         LD  A,B
         CP  C
         RET NC ; B>=C, A=B
         LD  A,C
         RET    ; C>B,  A=C


; inline
         CMP   R1,R2
         MOVCS R0,R1 ; R1>=R2, R0=R1
         MOVCC R0,R2 ; R2>R1,  R0=R2


I'm amazed at how few teams do any measurements, suggesting few have any idea if they are improving or getting worse. Or if their efforts are world class, or constitute professional malpractice.

Two of the easiest and most important metrics are defect potential and defect removal efficiency. Capers Jones, one of the more prolific, and certainly one of the most important, researchers in software engineering pioneered these measurements.

Defect potential is the total number of bugs found during development (tracked after the compiler gives a clean compile; ignore the syntax errors it finds) and for the first 90 days after shipping. Every bug reported, every mistake corrected in the code, counts. Sum this even for those that Joe fixes while he is buried in the IDE doing unit tests. No names need be tracked; this is all about the code, not the personalities.

Defect removal efficiency is simply the percentage of those removed prior to shipping. One would hope for 100% but few achieve that.

These two metrics are then used to do root cause analysis: why did a bug get shipped? What process can we change so it doesn't happen again? How can we tune the bug filters to be more effective?

Doing this well typically leads to a 10x reduction in shipped bugs over time. Here's some data from a client I worked with:

Over the course of seven quarters they reduced the number of shipped bugs by better than an order of magnitude by doing this root cause analysis.

What are common defect potentials? They are all over the map. Malpractice is when we ship 50 bugs/1000 lines of code (KLOC). 1/KLOC is routinely achieved by disciplined teams, and 0.1/KLOC by world-class outfits.

According to data Capers Jones shared with me, software in general has a defect removal efficiency of 87%. Firmware scores a hugely better 94%. We embedded people do an amazingly good job. But given that defect injection rates run 5-10%, at a million LOC 94% means we're shipping with over 3000 bugs.

What are your numbers? Do you track this, or anything?

Failure of the Week

Max Proskauer sent this:

And this is from Brett Garberman:

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

These jokes are archived here.

From Charlie Moher: How was copper wire invented ? Two Scotsman fighting over a penny.

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.

Click here to unsubscribe from the Embedded Muse, or drop Jack an email.