Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 459, December 5, 2022
Copyright 2022 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Embedded Studio cross platform IDE

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Quotes and Thoughts

“When you’re talking to Batman on ‘The Tonight Show,’ you’re not really talking to Batman. When you’re talking to an engineer, you’re talking to a real engineer.” Jay Leno

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Matthew Eshleman sent his test for the quality of a development team. It's a lot like the Joel Test:

Here is a list I wrote back in 2020, which has a good bit of overlap with your newsletter today. 
This list has been nicely received lately, and you were my primary source/inspiration on Item 14.

And some informal survey results associated with The Matthew Test, which you might find interesting.

Bertol Lekuona recommended a tool:

I would like to recommend a Source Navigation Tool: SourceTrail

It’s a free and open-source cross-platform source explorer that helps you get productive on unfamiliar source code. It works with C/C++, Python and Java languages.

I have only tried with C source code and get impressed how it makes easy to navigate around the source code. Highly recommended.

Around the Web

Here are a few tidbits I've run into on the Web this week.

A short but fascinating article in IEEE Spectrum is worth your time. It shows, among other interesting facts, how the cost per trillion transistors has declined to the hundred-dollar level. As an old-timer who remembers when transistors cost money this blows my mind.

A trillion of anything is a lot. I did a little calculation: 1 trillion grains of sand weighs 4 million kg. That's 200,000 twenty-five KG bags, which, at Home Depot prices, is over a million dollars... and transistors are made from sand. Pretty amazing.

Here's a quote from that IEEE article: "According to TechInsight’s forecast, the semiconductor industry is on track to produce almost 2 billion trillion (1021) devices this year. That’s more transistors than were cumulatively made in all the years prior to 2017."

Steve Leibson always writes interesting articles, and his (so far) two-part series about early microcontrollers is fascinating. Part 1 and part 2.

I'm beginning to despair about security issues in the embedded space; there are lots of threats, broad attack surfaces, and not a lot of interest on the part of management to proactively address the problem. In fact, management seems determined to build systems that snoop - see the latest Risk Forum posting for notes on how tax companies send your info to Facebook (and who knows how many others), how airports are designed to track us, and much more. Anyway, here's a worthwhile article about building secure systems, but it is a bit lacking in applying known good practices prior to writing the code. While the article's focus on test is indeed important, we'll never get secure or safe systems via testing alone.

How do waterfall and agile compare in terms of implementing unneeded features? One analysis suggests they are about equally bad, though combining the two can net worthwhile results. While I'm a bit skeptical about some of the assumptions used, the article is interesting.

Responses to "Give This to Your Boss"

The article in the last Muse generated a fair number of emails. David Strip writes:

Enjoyed your latest newsletter. I'd like to add a little more ammunition for version control, particularly distributed systems like GIT. Git has no master server or repository, every repository (at least those that are properly synched) have the latest up-to-date code. This means that even a failure to backup the master server doesn't end up in a loss, as there is no master server. Another feature of most VCS is the ability for multiple people to work on the code simultaneously with automated tools to reconcile changes made to the same files.

John Carter suggested:

Your vcs page is very outdated. I absolutely agree VCS is a must have and top priority, but way better DVCS's are available now.

In terms of ease of use Mercurial wins, in terms of universal adoption git has betamaxed it.

> So write or get a firmware standard.

Step zero... Before you even begin talking about coding standards, compiler warnings are now very effective and a far larger win than a coding standard everybody forgets.

Turn on warnings. In gcc, that's -W -Wall -Werror.

If you get a sea of red, most compilers allow fairly fine grained control over warnings, so every day turn one more on, let the team clean up the mess now made visible.

Once you have got that, then start talking about coding standards, but look around for a machine checkable one.

> Chuck Crap

Most software these days is not green fields new product stuff. It's working on existing systems, fixing, extending, handling component obsolescence, ...

Cheapest and best way to cut crap?

Have a proactive deprecation and obsolescence strategy.

Working with your sales and customer support groups, find a feature that is not being used, a product line that is not being sold, a feature that has been superseded --- delete it.

Or split it off into a dead "maintenance on demand only" branch.

Once you have done that, you will find all kinds of code and gnarly tangles are dead as well. Delete delete delete... That's what we have version control for. It makes us brave. Whoops, deleted too much, no problem, it's all still there in the VCS.

> Crunching every parameter revealed the answer: developers imprisoned in noisy cubicles, those who had no defense against frequent interruptions, did poorly.

Remember your opening line....

> With everyone returning to their offices I'm getting a lot of email complaining that any sort of software engineering is going by the wayside in a frantic compulsion to ship ASAP.

Well, a lot of us are finding we're way more productive working from home, especially a hybrid model.

I think we could be even more productive if we learnt a little more about how to do this new thing and adapted our processes to it.

Vlad Z faults Harvard Business:

> I hear from plenty of readers that their bosses just don't "get"
> software. Efforts to institute even limited methods to produce better
> code are thwarted by well-meaning but uninformed managers chanting the
> "can't you just write more code?" mantra.

There was a time when hardware folks did not get the software.  I remember an interview at the company, whose origins were analog, but at the time of the interview, it was a mix with software being a significant component.  The managers did not "get" the software.  But these days, it's all in the past.  If the manager is a EE, the SW development may not get the needed attention due to the bias, but they "get" it.

In my experience, managers do not get the software when they are neither with SW nor EE background.  That's when they don't get it.  Such managers are usually higher up on the ladder and have an MBA.  It happened to me several times, and there was simply no way to explain the engineering issues to them.

I fault Harvard Business School for this situation. It came up with and developed the idea that they can teach a smart individual to manage any business.  All their students need to know is accounting and advertising.  Unfortunately, that does not work because those managers don't listen to the engineers.  I.e., they don't even try to "get" it because they see themselves as better educated, smarter, and able to see the business issues better.  They don't care for the engineering issues because they see it all as noise; accounting and marketing is their main concern.

I am biased, of course, as I am speaking from my engineering experience. 

> It takes your developers 15 minutes, on average, to move from active
> perception of the office busyness to being totally and productively
> engaged in the cyberworld of coding. Yet a mere 11 minutes passes
> between interruptions for the average developer. Ever wonder why
> firmware costs so much?

That is one of the reasons why my last 25 working years I was a consultant.  I had no company meetings, no old software to maintain, no application or field engineers asked me for assistance, and my interactions were limited to a small number of people concerned with my project.  All the employers wanted from me was the result for the given project.  I had some 4 working hours of overlapping time with the rest of the company, and it was perfectly satisfactory for all involved.

BTW, I am an evening type, so my hours were from 10-11 AM onward.

> Finally, did you know great code, the really good stuff, that which
> has the highest reliability, costs the same as cruddy software? This
> goes against common sense.

No, it doesn't.  Well planned and well-managed project does not waste time on chasing bugs all over the place all the time.  A component is tested and is left alone unless something drastic happens.

From Greg Bollendonk:

Requirements are Required - you have covered this in the past.

Requirements represent the contract between stakeholders to define behavior and capability. Requirements are required regardless of development methodology; requirements are never optional. Teams must overcome the (false) belief that "agile allows us to skip basic disciplines and processes"; requirement activity does not mean Waterfall methodology. [ there are many more points in an essay about requirements :-) ]

Larry Marks looks at this from the hardware side:

> Software engineering isn't like building widgets or designing circuit boards.
No, it is not. The cost for errors (even one-wire changes) in chips or boards is many orders of magnitude greater. When I was designing VLSI, the care we took in design and simulation (unit-test) was painstakingly painful. We held programmers in disdain, even in the days of hard ROM, before flash and user-updates.

I've explained it this way, although it's not exactly original with me.
Each day when I go to work, I have 50,000 opportunities for error. If I make just one of them, the whole world falls in on me. "We're gonna have to spin the chip! It will cost $1 million (1980 bucks)! It will take six months to get a fab slot! The I/Os are gonna move! We'll have to spin the board! That's another $200K. We're gonna miss our window into Test and that will set us back another two months! Marketing overstated the volumes for the project and is looking for an opportunity to cut them! They are going to state that we missed the market window of opportunity and cut the volume by 80%! The project will be cancelled; we will all be laid off, and [every finger in the room points at me] it's all your fault!

Fortunately this complete scenario never happened to me, by dint of really painstaking effort. I've watched it, though.

That observation tells us something important about how we partition big projects. Find ways to break big systems down into many small, mostly independent parts. Or at least strip out as much as possible from the huge carcass of code you're planning to generate, putting the removed sections into their own tasks or even separate processors. Give these smaller sections to the superprogrammers. They'll crank out solutions fast.

This suggestion cannot stand without a caveat. In the early 1980s, at a chip design conference, we were taught that the majority of errors occurred at interfaces. As you often point out, English is imprecise and specifications are therefore imprecise. Throughout my career we were never quite able to drive this to zero. Fortunately, with regard to the scenario above, this often meant a minor board change, not a chip change.

Here are a couple of interface errors you've previously mentioned.

  • Galileo (metric or English units?)
  • 737 MAX (why weren't both Pitot tubes used instead of one? Why wasn't data from one validated?)
  • Therac-25 (Why wasn't data from keyboard latched, not continuously overwritten?)
  • NEAR failure (Why wasn't data smoothed or validated?)

I would take this far enough to ask whether you've considered recommending a special flavor of inspection for interfaces only. Get the right people in the room, who wouldn't otherwise be there.

Failure of the Week

From Steven Wheeler:

From Jonathan:

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.


Joke For The Week

These jokes are archived here.

A physicist, a biologist, and a chemist were going to the ocean for the first time. The physicist saw the ocean and was fascinated by the waves. He said he wanted to do some research on the fluid dynamics of the waves and walked into the ocean. He drowned and never returned.

The biologist said he wanted to do research on the flora and fauna inside the ocean and walked inside the ocean. He, too, never returned.

The chemist waited for a long time and afterwards, wrote the observation, "The physicist and the biologist are soluble in ocean water."

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.