Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 442, March 21, 2022
Copyright 2022 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Embedded Studio cross platform IDE

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

The Embedded Online Conference will take place next month. I'm giving a talk. The cost is $190, but if you register here with the promo code GANSSLE the cost is only $90. This event was a huge success last year.

In the last issue a reader mention Tracy Kidder's book The Soul of a New Machine, which is one of the best tomes about engineers at work. In this case they were building Data General's next great computer. Grant Beattie sent this article which is a follow up to the book twenty years later. A fun read.

Steve Leibson wrote a history of the 8008, the first commercial 8 bit MCU. It's an interesting read.

Quotes and Thoughts

Responding to the last issue's thoughts on abstractions, two readers sent in appropriate quotes.

From John Grant:

> I continue to see products doomed or at least crippled by an excessive
> number of layers.
I call that "lasagne code" -- it can be more impenetrable than spaghetti code.

John Hartman wrote:

A wise programmer I knew used to say:

"There is no problem that can't be solved by another layer of abstraction - except for the problem of too many layers of abstraction."

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Do you use PTC Axeda agent or Axeda Desktop Server? These are in hundreds of IoT devices. Recently some severe security vulnerabilities have been discovered. More info here. Incredibly, one of the problems is a hard-coded logon credential. A list of known devices with this code is here. Strangely the list includes NXP's LPC1768, which is just an MCU.

Using embedded Linux? Better check out CVE-2022-0847 which can give any user root privledge.

Freebies and Discounts

This month's giveaway is a copy of The Embedded Systems Dictionary, signed by yours truly.

Enter via this link.

Built-In Self Tests

Per Söderstam has some intriguing thoughts on BIST:

While the Muse is on the topic of testing I'd like to throw a musing of my own into the embeddedverse about the term BIST, or Built-in Self Testing.

More than once have I studied the customer requirements and nodded in agreement, understanding most of it and being able to assess the scope of the solution... up until the one-liner at the end: "The device shall have built-in self test". At this point my finely crafted mind-model of the solution typically collapses. Not that I don't understand the gist of the term but rather the usually complete lack of context or break-down of the requirement that, for a mixed signal embedded device, could mean whatever. Even more enerving is it when you join a project where the first round of hardware is being put through board bring-up and your hurried colleagues (not necessarily at fault) have solved the BIST design by entering a box named as such in the highest level of software architecture and pointed to the aforementioned requirements clause.

That box will unerringly land in your lap.

I've encountered BIST several times but never found a basic discussion of what it is, why you want it or when you need it.

Not uncommon , the requirement dissolves into nothingness or a quick memory test as the project progresses and the customer seldom complains as they weren't that sure of the requirement's meaning themselves. So, that is my first question to the public: does anyone have references to some good debate on what, why, when and if there are any design principles that one may lean against.

If not, this is as far as I've come:

  •  BIST is like security (and closely related to safety) that to succeed you have to take it into account from the start of your design. Depending on points below, there might (usually) be a need for HW support.
  • You need to have a sound understanding of why (the purpose) of BIST in your particular gadget. Is it a safety/availability/performance issue or is it about maintenance and service.
  • The scope and level. Catch every little glitch or just detect working or not. Locate the specific diode that is broken or point to the general neighbourhood of the system. Continuous or start-up only.
  • BIST is a cross-cutting aspect of your design (as with safety and security) and not a box to be filled in at a later stage.

BIST becomes a property that many parts of the system must support while not being its primary function.

I think there's another aspect to BIST: the datasheet. All too often marketing demands some unexplained form of BIST so it can appear on the product's datasheet. The tests may mean nothing, but are seen as yet another selling point.

When the first IBM PC came out the manual included the BIOS's source code. It started by testing the functionality of each machine instruction, halting if a failure was found. The logic of this baffled me: if instructions were to fail the code would halt or go off into the weeds. Either way the result would be a hung machine. The tests proved nothing.

Today I see plenty of products that run RAM tests. But a RAM failure, especially in an MCU, will almost certainly be massive. A push/pop or call/return in the code that reports the problem will fail. The implication is that these tests have to take place practically at the startup vector, be written in assembly, and avoid all RAM use.

In some instances it makes sense to look for a single event upset, such as from a cosmic ray, or to check the integrity of a data structure on boot. But your run-of-the-mill RAM test is generally pretty silly.

(And most of these are poorly implemented. The common 0x55, 0xaa test simply does not work. See this for details on creating robust tests).

Have you been ordered to include tests that make no sense?

Interrupting Interruptions

For my money one of the most important books about software productivity is DeMarco and Lister's Peopleware. For a decade the authors conducted "coding wars" at a number of different companies, pitting teams against each other on a standard set of software problems. The results showed that, using any measure of performance (speed, defects, etc.) the average of those in the 1st quartile outperformed the average in the 4th quartile by nearly a factor of 3.

Surprisingly, none of the factors you'd expect to matter correlated to the best and worst performers. Even experience mattered little, as long as the programmers had been working for at least 6 months.

They did find a very strong correlation between the office environment and team performance. Needless interruptions yielded poor performance. The best teams had private (read "quiet") offices and phones with "off" switches. Their study suggests that quiet time saves vast amounts of money.

Think about this. The almost minor tweak of getting some quiet time can, according to their data, multiply your productivity by 3x! That's an astonishing result. For the same salary your boss pays you now, he'd get essentially 3 of you.

Too many of us work in a sea of cubicles, despite the clear showing how ineffective they are. It's bad enough that there's no door and no privacy. Worse is when we're subjected to the phone calls of all of our neighbors. We hear the whispered agony as the poor sod in the cube next door tries to work it out with his spouse. We try to focus on our work... but being human the pathos of the drama grabs our attention till we're straining to hear the latest development. Is this an efficient use of an expensive person's time?

Later studies by other researchers found that after an interruption it takes 15 minutes to get into a state of "flow," that Spock-like trance where you're one with the computer. Yet the average developer gets interrupted every 11 minutes.

Yet the cube police will rarely listen to data and reason. They've invested in the cubes, and they've made a decision, By God! The cubicles are here to stay!

This is a case where we can only wage a defensive action. Educate your boss but resign yourself to failure. In the meantime, take some action to minimize the downside of the environment. Here are a few ideas:

  • Wear headphones and listen to music to drown out the divorce saga next door.
  • Turn the phone off. If it has no "off" switch, unplug the damn thing. In desperate situations attack the wire with a pair of wire cutters. Remember that a phone is a bell that anyone in the world can ring to bring you running. Conquer this madness for your most productive hours.
  • Know your most productive hours. I work best before lunch; that's when I schedule all of my creative work, all of the hard stuff. I leave the afternoons free for low-IQ activities like meetings, phone calls, and paperwork.
  • Disable the email. It's worse than the phone. Your two hundred closest friends who send the joke of the day are surely a delight, but if you respond to the email reader's "bing" you're little more than one of NASA's monkeys pressing a button to get a banana.
  • Put a curtain across the opening to simulate a poor man's door. Be sure  others understand that when it's closed you are not willing to hear from anyone unless it's an emergency.

The ultimate irony of cubicles is that shortly before he died in 2000, Robert Propst railed against cubes, calling them "monolithic insanity."

Robert Propst invented the Action Office, which was eventually perverted into the cubicle.

NRE v. Recurring Costs

Sometimes we engineers forget that engineering is a business endeavor. Our role is simple: help the company make a profit. Everything else is secondary. Without profits a company fails. With them it prospers.

It's critical we make engineering tradeoffs that facilitate that profit. Non-Recurring Engineering (NRE) costs (the price of developing a product) has to be balanced against other factors, like the cost to manufacture the widget. Saving $50k of engineering effort by adding $5 of parts in a system whose product lifecycle won't exceed, say, 1000 units, is often a sound engineering and business decision. Conversely, it can make sense to add a lot of effort to removing a few bucks from a high-volume product.

These judgments cannot be made without a lot of information about costs and volumes, which too few of us are privy to. This means we're operating in an information vacuum which makes it impossible to make the best decisions for the company. It's like driving with your eyes closed. Sure, you might get there, but probably won't.

A correspondent who wishes to remain anonymous sent an email recently which sparked my interest. He wrote: "Our finance department recently did us a great service by identifying the breakeven point in a development cost vs. product cost tradeoff analysis. This breakeven took into account lost opportunity cost in the higher development cost scenario, lost profit in the higher product cost scenario, gross vs. net revenue, and profit targets. The result was a guideline that our teams can tailor to their program (since the tradeoff is dependent on a particular products lifetime sales volumes)."

Wow! This is a highly disciplined outfit which has crossed traditional departmental barriers to give the engineers the information they need to build products optimized for feeding the bottom line.

You'd think more organizations would adopt such a gestalt approach to profits. Yet I suspect few do. By walling this information off from those who need it most the odds of creating a successful product or company wanes.

Instead, all we hear are the screams about schedules.

What about your company? Do you get these sorts of briefings?

Failure of the Week

From Alan Altman:


From Anthony:

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

These jokes are archived here.

From Clive Souter:

The person who invented predictive typing has died.

May he restaurant in peace.

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.