Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 408, October 19, 2020
Copyright 2020 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Embedded Studio cross platform IDE

Tip for sending me email: My email filters are super aggressive. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Quotes and Thoughts

In any project, large or small, each engineer's work is expected to be consistent and transparent so that another engineer can check it - by following its assumptions, logic and computations - for inadvertent errors. Henry Petroski

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Brian Cuthie addresses an issue I think is important with regards to watchdog timers:

Many systems today employ an RTOS of some kind, and therefore end up with multiple threads of execution, so it's important to make sure that the WDT accounts for the proper operation of the entire system. Simply kicking the hardware watchdog in the RTOS's idle loop is generally not sufficient to guarantee that a system is fully functional. A thread stuck in an infinite loop, for instance, would not necessarily cause the WDT to expire. 

A technique I've used for years is to create a variable that holds a bitfield representing each of the running threads, and then only kick the hardware WDT when all the bits are set. It looks something like this (some details omitted):

#define  THREAD1 0x01 // arbitrary thread names here
#define  THREAD2 0x02
#define  THREAD3 0x04
static  uint8_t wdt_accum; 
void  kickWDT(uint8_t thread) {
wdt_accum  |= thread;
if  (wdt_accum == THREADMASK) {
   <do  whatever it is that kicks the hardware WDT>
wdt_accum  = 0;

Using some linker magic you can place the wdt_accum somewhere in uninitialized memory where it can be read at startup to narrow the cause of a WDT event down to the thread involved. Usually that's sufficient to quickly locate the problem. Of course, if the hardware supports steering the WDT event to a non-maskable interrupt you can also save the effective PC, or even some portion of the stack, of each thread to get closer to the point of execution when the WDT expired.

Jeremy Overesch and several others recommend Compiler Explorer:

Reading the comments regarding Assembly in your latest Embedded Muse made me wonder if your readers have heard of Compiler Explorer (https://godbolt.org/).  It is a tool that takes higher level code from C/C++ and other languages, runs the a compiler on it, and shows the assembly results. While it's primarily tailored towards x86, when it comes to C/C++, it has nearly every GCC/Clang flavor, including ARM32&64, MSP430, and RISC-V.  Also, since it's completely open source, it's possible to download and make it work locally with any compiler that outputs assembly.  I've found it to be an invaluable tool to see what the compiler does under the hood and to quickly play with newer language concepts I don't quite understand.

The Sticky Bits blog has an article about using Compiler Explorer with test-driven development.

Freebies and Discounts

This month's giveaway is a LA104 mini logic analyzer. I reviewed it in Muse 406.

LA104 Logic Analyzer

Enter via this link.

Get it Wrong, Pay the Price

In one of Capers Jones' many papers he presents the following:

Defect removal efficiency (DRE) is the percentage of bugs found in all phases of developing a system up to 90 days after initial delivery. A 75% efficiency means that percentage of bugs was removed; another 25% lurk even three months after customer delivery.

What is the cost to fix these defects? There are a lot of numbers bandied about. Remember, these errors are those found in the field, post shipping, and are the most expensive of all. Some data points to something like $5000/shipped bug, but I've worked with customers where the numbers are much, much worse, typically in regulated environments. These post-shipping problems probably involve multiple people, including software engineers, managers, QA, telephone support and even field support. That number neglects the lousy image bad code gives of the vendor, which may ultimately be even more expensive to the business.

The post-compile total number of bugs in firmware typically runs 5 to 10% of the total number of lines of code. Assuming the mean of that, 7.5%, and we find these numbers for different sized projects with each of the three defect removal efficiencies, assuming $5k/fix:

Cost of bugs post-shipping

I can't tell you the number of times people have told me they don't have time or a budget for process improvement. Given that the engineering costs to build a system is $20 to $40 per line of code, fixing fielded bugs in a 100KLOC device at a 75% DRE will consume about a third of all of the money originally spent on engineering the code in the first place.

(Disclaimer: the numbers vary tremendously. Use these to get a feel for the problem rather than an exact number. An impressionistic painting shows the shape and nature of an image; it isn't a high-res photo. These metrics are similarly impressionistic, though that does not diminish their importance.)

Responses to Expanding One's Capabilities

A reader asked how one can go about getting less pigeonholed; widening one's skills and adding value to a career. Several people responded.

Steve Johnson doesn't specialize:

As a Baby Boomer, I found myself behaving much more like an Gen-Xer in that I switched jobs fairly frequently (every 3 years or so) during my early career.  Later, I found employers who were better at offering a breadth of experiences.  

My conscious choice was to alternate between technical, technical management, and customer-facing responsibilities in my job assignments.  This led me from Bell Labs to large pharma companies, with stints at start-ups and small businesses along the way.  At various times I was a software architect, a software development manager, a technical marketing manager, a patent and trademark administrator, an applied mathematician, etc.  

I started with Z80 Assembler (hand-coded through a hex editor), and picked up BASIC, C, PL/M, PL/SQL, Haskell, and F# along the way.  I never advanced beyond "Senior Manager" grade in a company (never interested in doing so), and retired six years ago to a very active post-retirement life doing intellectual property consulting, personal projects with Raspberry Pi, Arduino, Atmel, and CMM2, and board of director positions on community arts organizations.

As Robert Heinlein said, "Specialization is for Ants"

I agree with Richard Prosser about courting startups:

My suggestion would be to look for employment in smaller companies or startups. These tend to require people with a larger range of skills, and ask employees to cover a wider range of roles.

There are dangers though that the company _may_ be more likely to go out of business or _may_ get swallowed by a large company that then imposes its own idea of who does what. Or just grows to become a larger company with specialists.

Nick Soroka has a trifecta of suggestions:

I would suggest they look into: 

  1.  Consulting work, either as a sole proprietor after obtaining their PE or with a product design firm.  I believe there are opportunities to face a wide variety of challenges from different products which they might find interesting.
  2.  Research, specifically things involving material science or chemical engineering.  The future of which will require extensive process control challenges which are highly dependent on both mechanical/electrical/software choices for the design.
  3.  Start their own product design company.  A friend of mine designs peanut dryer controllers for farmers and faces challenges in both mechanical and electrical design.
Why Did You Become an Engineer?

In the last Muse a reader asked why we became engineers. Christopher Eddy's contribution makes me wonder about genetics: can 20k or so genes encode an engineer's personality? Or, as some believe epigenetic factors affected later generations after the Dutch Hongerwinter in 1944-45 could there be a similar effect for nerds? (For a wonderful book on genetics which has some - in my opinion, speculative - discussion of the Hongerwinter and epigenetics, see The Gene: An Intimate History, by Siddhartha Mukherjee). Christopher writes:

At 10 years old, in the 70's, I walked into a Radio Shack and was fascinated.

The first step was resistors and LEDs. The next step was a BCD counter kit, 277-103. It was a steep ride downhill after that.

I used to joke that I knew the catalog numbers for most of the parts and their prices.

The G&T program in junior high school exposed us to programming.. on a Radio shack TRS-80 Model 1 level 2 and a Model 3. TRSDOS anyone?

At the age of 13, I moved to a house with a junk yard nearby. Our local telco was moving from relays to the ESS system, so there were mountains of parts to be had.

In those days, there were also tons of vacuum tube televisions parked by the road, and so I also had lots of fun with them.

When it was time for college, I do not recall taking any other profession seriously. I earned a BSEE from Pitt, but also added the PA P.E. certification soon thereafter.

But at 29, it became crystal clear. I had been adopted, which I did know. And then my maternal side found me. My grandfather had also been an engineer, a P.E., worked for himself, and I had his hair and body type.. all of which I had done before they found me. A former colleague of his even said that I had his mannerisms.

I also heard of his health record, but did not listen carefully. I later tracked his heart history to the letter, and survived two heart attacks. Turns out that sitting for a living is not very good for us.

Nature versus nurture??

Gordon Dick's path was a little different:

Regarding the question "What childhood experiences influenced you to become an engineer". My answer parallels yours to some degree. As a boy I was always "building things" initially out of wood but out of metal later. When I was about 8 or 9 a neighbour coached me on how to make a Morse code transmitter out of a Model T ignition coil. (Very illegal but then I don't think I knew.)

My father was not an engineer, he ran a small town repair shop in SK. However he was very smart and could design and  build things out of metal. Before I was a teenager he had me helping him in the shop. (This was a bit like forced indenture.) However it was there I learned how to repair engines, do simple lathe work, use an arc welder and an oxy/acetylene  welder to braze or weld. I guess it was with him that I learned what a good work ethic was. When I was in my teens before I was allowed to drive I built a gas powered go-kart mostly by myself. After highschool I worked at a natural gas compressor station. It was here that a technical school electronics grad impressed me. This led me to enroll in a technical school electronics program. It was here that one of my instructors impressed me, he was an engineer. That led to enrolling in an EE program and then in an MSc program. From the MSc program I landed an instructor position in a large technical school where I stayed until retiring. It was in the technical college where I learned Assembler programming for 8080, 8088, PIC and HC11 micros. Later I moved to programming micros in C. To this day I still "build things" mostly with micros in them.

To add a twist to these stories, I'll include another personal anecdote. As I mentioned last issue, my dad was a mechanical engineer who spent his career mostly in the space program. He did not fall into the field in the same way as those quoted above. Just before WWII his parents bought a diner just outside of Grumman's gates in Bethpage, NY. With the war Grumman's business boomed and the diner prospered. In senior year of high school his dad promised to set him up with his own diner, which was all my dad wanted. A teacher more or less forced him to take a scholarship exam, pretty much against his will, and he earned a full ride to MIT. So he sort of fell into engineering accidentally.

Failure of the Week

Ray Keefe sends this week's failure. He writes "This is from a running app on my phone. The numbers come good after a bit so it looks like it draws the GUI before it has completed the calculations and must not initialise the variables beforehand or mark them as not ready."

Software Failure

This Week's Cool Product

Bill Schweber has an article in EDN about using bus bars on a PCB to provide low-impedance power rails across a PCB. This is important in higher-power large boards. For example, Storm Power Components sells bars used like this:

This is hardly a new concept. Designing boards for Nova minicomputers in the 1970s we used these as these were 15 x 15" boards: enormous, and with hundreds of TTL ICs quite a few amps were needed. Multilayer boards weren't around back then. The bus bars distributed the massive columbic flow and eliminated board flex.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

These jokes are archived here.

Flon's Law: There is not now, and never will be, a language in which it is the least bit difficult to write bad programs.

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.