Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 426, July 19, 2021
Copyright 2021 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Embedded Studio cross platform IDE

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Quotes and Thoughts

Lehman's first law: A system that is used will be changed.

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Anticipating Errors

Is it a bug? Defect? Error? We're pretty sloppy in the use of these words. I prefer "error" as "bug" sounds like something that snuck in while no one was looking; "error" implies a mistake made by an engineer, generally a more accurate moniker. (It's true the the most insidious errors occur in requirements, which are often elicited before the development team gets involved, but someone is responsible for those errors).

Regardless, there's no doubt mistakes will be made resulting in bugs that no one wants. It seems to me a wise developer will have a strategy for both anticipating these errors and mitigating against their effects.

Anticipating errors means recognizing that we're imperfect and that our code will have mistakes. It implies taking action before the error exhibits a symptom to deal with the possibility of an error. Here are a few thoughts:

  • Detect them as early as possible, as close to the sources as possible. One tool is the aggressive use of the assert() macro. I've covered assert() many times in this newsletter. Studies show that using plenty of these leads to many fewer shipped bugs and a shorter schedule. But there are other tricks that are rarely used. Dan Saks wrote a seminal article about compile-time assertions back in 2005 which is still relevant. And Miro Samek's article is very worthwhile. (Always add a comment indicating why the assert() is there.)
  • Be pessimistic. Assume everything will go wrong. For instance, a null "default" case in a switch statement often indicates excessive optimism.
  • On all interfaces ask what could go wrong. Could a bad argument or one out of range be passed? A null pointer?
  • Anticipate sensor failures. Don't read a thermistor and display a temperature; test to see if the reading makes sense. Many of the Failure of the Week pictures are of thermometers displaying crazy results. In Muse 269 Bill Gatliff had a very thought-provoking riff on this where he argues one should compare sensor readings to the real-world physics of a system to decide if the data is bogus. Had Boeing noted this the 737 MAX accidents probably would not have occurred. I explored the lessons we should learn from those accidents here.
  • On all calculations ask what could go wrong - like an integer overflow or perhaps a loss of normalization with floating point.

Mitigation is a term often found in safety-critical systems. Generally these devices include mitigations against possible problems, like an IV bag being empty or not present. But we should go further and create them for errors/bugs that are not expected operating modes. It's one thing to anticipate an error; but that requires an appropriate mitigation. In many systems the very least of these is a way to log and report the errors. Often that's about all one can do: report and reboot. Or perhaps we can put the system in a degraded safe mode, like the limp-home feature in modern cars. Spacecraft often have a safe mode that orients the vehicle so the solar cells are aimed at the sun and that does little more than receive commands from home.

In my opinion mitigation is one of the holy grails of software engineering. When a calculation overflow occurs deep down in the call stack it can be awfully hard to take corrective action - or to even know what that action should be. Maybe in some cases it's OK to return a random number. Usually, though, that will put the device into some unknown and arbitrary state we'd rather avoid.

Sharpening the Saw

We had three large trees taken down recently. For some time I tried to convince myself that we could fell them but wise wife talked me out of it. Watching the tree service do the work made me realize just how little I know about safely making trees go "timber!" These experts downed the trees with accuracy of a billiard player pocketing a ball. I never tire of watching pros work, especially those in the trades. White collar workers are sometimes subtly biased against the trades, but those folks work with every bit of the skill and precision we employ when we sling bits for a living.

I asked the service to leave the trees in the yard, and have been cutting them up for turning blanks, firewood, and hope to take the best stuff to the sawmill to cut up for lumber. On nice days I run the chainsaw and splitter, which is both fun and a welcome diversion from the keyboard. As soon as a chain starts to dull it gets swapped out for a sharp one. In the evenings I dismantle the saw, clean it, and sharpen the chains. Then the tool is ready for use whenever the opportunity presents itself.

I hate dull tools.

Stephen Covey's seventh habit in "The Seven Habits of Highly Effective People" is "sharpen the saw,"  by which he means one should spend time getting better and improving oneself. It's a pithy bit of wisdom most of us ignore. Dull workers and dull tools cripple teams.

A basic tenant of eXtreme Programming is that everything changes, all of the time. Technologies, tools, processes and people constantly evolve and at times shift suddenly and radically. Unless we're dynamically engaged in self-improvement we'll be run over by these changes and will be at the best inefficient; at worse, obsolete. Dull. Both are career-killers in these hyper-competitive times. The big boss really needs those 8 figure bonuses, and will happily replace your team with any of the many lower-cost alternatives… especially when they can make a better productivity argument.

So I suggest a resolution: spend two hours a week learning something new about building embedded systems. Learn a bit more about your tools so you can exploit worthwhile features you haven't been using. Take classes, on-line or otherwise. Read a technical book every month.

Bob Dylan anticipated Covey decades earlier, when he wrote "he not busy being born is busy dying."

More on Working With Your Hands

Last issue's take on working with your hands prompted many replies and stories. Here are a few of them.

Steve Johnson wrote:

I went to a college prep high school in Chicago, and pretty much aced all of the courses.  While there, I got honors awards in English and Music :-)  For fun, I fabricated model rocket engines on the workbench in the basement of our family home, with steel nozzle molds made by the machinists at the factory my father worked at.

My undergrad college decision was based around where I could pursue music in addition to physics and mathematics, and without much guidance, I chose a place that did OK in all of them, but not spectacularly in any of them.  However, as a poor first-generation college attendee, they waived my tuition and gave me a job in the Physics Department's electronics and machine shop, where I was given the occasional assignment to build teaching circuits, assemble data acquisition equipment, or write analytic programs for their HP desktop calculators and plotters.  Some friends and I inhabited the machine shop during spring break in 1977, and built an R2-D2 using windshield wiper motors, telephone touch-pads, and discarded CB radios.  

Some of the same friends and I rehabbed the university's observatory, rebuilding the telescope and restoring the proper operation of the dome.  Both projects required a bit of machining; the shop included numerous WWII-surplus mills, lathes, etc. as well as a room-sized Van de Graaf generator, boxes of high-power transmitter tubes, and a couple of rusty 55-gallon drums of 1-2 dichloroethylene.  At the same time, a math prof and I built parallel "Digital Group" Z-80 - based pre-S100 computers and got them working.  I also had three summer internships at Argonne National Laboratory, building and using lab equipment for physics experiments.  My senior-year internship was at Gould Corporate Research Laboratory, where I was involved in designing the circuitry for a X-Y plotter product.

My grad school decision was based entirely on which school would give me an assistantship, a 100% pragmatic decision, as I was still broke.  I had a consulting job working on the software for an RCA 1802-based 6-axis optical encoder-driven surveying device (this project ended ignominiously with the deployment of GPS). Two years later, I was out interviewing for jobs with my soon-to-be freshly minted Masters in Computer Engineering.  I had a nice interview at Xerox (this is at the time of the Xerox Alto), but turned it down because it didn't involve any physical prototyping, only programming.  I had a great interview at H-P, where I came up with a very minimal instruction set for a RISC CPU design.  I took a job at Bell Labs, but got bored very quickly as the project I'd been hired for was cancelled between the time I accepted and the time I started.  I quickly transitioned to a medical school's Anesthesia department, where I was very happy with a hands-on job designing, building, and programming a novel monitoring system for use in complex surgeries.  This was the first of many medical devices, and the start of my focus on software quality.

Throughout my career, I've attempted to keep fairly close proximity to the physical manifestation of my (or my group's) work, but this has become more difficult as systems become more complex and larger.  Now, post-retirement, my projects and consulting work are still as tangible as I can manage.  

Bob Dunlop, like many others, was influenced by his father:

I will not claim to be an engineer - at least not by formal education, though I did spend around 10 - 12 years writing software for a major corporation's IT department. I got to the point of working on real applications after a 2 -3 year stint teaching myself to program  in HP Basic on an HP 11C calculator, which eventually led to developing Windows GUI front-ends for "enterprise" SQL DBMS running in a corporate environment. In my case enterprise meant 400 users, trivial by current standards.

My Dad was a Mechanical Engineer by training and education, and his work experience alternated between dynamic seatbelt testing systems (mid-Sixties) and testing/approving automotive lighting/safety systems for the state of California (headlights, turn indicators, etc.). He, with his small cadre of engineers, mechanics and engineering students from U.C. Berkeley, designed, constructed and employed their test-bed systems. I grew-up using tools and building stuff (some actually useful, most ended-up as a "learning experience").

I completely get the evolution of hands-on design and construction, whether hardware or software. Since my high school days (several decades in the past) I've followed a pretty strange career path driven by a combination of insatiable curiosity and A.D.D. In that journey I've counted commercial SCUBA instruction, portrait photography, heavy construction (as journeyman carpenter, and later contractor), software development and finally Nursing as a Critical Care RN and Asst. Prof. of Nursing.

Regrets? Actually not too many, aside from having accumulated little in the way of retirement. The ten years spent as a hospital Nurse saved me from a retirement spent standing on a corner waving signs for real estate. The one area I didn't pursue which occasionally led to twinges of regret was underwater photography/cinematography. I had an opportunity with Al Giddings early on, but chose steadier employment and income to support my young family.

Now fully retired an with an OK retirement income I'm pursuing the arts with fused glass and ceramics - once again a combination of creative design and hand-built items, some practical and some not... Oh, and I still dabble in embedded systems - endless hours of fun, learning, and frustration.

Overall I'm at peace with what I've accomplished and learned over the years.

John Beatty sent this:

A hobby engineer writes:

My father handed down to me, aged 7, his 1925 copy of 'The Boy Electrician' (available on Project Gutenberg). We built from scratch a simple telephone. He was able to source the microphone's carbon granules from a work colleague but the rest we improvised.

In my working life I was a civil engineer working with water - water supply, sewerage, drainage and irrigation. Now I'm fortunate to volunteer on a local canal restoration in my retirement.

These days I build and program simple embedded circuits - but always to solve a practical problem. The satisfaction to be had in buying a small module (usually from China), reading or researching its data sheet and bit-banging a signal to drive it successfully from a home-made pcb is the same satisfaction I had when my first telephone worked.

Recent projects include a control circuit and a data logger to work a vacuum pump which maintains a siphon supplying the canal with water, and an echo water level meter with GSM transceiver.

All done in assembler. No doubt a program in C would be easier for others to follow but then I'd have to rely on libraries written by others with their concomitant vagaries in the use of timers and interrupts - and the libraries introduce another layer of documentation to be digested.

That moment of: Yes! Cracked it! has been my joy throughout my engineering experience.

Tjark van Dijk remembers tubes (valves):

When I was around 10-12, my father lent me his self built tube radio, which was sitting in a thin plywood box, open from the backside so all glowing tubes were visible. As sensitivity was high, I could listen to MW stations around Europe and beyond.
It fascinated me, that my dad could make this. And what he could, I should be able to do as well.

My parents gave me a kit to switch on a lamp or a mini doorbell with switches, pushbutton and battery. Quickly this was too easy.

So I created model train routes. Created my own lamp posts because then it costed no money (model stuff is so expensive, especially at that age). And learned what I could from the library, going in a few months from the children's department to the adult technical section.

Reading Elektor which was still a young magazine. And trying stuff. Making a motor from a cork, magnet, 6 needles (4 for the commutator, 2 as axle) and copper wire. It worked. I learned about arc flashing and how to harvest a carbon pin from batteries so I could cut aluminum paper.
An old engine bobine was nice for creating sparks and shocks. Also I learned about AC and how opposite phase.reduces voltage. So I had 2 train transformers hooked up with a light bulb in between and saw that as I increased the voltage on one transformer, the light dimmed more. And at some point my crocodile hookup wire started to glow and all insulation was gone. So I ended up with an experience and some crocodile clips.

My uncle had a radio and TV store and a repair shop. He let me analyse and repair stuff during holidays. Almost never taking back task once given to me. He also taught me an important service call question. He said, "when someones calls in for a defect, never ask: what is broken or what is wrong. Because then the cliënt will fill in his own 'knowledge' and say 'it's probably a fuse. Ask: what is not working?. Then drill down so the service engineer knows what to repair. No vision, no sound means power supply. Sound but vision is a horizontal line means: vertical deflection defect, and so on".

I love to develop, to create from an idea to something working. In my youth disco was hot and so I made an LED light show. WIth red, yellow and green LEDs, because blue and white were still not invented. When it came to choosing a study, it was EE at university. Soon turned out that this involved very very fundamental theory and only little practical work. So I changed to a Bachelor EE study which suited my interests better. 

From there I started as an intern with a Japanese company which drilled me in development disciplines and testing. CE was not defined, but there were some guidelines like BASEEFA and some others on how to test if a product is rugged enough to be used in an industrial environment. Not completely coïncidentally when CE was defined, it looked quite like the guidelines we already used for years.

Working in a large company was not really my piece of cake. In these 6 years I met only once with a real user of our products. And the work was starting to repeat itself. Yet another new and better version of an existing instrument to be developed. So I started on my own, creating electronic products accompanied with software. To control mechanical systems. Later on it was labelled "mechatronics", a new name for something I did for years already.

I remember clearly when I worked on a truck, sitting next to a 1000 hp diesel engine driving a system that I needed to control. With a multitasking BASIC program on a 16KB microprocessor. Finding out when the control loop would become instable. And when that happens, the diesel next to me was screaming and oscillating. That was when it hit me hard that I love to make and program products that control things. A PC programmer may change a number and then for a million is removed from a warehouse stock. I flip 1 bit and a diesel engine starts. That is where I get my kick out of.

As I hired engineers, my work shifted to management, procurement and sales. Which was not very satisfying, except that in sales I could use my engineering  skills and come back to the office with an order and a concept electronic design.

Currently I work alone again, developing products. And a steady need to learn, grow and create things. Like a greenhouse to grow my own food. With controller watering, ventilation, temperature and light. And a nice corner to sit as well. To think of new issues to solve or improve, to enjoy the work and to see nature do it's work as well. Which in many ways is humbling. 

Never stop learning, always create and make it happen. 

Ed Sutter is another hands-on type::

I didn't get started in much "hands on" stuff till I went in the Army (75-78) but did manage to build a few dog houses and speaker enclosures in my dad's -very small- workroom.

My initial motivation (post Army) was simply survival...

I wanted to know how everything worked so I could fix it (and not pay a repairman). While in college), my sister bought me a Time/Life book called:"How things work in your home (and what to do when they don't)". Between that and several Time-Life book sets I managed to get through life (I'm 64) with just 1 or 2 emergency visits by a repairman (right now I have the motor from my attic ventilator sitting on my work-bench).

I absolutely love the "conquest" of fixing and/or building things.  I also do a fair amount of woodworking (much smaller shop than yours); and I find that half the fun is designing the piece on sketchup prior to actual construction.   I also have an X-Carve CNC and LulzBot 3D printer just for fun (and many custom gifts!).

I'm still doing embedded systems firmware (chose to turn down any management job) and I still enjoy the work.  Things are a "bit" more complex than "back in the day" for sure (I'm currently working with iMXRT1062: 600MHz micro-controller!); and documentation can be quite a frustration; but it still excites me to get something working.

Paul Marshall wrote:

Why did I become an engineer?

When I was about 9 or 10 I would look at the back of radios, TVs, hifi, whatever in the family home. What was inside? How was it doing what it did? Through the vent slots I could see coloured wires, circuit boards, resistors, capacitors (even a valve, or tube as you would have it Stateside, in the old TV). Wow.

I so wanted to get the back off and look; within a year or so I started, practicing on old radios donated by aunts, uncles, neighbours. Then I struck up an acquaintance with a guy my father's age, who had an allotment garden next door. He was a professional Electrical Engineer as well as a keen gardener. I soon learned about voltage, current, amplification, transformers. And SAFETY, which stood me in good stead for the tinkering I was getting into.  

Meanwhile, in school we were starting to consider some of this stuff theoretically in science and physics. My physics tutor was interested in music synthesizers, as I was by this time. We decided to build one. I bought a soldering iron, built circuits on veroboard which never worked properly but that didn't matter. School subject choices came up at 13-14, and I knew exactly what I wanted to do. I took GCE 'O' levels, then in the sixth form, 'A' levels. We did electricity and electronics labs in Physics; I would finish my own at lightning speed, and then go round helping other students with theirs (I was comparatively poor at other topics, however. My Physics tutor once observed that I would probably pass the exam if I stuck to  electricity and electronics questions. I followed his advice and passed).

Then it was off to university to get my bachelor's degree. I spent my third year of the four-year course in industry, working for a company who made video monitors - this was in the mid-80s. Practical applications. The standard product has a 35kHz scan frequency, a customer asks 'can I have 37kHz?' Sure, let's work out how to do that, tweak the circuit, test it, ship it. Make it do thy bidding!

After graduating in 1985, I joined another university as a research associate. I was placed with a company who manufactured hazardous-area electrical equipment. They were beginning to design telemetry and communications into 1100-volt switches for use in coal mines, and this was my first project.

I discovered embedded systems, and became instantly and irreversibly hooked. 8-bit processors, ROM, RAM and I/O. Optical coupling, frequency shift keying, cyclic redundancy checks. And we had to make it all WORK in a harsh environment (spikes, surges, X-rays from vacuum switches etc). Once it worked reliably, we had to do it again in 3300V equipment. And then again at 11kV.   

My two-year research associateship ended, and the company offered me a job as a development engineer. I've been there ever since.

My company moved into control and monitoring systems, which entailed a jump to 16-bit technology, and then variable-speed inverter drives, which took us into 32-bit DSPs and some serious maths (I had to go back to college and take a masters module to figure out what was going on). Along the way, I joined the UK Institution of Electrical Engineers (now IET) and got chartered EE status in 2001. But I've never lost the fascination for the devices and systems which I deal with, or the satisfaction of making something work, or perhaps work differently, in over 40 years of doing this stuff. I play keyboards in a band - wow, I need a box to re-map MIDI data on the fly, so that one keyboard's (fixed) controls can access parameters on a second unit? No problem. DMX lighting control? Yes, I can make that work too. I can't imagine not doing this for both
a living and a hobby.      

Failure of the Week

From Bill Finger:

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

These jokes are archived here.

Elephant, n.: a mouse with an operating system

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.