Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 434, November 15, 2021
Copyright 2021 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Quotes and Thoughts

Unless you record your estimates and compare them to what actually happened, you will forever be guessing, not estimating. Karl Wiegers

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Sergio Caprille sent an article about handling GPIOs in an ARM Cortex M.

A great, detailed article about spinlocks is here.

Jon Reeves wrote: Quite a few people asked for a Windows version of my Mimicc tool, which I managed to officially release this morning. (Mimicc is a clang-based C/C++ compiler and mocking framework that turns function and method declarations directly into runnable mocks.) You can read the announcement here: https://mimicc.dev/2021/10/26/mimicc-release-0-11-1-rc1/. Or see it in the downloads section here: https://mimicc.dev/#download-all

Analog Devices has a very nice intro to I2C, PMBus and SMBus.

A reader suggested these forum communities: https://www.allaboutcircuits.com/forums/embedded/ and http://discord.gg/embedded 

Freebies and Discounts

John Taylor generously offered to send four lucky winners of this month's contest a copy of his book "Patterns in the Machine: A Software Engineering Guide to Embedded Development". This looks to be a really up-to-date volume of interest to all embedded developers.

Enter via this link.

On Feedback

Australian David Warren is not a name most of us know. But in no small way Mr. Warren contributed to air safety. For he invented the “black box,” the cockpit recorders that adorn all passenger aircraft.

Mr. Warren’s father died in 1934, when David was only 9 years old, in one of Australia’s first serious airplane accidents. No doubt that provided some motivation to the young Warren. He inherited his father’s ham radio, which inspired him to learn about technology, as ham radio has done for so many budding engineers.

But his quest to improve air safety by logging events was rejected by everyone. His boss threatened to fire him if he even talked about his brainchild. Finally in 1954 he published a paper on the idea, and had a working unit a few years later.

As so often happens, at about the same time events conspired to turn his fledgling idea into an industry. In 1956 a Super Constellation collided with a DC-7 over the Grand Canyon, killing 128 people. It was, at the time, the worst air disaster in US history. The CAA had been asking Congress for more money for RADARs and other safety equipment; in fact, that very year legislators cut their budget. But the public outcry made Congress relent, and the CAA morphed into the much-better-funded  FAA in 1958. Soon black boxes were required equipment.

By recording the events leading up to an accident, and then relentlessly analyzing the data and other supporting information, officials created a feedback loop. Lessons learned from each accident leads to ever-better safety. While every crash is a tragedy, the ashes give life to others.

In the 50s, pre-recorder, the accident rate for commercial aviation was in excess of 50 incidents that involved death or destruction of the aircraft per million departures. Now the number is practically unmeasurable. Air travel is the safest form of transportation, which is wonderfully ironic considering the extremely hostile environment at 40,000 feet and our utter reliance on a wealth of complex technologies.

Airline crashes v black boxes

One of the most important graphs you'll ever see about feedback. Feedback from black boxes revolutionized air travel.

Wise developers understand the power of feedback loops. Some - not  much - of the code I read includes logging features that store unexpected events for later analysis. Given that we know software is awfully hard to get right, if one has the resources such logging should be de rigor. Alas, it’s not.

Mr. Warren never made a dime on his invention. But generations of travelers owe him a dept of gratitude for it.

Book Review - Embedded Software Timing

Peter Gliwa sent me a copy of his new book Embedded Software Timing. That's a very compelling title as too often we're sort of coding into a black hole in terms of meeting real-time constraints. We have no idea if that code snippet will execute in a microsecond or a week. If the system seems to work, we ship. But are there lurking timing problems? Is there any timing margin? What if the clock's crystal drifts a bit and slows the code down... or what if you're using an RC clock that is as drifty as Sahara sands in a windstorm?

An aside: On margins. My dad was a mechanical engineer. He told a story of how, at Grumman in the 1950s, he was working on the F11-F fighter airplane. The Navy ran a test to try and break the tail section. It was too strong; the structure just would not fail. The Navy was unhappy as that extra strength meant the aircraft was heavier than it needed to be. When a tail-mounted antenna snapped they reluctantly accepted the plane. There was an excess of design margin in the tail.

Yet too often we don't have enough, or even any, margin in our designs. If costs and other parameters permit, when designing a circuit using ±5% resistors, I try and squeak in a bit more margin. What if a batch of resistors is slightly out of spec? Suppose a user brings the device outside when it's a bit colder than anticipated or guaranteed? I'd prefer things work rather than be on the hairy edge of failure. The same goes for timing in firmware. "Works" just isn't good enough if there's only a ns of wiggle room before the code fails.

We live in a golden age of electronics and embedded where even a teenager can make systems that work. Professionals, however, design products that are robust over temperature, humidity and the like. We add margin for the sake of reliability.

The book's first two chapters are pretty elementary, covering the V-model of software development and the basics of microprocessor operations. Things then pick up. The POSIX and OSEK RTOSes are covered; both are important and often, especially OSEK, poorly understood by developers. I was interested to see a good description of the tradeoffs between cooperative and preemptive multitasking. The former isn't often seen in this business but Peter points out some great uses for it. (Rabbit Semiconductor users will remember Dynamic C which had cooperative multitasking built in.)

Timing terms are covered in detail. While somewhat academic I found this interesting. WCET (Worst Case Execution Time) is pretty common; others less so.

But what about measuring these? Or predicting them? The latter has been hugely frustrating as few tools exist. The book shows the use of one: AbsInt, a great tool that has been around a long time. Peter shows how it can't really give you an absolute WCET but will produce an upper bound. He gives a good analysis of its limitations.

AbsInt is a nice solution, but few developers will be willing to fork out the big bucks it commands. So for most of us the notion of predicting performance before running the code remains elusive.

Simulation is covered well. Alas, tools like Synopsos' Virtualizer are also priced out of reach for many.

The chapter on instrumenting the code to measure WCET and other timing parameters is excellent. Some approaches detailed are old hat; others you may not have seen before.

Chapter 6 is a fun romp through real timing problems developers have encountered and their solutions, including one that saved an OEM €12m.

A few MCU vendors now provide multicore devices that run in lockstep with each other to mitigate against hardware failures. I don't see a lot written about this; this book is one of the few references that provides some valuable insight.

Peter conducts interviews with a number of experts that are interesting and add good context to the book.

The book is very automotive-centric though most of the lessons do apply to embedded work in general. An appendix on AUTOSAR, for instance, isn't useful for most of us, and the section on safety standards is too superficial to provide much insight. (To be fair, the scope of these standards is huge and often baffling).

It's a beautiful tome with lavish color illustrations throughout. Folks working in the automotive field will get a lot out of it. Others perhaps less so. I found it quite fascinating as the subject of firmware timing gets too little attention. And there are just not enough practical solutions and tools extant. This book is sort of an initial foray into an important subject, and I hope as time goes on we find more solutions to building reliable real-time code.

Multitasking During Sex

Actress Jennifer Connelly wrote in The Atlantic: “I do like to read a book while having sex. And talk on the phone. You can get so much done.”

Though some people are ardent multitaskers, surely there are some things worth your entire attention.

Are you a multitasker? Do you shave, text and drive at the same time?

Quite a few readers have written about boss demands that they work on two or more projects simultaneously. But evidence continues to emerge that we humans just do not do well when devoting our attention to more than one activity at a time. I've had the pleasure to work with a very smart individual who can listen to a pair of conversations and "get" every nuance of each, but this is a rare quality indeed. Most of us struggle with one, resulting in the all-too-common communications breakdowns between spouses.

My wife and I share an office. When I'm not traveling we spend each day working companionably together in that room. But she knows that when I'm writing or working on a complex problem interruptions are not welcome. During those times my email client is off, and my attention is laser-focused on the task at hand. Hours go by until I'll suddenly realize the day is over. For me, this is an extremely productive  way to work... but I wonder that is an artifact of being old. Are twenty-somethings better at multitasking? Does growing up amid the cacophony of Twitter and texting train one to effectively divide one's attention between multiple activities?

In their seminal book Peopleware DeMarco and Lister demonstrated that interruptions cripple productivity. Developers need a full fifteen minutes to reestablish their focus after an interruption - and the average engineer is bothered every eleven minutes.

Their work, though, is a couple of decades old, predating Hulu, Facebook, YouTube and all of the rest of the e-interrupters. My millennial daughter watches TV while studying. I can't even listen to a rock station on the radio when trying to concentrate as the DJ jabber distracts me no matter how far to the left the volume button is cranked. But having grown up in a household of seven I need some background noise, so listen to classical stations. Lately I stream a station in Austria since the little bit of chatter is in German, which I mostly don't understand. And I never answer the phone while driving as the latter task demands all of my attention.

Doing multiple projects? Gerald Weinburg (Quality Software Management Volume 1, Systems Thinking, 1992) showed that switching between three tasks can incur a 40% context-switch penalty. That's nearly half your work week lost.

But I'm an old guy. What about you? Do you multitask? If so, do you feel productive do so?

Failure of the Week

Bryan sent this:

And Tim Benson provided this gem,writing "Considering that the oldest person alive was born in 1903, I don’t think anyone can drink here! I wonder if the 19 in the 1900 is hard wired?"

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Picomole is creating breath analysis medical equipment. We are looking for a firmware engineer to work on our breath sampler. The position is in Moncton, NB, Canada. We are looking for somebody with Cortex-M related experience. Apply at hr@picomole.com, cheers!

Timesys, headquartered in Pittsburgh, PA, is an innovative embedded software and cybersecurity company with more than 20 years of experience in the field. We build, secure, and maintain open source embedded software platforms. We're hiring for a variety of positions at all levels: Technical Program Manager, Lead Embedded Software Engineer, Senior Embedded Software Engineer, Embedded Software Engineer, Senior Software Developer (Full-Stack), and Software Developer (Full-Stack). Remote candidates are being considered for senior positions. You can view the full list of positions and more details here.

Varda Space Industries is moving industry into space and bringing real products back to Earth. We are seeking someone to join us in the role of Embedded Systems Engineer, developing low-level firmware for our avionics platform. You will write firmware in C++ in an RTOS environment to control spacecraft control and manufacturing functions. We also have many other Job Openings

Aqua Robur is an award-winning tech company in Gothenburg, Sweden, on its journey to become an important part of every city’s water infrastructure.
We are looking for a talented embedded developer to join our team to boost the development of our IoT devices. In this role you will work  with Zephyr RTOS and LPWAN radio modems such as NB-IoT, LTE-M and LoRa. Apply on the link or contact our CTO.

AppHarvest has an immediate full time, direct hire opening for a Principal Firmware Engineer in Burlington, MA (not remote). This is for a robotics team that is in pre-production research stages. Interested candidates should email their resume / questions to jeanne.petrangelo@appharvest.com.
Needed experience:
* RTOS and bare metal coding in C / C++
* Communication protocols (i.e. I2C, SPI, UART, CAN, Ethernet)
* Distributed systems
* Build systems (i.e. Make, Cmake)
* The ability to make nuanced priority decisions to balance the need to deliver rapid prototype iterations with the need for reliability

Desired experience:
* Bash scripting, Linux
* Automated unit testing

Joke For The Week

These jokes are archived here.

Technical Terms in Plain English:

We are still clueless.

We just hired three kids fresh out of college.

We know who to blame.

It works only so so, but looks very hi-tech.

We are so far behind schedule the customer is happy just to get it delivered.

The darn thing blew up when we threw the switch.

We are so surprised that the stupid thing works.

The only person who understood the thing quit.

It is so wrapped up in red tape that the situation is about hopeless.

Forget it! We have enough problems for now.

Let’s spread the responsibility for the mistake.

We’ll listen to what you have to say as long as it doesn’t interfere with what we’ve already done.

I can’t wait to hear this nonsense!

Come into my office, I’m lonely.

Parts not interchangeable with the previous design.

Too darn heavy to lift!

Lighter than RUGGED.

One finally worked.

Achieved when the power switch is off.

Impossible to fix if broken.

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.