You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• On Feedback
• Book Review - Embedded Software Timing
• Multitasking During Sex
• Failure of the Week
• Jobs!
• Joke for the Week
• About The Embedded Muse

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Unless you record your estimates and compare them to what actually happened, you will forever be guessing, not estimating. Karl Wiegers

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Sergio Caprille sent an article about handling GPIOs in an ARM Cortex M.

A great, detailed article about spinlocks is here.

Jon Reeves wrote: Quite a few people asked for a Windows version of my Mimicc tool, which I managed to officially release this morning. (Mimicc is a clang-based C/C++ compiler and mocking framework that turns function and method declarations directly into runnable mocks.) You can read the announcement here: https://mimicc.dev/2021/10/26/mimicc-release-0-11-1-rc1/. Or see it in the downloads section here: https://mimicc.dev/#download-all

Analog Devices has a very nice intro to I2C, PMBus and SMBus.

A reader suggested these forum communities: https://www.allaboutcircuits.com/forums/embedded/ and http://discord.gg/embedded 

John Taylor generously offered to send four lucky winners of this month's contest a copy of his book "Patterns in the Machine: A Software Engineering Guide to Embedded Development". This looks to be a really up-to-date volume of interest to all embedded developers.

Enter via this link.

Australian David Warren is not a name most of us know. But in no small way Mr. Warren contributed to air safety. For he invented the “black box,” the cockpit recorders that adorn all passenger aircraft.

Mr. Warren’s father died in 1934, when David was only 9 years old, in one of Australia’s first serious airplane accidents. No doubt that provided some motivation to the young Warren. He inherited his father’s ham radio, which inspired him to learn about technology, as ham radio has done for so many budding engineers.

But his quest to improve air safety by logging events was rejected by everyone. His boss threatened to fire him if he even talked about his brainchild. Finally in 1954 he published a paper on the idea, and had a working unit a few years later.

As so often happens, at about the same time events conspired to turn his fledgling idea into an industry. In 1956 a Super Constellation collided with a DC-7 over the Grand Canyon, killing 128 people. It was, at the time, the worst air disaster in US history. The CAA had been asking Congress for more money for RADARs and other safety equipment; in fact, that very year legislators cut their budget. But the public outcry made Congress relent, and the CAA morphed into the much-better-funded  FAA in 1958. Soon black boxes were required equipment.

By recording the events leading up to an accident, and then relentlessly analyzing the data and other supporting information, officials created a feedback loop. Lessons learned from each accident leads to ever-better safety. While every crash is a tragedy, the ashes give life to others.

In the 50s, pre-recorder, the accident rate for commercial aviation was in excess of 50 incidents that involved death or destruction of the aircraft per million departures. Now the number is practically unmeasurable. Air travel is the safest form of transportation, which is wonderfully ironic considering the extremely hostile environment at 40,000 feet and our utter reliance on a wealth of complex technologies.

One of the most important graphs you'll ever see about feedback. Feedback from black boxes revolutionized air travel.

Wise developers understand the power of feedback loops. Some - not  much - of the code I read includes logging features that store unexpected events for later analysis. Given that we know software is awfully hard to get right, if one has the resources such logging should be de rigor. Alas, it’s not.

Mr. Warren never made a dime on his invention. But generations of travelers owe him a dept of gratitude for it.

Peter Gliwa sent me a copy of his new book Embedded Software Timing. That's a very compelling title as too often we're sort of coding into a black hole in terms of meeting real-time constraints. We have no idea if that code snippet will execute in a microsecond or a week. If the system seems to work, we ship. But are there lurking timing problems? Is there any timing margin? What if the clock's crystal drifts a bit and slows the code down... or what if you're using an RC clock that is as drifty as Sahara sands in a windstorm?

The book's first two chapters are pretty elementary, covering the V-model of software development and the basics of microprocessor operations. Things then pick up. The POSIX and OSEK RTOSes are covered; both are important and often, especially OSEK, poorly understood by developers. I was interested to see a good description of the tradeoffs between cooperative and preemptive multitasking. The former isn't often seen in this business but Peter points out some great uses for it. (Rabbit Semiconductor users will remember Dynamic C which had cooperative multitasking built in.)

Timing terms are covered in detail. While somewhat academic I found this interesting. WCET (Worst Case Execution Time) is pretty common; others less so.

But what about measuring these? Or predicting them? The latter has been hugely frustrating as few tools exist. The book shows the use of one: AbsInt, a great tool that has been around a long time. Peter shows how it can't really give you an absolute WCET but will produce an upper bound. He gives a good analysis of its limitations.

AbsInt is a nice solution, but few developers will be willing to fork out the big bucks it commands. So for most of us the notion of predicting performance before running the code remains elusive.

Simulation is covered well. Alas, tools like Synopsos' Virtualizer are also priced out of reach for many.

The chapter on instrumenting the code to measure WCET and other timing parameters is excellent. Some approaches detailed are old hat; others you may not have seen before.

Chapter 6 is a fun romp through real timing problems developers have encountered and their solutions, including one that saved an OEM €12m.

A few MCU vendors now provide multicore devices that run in lockstep with each other to mitigate against hardware failures. I don't see a lot written about this; this book is one of the few references that provides some valuable insight.

Peter conducts interviews with a number of experts that are interesting and add good context to the book.

The book is very automotive-centric though most of the lessons do apply to embedded work in general. An appendix on AUTOSAR, for instance, isn't useful for most of us, and the section on safety standards is too superficial to provide much insight. (To be fair, the scope of these standards is huge and often baffling).

It's a beautiful tome with lavish color illustrations throughout. Folks working in the automotive field will get a lot out of it. Others perhaps less so. I found it quite fascinating as the subject of firmware timing gets too little attention. And there are just not enough practical solutions and tools extant. This book is sort of an initial foray into an important subject, and I hope as time goes on we find more solutions to building reliable real-time code.

Actress Jennifer Connelly wrote in The Atlantic: “I do like to read a book while having sex. And talk on the phone. You can get so much done.”

Though some people are ardent multitaskers, surely there are some things worth your entire attention.

Are you a multitasker? Do you shave, text and drive at the same time?

Quite a few readers have written about boss demands that they work on two or more projects simultaneously. But evidence continues to emerge that we humans just do not do well when devoting our attention to more than one activity at a time. I've had the pleasure to work with a very smart individual who can listen to a pair of conversations and "get" every nuance of each, but this is a rare quality indeed. Most of us struggle with one, resulting in the all-too-common communications breakdowns between spouses.

My wife and I share an office. When I'm not traveling we spend each day working companionably together in that room. But she knows that when I'm writing or working on a complex problem interruptions are not welcome. During those times my email client is off, and my attention is laser-focused on the task at hand. Hours go by until I'll suddenly realize the day is over. For me, this is an extremely productive  way to work... but I wonder that is an artifact of being old. Are twenty-somethings better at multitasking? Does growing up amid the cacophony of Twitter and texting train one to effectively divide one's attention between multiple activities?

In their seminal book Peopleware DeMarco and Lister demonstrated that interruptions cripple productivity. Developers need a full fifteen minutes to reestablish their focus after an interruption - and the average engineer is bothered every eleven minutes.

Their work, though, is a couple of decades old, predating Hulu, Facebook, YouTube and all of the rest of the e-interrupters. My millennial daughter watches TV while studying. I can't even listen to a rock station on the radio when trying to concentrate as the DJ jabber distracts me no matter how far to the left the volume button is cranked. But having grown up in a household of seven I need some background noise, so listen to classical stations. Lately I stream a station in Austria since the little bit of chatter is in German, which I mostly don't understand. And I never answer the phone while driving as the latter task demands all of my attention.

Doing multiple projects? Gerald Weinburg (Quality Software Management Volume 1, Systems Thinking, 1992) showed that switching between three tasks can incur a 40% context-switch penalty. That's nearly half your work week lost.

But I'm an old guy. What about you? Do you multitask? If so, do you feel productive do so?

Bryan sent this:

And Tim Benson provided this gem,writing "Considering that the oldest person alive was born in 1903, I don’t think anyone can drink here! I wonder if the 19 in the 1900 is hard wired?"

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Technical Terms in Plain English:

1. A NUMBER OF DIFFERENT APPROACHES ARE BEING TRIED
We are still clueless.

2. EXTENSIVE REPORT IS BEING PREPARED ON A FRESH APPROACH TO THE PROBLEM
We just hired three kids fresh out of college.

3. CLOSE PROJECT COORDINATION
We know who to blame.

4. MAJOR TECHNOLOGICAL BREAKTHROUGH
It works only so so, but looks very hi-tech.

5. CUSTOMER SATISFACTION IS DELIVERED ASSURED
We are so far behind schedule the customer is happy just to get it delivered.

6. PRELIMINARY OPERATIONAL TESTS WERE INCONCLUSIVE
The darn thing blew up when we threw the switch.

7. TEST RESULTS WERE EXTREMELY GRATIFYING
We are so surprised that the stupid thing works.

8. THE ENTIRE CONCEPT WILL HAVE TO BE ABANDONED
The only person who understood the thing quit.

9. IT IS IN THE PROCESS
It is so wrapped up in red tape that the situation is about hopeless.

10. WE WILL LOOK INTO IT
Forget it! We have enough problems for now.

11. PLEASE NOTE AND INITIAL
Let’s spread the responsibility for the mistake.

12. GIVE US THE BENEFIT OF YOUR THINKING
We’ll listen to what you have to say as long as it doesn’t interfere with what we’ve already done.

13. GIVE US YOUR INTERPRETATION
I can’t wait to hear this nonsense!

14. SEE ME OR LET’S DISCUSS
Come into my office, I’m lonely.

15. ALL NEW
Parts not interchangeable with the previous design.

16. RUGGED
Too darn heavy to lift!

17. LIGHTWEIGHT
Lighter than RUGGED.

18. YEARS OF DEVELOPMENT
One finally worked.

19. ENERGY SAVING
Achieved when the power switch is off.

20. LOW MAINTENANCE
Impossible to fix if broken.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.