You may redistribute this newsletter for non-commercial purposes. For commercial use contact firstname.lastname@example.org. To subscribe or unsubscribe go here or drop Jack an email.
Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.
I'll be speaking at the Embedded Online Conference, which takes place from May 17-20. One talk that sounds particularly interesting is by Steve Scandore, lead software person on the Mars Perseverance Rover. Register with code GANSSLE90 and get $100 off.
Jack's latest blog: a review of Marvelous Magnetic Machines
|Quotes and Thoughts
To expect the unexpected shows a thoroughly modern intellect. Oscar Wilde, who was not thinking about software engineering, but is a pretty good mantra for us.
|Tools and Tips
Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.
I reviewed the DMMCheck a while back. This is a little PCB that has precisely-calibrated voltages and resistance values that you can use to check the accuracy of your DMM. It's a great little product that I use regularly. Alas, it is no longer available, but the company has a higher-priced ($123) version with more features called the DMMCheck Plus. It sure is cheaper than sending your gear to a cal lab, if you don't need official calibration stickers.
|Freebies and Discounts
The speaker lineup for the Embedded Online Conference is pretty amazing! Sign up with promo code GANSSLE90 and wonderful things will happen like reversal of male pattern baldness, a guest spot on Teen Vogue magazine, and a boost of what JFK called "vim and vigor." It will also get you $100 off the $190 registration fee (which goes up to $290 May 1).
|On Showing Up
I think it was Woody Allen who said that 80% of life is just showing up. This might be the most important lesson in business, especially small businesses like consultants.
We had several very dead, very large trees threatening the house and barn last year. They need to come down, so I called three arborists for quotes, leaving voicemails.
One called back within an hour and was here that very afternoon. He gave me a quote that day.
Another texted back that he'd be here that evening. Later, he delayed the time. He didn't show up. The next day he texted an apology and a new time. No show.
Another apology, and another time. No show. I finally texted back that I don't care to work with people who don't respect my time.
The third's receptionist called and said I'd hear from a crew member the next day. A week went by.
Needless to say, the job went to the responsive first person. Now, this is a $6000 contract, which is not chicken feed. The other two lost out because they couldn't be bothered to show up.
That is a cardinal sin in business… and in life. "The dog ate my homework" might work in school, but not in adult life.
A lot of small outfits contact me for business advice. The first rule of business is to be responsive. Be easy to do business with. Yet too many just can't learn that lesson.
How many companies start a "weekly" blog or "monthly" newsletter and then start missing posting dates? Lots. I bet most. Deadlines are a pain, but a commitment is a commitment, and missed, is a strike against the company.
I know of only two embedded consulting companies that reliably keep up with their newsletters month after month. Two, out of perhaps thousands.
As a consumer of services, how can I give my business to an outfit that can't meet even these simple pledges?
Are you a consultant? If so, is your web site always up to date? Do you follow up with leads that same day? Are your company brochures complete, well written, and current?
If you're an employee, is your resume a living document you work on whenever there's new data or a change in status? Just as milk in the fridge has a shelf life, so does your resume. A mad scramble to update the resume when there's a crisis leads to forgotten information and poor editing.
And editing is critical. I once received a resume which included:
... it then went on and listed the twelve jobs this person held in the previous seven years.
|The Common Weakness Enumeration
The Common Weakness Enumeration is a list of unfortunately-common problems compiled by Mitre corporation and community members that developers should be aware of. Hundreds of weaknesses are detailed. Many are well-known
Some are pet peeves of mine, like CWE-1116 "Inaccurate Comments." If the comments and the code disagree, either the developer was not sure what he was trying to accomplish, or he was just lazy. Comments are important, and a craftsman who takes pride in his work insures they are correct.
Some, like CWE-615 "Inclusion of Sensitive Information in Source Code Comments" seem rather bogus to me. The cited weakness is that hackers can use these comments as clues to find ways to reverse engineer the code. Yet comments are not included in binaries; if the source is available, the attackers already have the keys to the kingdom.
Then there are weaknesses like CWE-1043 "Data Element Aggregating an Excessively Large Number of Non-Primitive Elements". An example might be malloc()s in a loop. Surely, though, there's nothing inherently wrong about this, despite a potential for misuse. This is a case where experience can triumph over a rule.
In some cases more guidance would be useful. CWE-1121 "Excessive McCabe Cyclomatic Complexity" is a valid concern. But what does "excessive" mean? The CWE does not say… and indeed there's not a tremendous amount of consensus about this. Typically a max of 10-15 is accepted. Many academics are absolutists: a function exceeding a complexity of X must be rewritten. I disagree. Each case in a switch statements increments the complexity, and there are times a big switch is the clearest way to express a complicated idea.
How do you feel about CWE-1123 "Excessive Use of Self-Modifying Code"? Once upon a time self-modifying code was unavoidable in some applications (in Assembly language) but I can't imagine how that would work in C. One of my Linux Bash scripts does generate files of Bash commands that are then executed, so I suppose that is self-modifying code. But this is so rare it's a unicorn event. I'd change "excessive use" to "any use without a heck of a good reason."
Hardware people don't get a pass. There is an entire section of hardware weaknesses. Though some of these enumerated problems really should be categorized as software issues, others are not. An example: CWE-1319 "Improper Protection against Electromagnetic Fault Injection".
For those who want a "top ten problems", this list is CWE's 25 most important issue. And they are indeed important, like not validating inputs and NULL pointer dereferences. Even if tthe data is in-range check to see it make sense.
Take a gander at the CWE. You might learn something.
My very first engineering class in college had some silly name like "Engineering Fundamentals." I don't remember much about it, other than that we were shown a video of the Tacoma Narrows bridge failing in 1940, four months after it opened. Decades later I'm struck that this disaster was used to educate hundreds of thousands of engineers about the perils of that particular failure mode.
We can learn from the disasters of other engineers.
We must learn from the disasters of other engineers.
I have a huge file of disasters in the embedded field and share some of those in my seminars, as only a fool would want to repeat mistakes made in the past. Your errors can teach others things to avoid.
As mentioned above, one weakness in the CWE is not validating inputs. One of the most important lessons we should have learned in recent years - one who importance I can't stress enough - is from the two 737 Max crashes. This is the peril of believing sensor data. The angle-of-attack sensor started giving readings that were valid, but crazy when compared to their airplane's physics or other inputs. Yet the computer accepted those as legit. I have written about this before, but feel this lesson is so profound it needs to be shouted from the rooftops.
Sensors fail. That should be a meme burned brightly into our psyches. But a sensor can fail and produce valid but bogus data.
Consider this excerpt from the flight data recorder on the Ethiopian Airlines crash:
The AOA (angle of attack) sensor went from zero to 75 degrees in under one second. That in itself is unbelievable, and the software should have rejected it. It's unlikely a commercial airliner would suddenly rotate so abruptly.
Next, note that the altitude didn't change; if the aircraft were pointed up, it would probably go up.
Further, the airspeed remained mostly constant. At a 75 degrees up angle airspeed would have dissipated faster than a politician backpedaling on a campaign promise.
The flight data recorder had 1790 channels of data fed to it. One gave an absurd value; many of the others contradicted that. Had the software evaluated the AOA sensor against the other channels it could have behaved more reasonably. Even absent the other data, comparing the 75 degree rotation in less than a second to a physical model of what a commercial aircraft can do would have sounded alarms.
Interestingly, the data was in range - it was a value that presumably could have been produced by that sensor. A simple out-of-range test would not have helped. Instead, the sensor data should be compared to other inputs, and to a common sense "come on, a two hundred ton aircraft is not going to rotate 75 degrees in a second."
The moral for all of us embedded people: Always compare sensor data to the physics and to other inputs. If it seems odd, respond appropriately.
|Failure of the Week
From Rich Painter:
|This Week's Cool Product
Micro Digital has a new version of their RTOS which is designed from the outset for high security. SecureSMX gives full isolation between partitions by providing the following:
It's available for Cortex-v7M and Cortex-v8M architectures, which account for most of the Arm MCUs available today. I haven't tried it, but I've long been a fan of their innovative eheap and GoFast libraries. The former is a much improved C heap manager, and the latter a blazing-fast floating point library.
Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.
Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.
|Joke For The Week
These jokes are archived here.
From Michael J. Linden:
I have got some tech problems at home. Alexa and Siri found out about each other, and now neither one of them will talk to me…
|About The Embedded Muse
The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at email@example.com.
The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.