Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 451, August 1, 2022
Copyright 2022 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Embedded Studio cross platform IDE

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Last week's "joke" was about an individual's troubles stemming from license plates that read "NULL." Several readers wrote that Radiolab did a take on this here.

Quotes and Thoughts

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

- Brian W. Kernighan
Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

In the last issue I mentioned the issue of dealing with power transients. Brian Robinson wrote that there's a good take on that on LinkedIn here.

(That URL is 245 characters long. That's long enough to differentiate about 10360 individual objects. Given that there are only 1070 protons in the universe one wonders what these web developers are thinking.)

Freebies and Discounts

The giveaway is on vacation for the summer.

Docs in English

The quote in the issue 449 was "If you can't write it down in English, you can't code it." Not everyone agreed.

There were two general objections. The first is that English is an imprecise language that is easily abused. It's pretty easy to write sentences that are redundant, recursive, vague and open to multiple interpretations. I agree! But that's no excuse to abdicate our responsibility to correctly document our work. An imperfect though useful solution is to relentlessly edit our written word, to sharpen our intent and to strive for clarity above all else. English might not be an ideal solution, but the perfect is often the enemy of the good.

The second objection was that English is far from a universal language. Wikipedia pegs the number of languages used in the world at as many as 7000. Should a Farsi speaker be required to document in English, which may be a very foreign lingo?

I've had to work on code that was commented in Chinese and other languages. It's a bit jarring for a Westerner to come across:

while(1){            // 這是評論

If clarity is our goal, and it should be, then a Chinese speaker will find that comment much more informative than a phrase couched in, say, English, when that may be constructed in a grammatically incorrect fashion due to the author's unfamiliarity with the language.

At a dinner with executives of a company in the Czech Republic some years ago the president told me of a major problem they were facing. They had several million lines of C code which was all commented in Czech, yet (according to him) there were only 10m Czech speakers in the world. The company's growth was stalled as the pool of developers was very limited.

In my travels I've come across organizations whose policies varied widely. Some had no rules; their code from a plethora of nationalities was documented in numerous languages, and the Babel-like issues were dealt with by constraining certain parts of the code to specific countries. In many, many other cases the rule was to use English, period. For better or for worse. Some of the docs were incomprehensible due to the authors' poor grasp of the language.

I don't know what the right answer is, but do believe that documentation is a vital part of what we do, and poor docs are a risk to the viability of a company. Risk must be managed, so companies should have a policy that mitigates that risk.

In my experience the problems are worse at startups. These companies code for today, not for next week. Documentation is the least of their concerns; nothing overrides shipping ASAP. Generally the companies either fail, in which case the lousy docs don't matter, or succeed/get-acquired. In that case you almost always see a reckoning. The code haphazardly slapped together by a brilliant founder who soon dumps the mess on a raft of new hires becomes the basis for the company's products. It's a mess, "works" (mostly), and acts as an anchor on engineering productivity for years to come. Few outfits dare a rewrite. The technical debt isn't reflected (directly) on the balance sheet, but it's there, for those in the know.

The accountants haven't caught up with these issues, even though software has been a critical business issue for 60+ years. For instance, Microsoft's latest 10-K filing has 10,031 words detailing risks that could cause material harm to the company's business, yet there's not a peep about poor documentation, the risk of losing poorly-backed-up code, or the other exigencies of software engineering. Intel's 10-K has almost 14k words outlining their perceived risks, with nary a mention of software engineering. However, the 10-K does show they made $20 billion in profit last year, and the CEO was granted $178 million in compensation. I guess these paltry figures explain the company's desperate need for a share of the $50 billion Intel and the rest of the semiconductor industry is demanding from American taxpayers.

Harold Kraus sent this ("DER" is Designated Engineering Representative, a person who effectively signs off that the precepts of the avionics standard (DO-178C) were followed in creating software for commercial aviation):

"If you can't write it down in English, you can't code it."

My first DER taught:

non scripta, non est.

If it is not written down, it didn't happen.

Somehow the idea of using Latin, a dead language, to make this point is both fun and ironic.

The Shirley Test

Responding to my comment in Muse 450 about Shirley, the young tester who could always break our code, Steve Wheeler wrote:

(Back in the 1970s we had our own form of fuzz testing; we called it Shirley Testing. Shirley was a wonderful young woman with a limited education who had a preternatural knack for breaking our code. In our youthful hubris we built The Most Robust Firmware Ever... but Shirley was sure to find a combination of inputs that embarrassed us and forced the team to write code that questioned every input.)

When I worked on word processors for NBI back in the 1980s, we had one guy in the testing group who had that kind of knack. He'd submit bug reports with instructions for reproducing the problem that included things like, "enter 24 blank lines, then hit tab 27 times, then backspace and page up." How does one even think to try such things, let alone keep track of them well enough to reproduce them? My mind just doesn't work that way. It's good to have someone like that finding the weird edge cases and holes, though.

Daniel McBrearty also responded:

The Shirley Principle struck a chord with me. It doesn't only apply to software. Years ago in my analogue design days I did a system that had a lot of eurocards with linear voltage regulators (remember those). It was a big system and we only had one shot to get it right. It HAD to be on time as well. The regs were overspecified, had generous heatsinks, and every other conceivable protection was in place. I thought.

Anyway a week before it had to be installed and switched on I gave it to our new job student, let's call him Jeff, a young man with long hair and a love of bizarre metal music, to rack it up and see how it works. An hour later he managed to get an army of red LEDs on the front, and I discovered why some previous designs I'd seen had 3 chunky diodes around those regulators. When you have split supplies, and you hot plug a card, you can get the negative rail coming up slightly before the positive (or maybe vice versa) and the opposing regulator gets very unhappy. You now have a system with only one rail, obviously catastrophic. Especially when it drives a room full of BIG power amps which in turn feed about 1800 speakers.

Well, I was VERY grateful to Jeff and my opinion of him instantly went up. We had plenty time to add the necessary diodes (ah the days of through hole) and all was well (and is to this day). He did pretty well too, he's now HOD at that establishment.

I'm very much in favour of getting people who do NOT know the system, or "what is correct" to try stuff. Give them free reign. It's just amazing how often we, as designers, develop tunnel vision that forgets that our set of pre-conceived start conditions is only a subset (a small one) of "the universe".

One reason that people who say "but that can never happen" should make us suspicious. That phrase, right there, can sink your ship.

And this from Lous Mamakos:

Regarding Fuzz testing..  years ago in the pre-historic, pre-commercial days of the Internet, I was writing a TCP/IP stack for the UNIVAC 1100 computers at the University of Maryland as an undergrad project. 

Probably 1980 or 1981; still sort of early days for the Internet protocols as they escaped out of the labs into other academic environments.

They graybeards at the time described a testing technique for those implementing network protocols.  They were "Christmas Tree Packets".   Light up some or all of those random bit fields in the protocol headers to see if your implementation survived.  What does TCP do with a packet that has both a SYN and URG and RST bits in the header -- a nonsense combination -- but it really ought not to crash.  Or insane values in sequence numbers.  Or the IP total length field point past the end of your packet buffer.  All things that "can't" or "shouldn't" happen.

I think that experience really left an impression - stuff coming in off the wire was inherently untrustworthy and hostile and you had to come to it with a defensive, adversarial point of view when writing code.

Happy and Productive

"Since we started letting the developers use the [chaos, Capability Maturity Model, Personal Software Process, Test-Driven Development] process they have been much happier and more productive."

Really? I can't tell you how many managers have told me some version of this story. It's almost always complete nonsense.

Without productivity metrics it's impossible to know if the team is getting better or worse. Few firmware groups measure anything related to performance. Or quality. There are lots of vague statements about "improving productivity/quality," and it's not unusual to hear a dictate from on-high  to build a "best-in-class" development team. But those statements are meaningless without metrics that establish both where the group is now, the desired outcome, a path to get there, and metrics to determine if the team is meeting these goals.

Some teams will make a change, perhaps adopting a new agile approach, and then loudly broadcast their successes. But without numbers I'm skeptical. Is the result real? Measurable? Is it a momentary Hawthorne blip?

Developers will sometimes report a strong feeling that "things are better." That's swell. But it's not engineering. Engineering is the use of science, technology, skilled people, process, a body of knowledge - and measurements - to solve a problem.

Engineers use metrics to gauge a parameter. The meter reads 2.5k ohms when nominal is 2k.  That signal's rise time is twice what we designed for. The ISR runs in 83 usec, yet the interrupt comes every 70. We promised to deliver the product for $23 in quantities of 100k, and so have to engineer two bucks out of the cost of goods.

Some groups get it right. I was accosted at the ESC by a VP who experienced a 40% schedule improvement by the use of code inspections and standards. He had baseline data to compare against. That's real engineering. When a developer or team lead reports a "sense" or a "feeling" that things are better, that's  not an engineering assessment. It's religion.

Metrics are tricky. People are smart and will do what is needed to maximize their return. I remember instituting productivity goals in an electronics manufacturing facility. The workers started cranking out more gear, but quality slumped. Adding a metric for that caused inventory to skyrocket as the techs just tossed broken boards in a pile, rather than repair them.

Metrics are even more difficult in software engineering. The data is always noisy. The metrics are like an impressionistic painting that yields a fuzzy view.

But data with error bands is far superior to no data at all.

Failure of the Week

From Daniel Hoenig:

And this pricelss gem from Todd Hayes:

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Firmware Solution Engineer - Memfault - Boston, MA or Remote (EST time zones)
Memfault is hiring a Firmware Solution Engineer to join our team building the first observability platform for hardware teams. In this role, you will help architect and implement new features for our embedded firmware SDK and engage directly with customers to help them integrate our platform.
Read the full job description and apply here: https://jobs.lever.co/memfault/9fa58d3b-2415-4d80-9e5c-be989f90c2e4

Wind River, a leader in mission critical systems that run mostly in the embedded space, is looking for a Senior Linux Developer. The position is being advertised as near Ottawa, but might potentially be remote. The long-time reader of The Embedded Muse who sent in this link tells me, "I've been at Wind River for about a year and a half, and it's a great place to work where engineers are truly supported and valued."

Senior Firmware Developer with Omnisite in Indianapolis, Indiana
OmniSite is a family-owned manufacturing company and industry leader in remote alarm monitoring providing products to water utilities, wastewater departments, residential customers, and other industries across North America.
Senior Firmware Developers work with our engineering team to execute our complete embedded software development lifecycle to create scalable, maintainable, and optimized systems for highly reliable products.  We greatly value firmware developers who enjoy problem solving with a diverse group of people. If you fit this description, we can offer you work life balance, a flexible work schedule, and a strong, goal-oriented team.

Embedded Software Engineer.  Located in the northern suburbs of Atlanta, Nordson corporation is looking for a new team member that is able to quickly start contributing with a team of seasoned software engineers.  You will be writing code for sophisticated embedded systems that are primarily used to control adhesive deposition in automated production lines.  Experience with C/C++ and employing embedded libraries/RTOS required.  Experience with ARM microcontrollers and Segger stacks a plus.  Apply at: Careers (myworkdayjobs.com)

Senior Hardware Engineer.  Located in the northern suburbs of Atlanta, Nordson corporation is looking for a new team member that can walk in the door and start designing embedded PCA's that involve the following types of circuits: digital, analog, microcontroller, power switching, Ethernet, USB, CPLD/FGPA, etc.  You won't be stuck doing the same type design work over and over since Nordson equipment utilizes many different types of electronics.  Apply at: Careers (myworkdayjobs.com)

AppliedLogix, an electronics and embedded design services firm in Rochester, NY, is looking for engineers to join our growing team, working on a broad range of challenging, embedded systems projects.
- Our current need is for an Electrical Engineer with an analog/mixed-signal focus, comfortable in Altium, with 5+ years of experience. 
- We are a flat, remote-first, flexible, "for engineers by engineers" organization founded on passing maximum value through to the team that creates it.
- For more information, visit https://appliedlogix.com/open-positions/

Joke For The Week

These jokes are archived here.

This is from Will Cooke:

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.