Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 0400, July 15, 2019
Copyright 2019 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

Express Logic

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

Matt Bennett pointed out that the last Muse, number 377, means this one is issue 0400. When was the last time you used octal? In the early days of the 8008 all of Intel's tools were octal, which was quite weird for an 8-bit machine. My collage had a Univac 1108, a 36-bit mainframe - those tools also used octal. That's also strange, as hex would have meant much shorter integers. My first exposure to hex was on an IBM 360 and I remember how very odd it seemed at the time. When someone close to me turned 64 last year I reassured her that she was actually only 40. 0x40.

Jack's latest blog: Fun definitions of embedded terms.

Latest video: A review of Siglent's SDS5000X

Quotes and Thoughts

When you can measure what your are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind. Lord Kelvin

Tools and Tips

SEGGER Embedded Studio The leading cross platform IDE

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Martin Ettl suggested:

I was reading your information about static code analysis tools and noticed that Cppcheck is missing.

This open source tool is heavily utilized in our company to reduce bugs. Cppcheck can be found here: http://cppcheck.sourceforge.net/

A number of readers have sent in comments (responding to Muse 376 and 377) about how they use the assert() macro. Many report errors back to the mothership over the Internet. Percepio has codified this into an interesting product (full disclosure: I have participated in several advisory phone calls with them about this).

Freebies and Discounts

Mark Walter won the Joulescope current-measuring tool in last month's contest.

The folks at LitePlacer are offering a pick and place machine, in kit form, to Muse readers this month. This looks like a very cool robot:

This month's contest winner gets a LitePlacer pick and place machine

Enter via this link.

On Getting it Right

George Farmer wrote about some new issues in th 737 MAX, and then had this to say:

On a possible related note (I know you have devoted a lot of time to this topic), for our safety-critical systems we designed in Aerospace, we adopted the NASA mantra "If it's on the same die, it won't fly."  Among other things, this meant we couldn't rely on or even use the hardware watchdog timer built into the die of the microprocessor, primarily because they were WAY too easy to defeat.  Indeed, it had to be external and a totally independent entity (also Rad hardened, etc.).  Indeed, I use to demonstrate just how easy it was to defeat an internal WDT by exposing an 8751 (EPROM) micro to the overhead fluorescent lighting.   With the WDT enabled, the overhead lights would cause the micro to freeze-up within seconds of removing the opaque erase window cover.

We went one step beyond by implementing signature-based watchdogs (versus event-based).  These signature-based watchdogs not only required a unique pattern to be written to the chip, the pattern was actually a Ticket Check of critical tasks that had to occur in proper sequence, as well as each within a specified window of time.  Each critical task had its own unique bit mask that was submitted to the external WDT at the end of its run.  Failure of one or more critical tasks to execute - either in sequence or on time - blew the total signature out of the water, causing the external WDT to trip.  This would catch priority inversion, most SEU scenarios, etc.

Some people - both hardware and software designers as well as cost hawks - were quick to lambaste these topologies, stating they were over-engineered, too costly, too much a pain in the ass during development, or absolutely unnecessary.  This was especially true after I left aerospace and went into  the off-road heavy construction sector.   I wish I had a dollar for every time I heard someone viscerally say "It's time to shoot the engineer and get on with business...," or "This isn't Rocket Science...," etc.

The naysayers who blast conservative designs may have gotten one thing right:  It truly isn't Rocket Science.   However,  in many cases it certainly is Control Systems 101.  Those who exalt on the so-called Virtues of Risk and declare engineers as being too risk adverse typically don't have a clue on just exactly what risk is, or even how to calculate and assess said risks.  More often than not, people like this are promoted, leaving behind a ton of carnage in their wake (The Peter Principle).

I go one step farther by saying that most, if not all engineers, if they cannot design-out the risks all together due to, say, limitations of the technology, they actually do take risks.  This is why we use tools like Pugh matrices, KT, FMEAs, etc.  However, there is a HUGE difference between taking a calculated risk - one that has been truly mitigated to as low as reasonably achievable - versus taking flippant risks based on greed, arrogance, self-preservation, or just plain stupidity.

When I was an undergrad working on my engineering degree, we were required to take an engineering ethics course.  At the end of the semester we signed a pledge - basically the equivalent of a Hippocratic Oath - stating that we would adhere to the laws of physics and principles of sound engineering practices, and never knowingly do anything that would put people in Harm's Way.  After 34+ years I still take that oath VERY seriously.

Faster... Better... Cheaper... Pick any two.

When I went to engineering school ethics were never mentioned, but that was in the early 70s. IEEE's code of ethics includes this:

[We agree] 1. to hold paramount the safety, health, and welfare of the public, to strive to comply with ethical design and sustainable development practices, and to disclose promptly factors that might endanger the public or the environment;

That's a pretty good place to start.

Code Inspections

What's the cheapest way to get rid of bugs?

Don't put them in in the first place.

The statistics are dramatic. Most code starts life with around five to ten bugs per hundred lines of code. Without inspections figure on finding about 95% of those pre-shipping. For a 100KLOC program that means shipping with hundreds of defects! (Note that some sources, such as Stan Rifkin, "The Business Case for Software Process Improvement", Fifth SEPG National Meeting, 26-29 April 1993, claim far worse numbers, on the order of 6 shipped bugs per hundred lines of code - six thousand in a 100KLOC project. My data,shows that firmware generally does not sink to that abysmal level.)

It's important to understand that testing, as implemented by most companies, just does not work. Many studies confirm that tests exercise only about half of the code. Exception handlers are notoriously problematic. Also consider nested conditionals: IFs nested five or six levels deep can yield hundreds of possible conditions. How many test regimes check all of those possibilities?

Cyclomatic complexity, among other things, tells us the minimum number of tests needed to completely exercise a function. I found that one function in the Apache web server scored a complexity index of 725. Who constructs nearly a thousand tests for a single function?

Another analysis of the testing problem is scarier. In "The Software Quality Challenge" (Crosstalk, June 2008) Watts Humphrey shows that a program with 100 decisions can have nearly 200,000 possible paths. With 400 decisions that explodes to 1011 paths, each requiring a test! Though he describes a pathological worst-case situation, the implications are important.

In one example on real-world code, I ran a 10,000 line C program through LDRA's Unit tool, which auto-generates unit test code. It produced 50,000 lines of tests. How many of us write 5 lines of test code for each line of firmware?

Some approaches to test are much more effective. Code coverage, used in the avionics and some other industries, can approach 100% testing, albeit with huge costs. Test Driven Development advocates also claim good results, though reliable supporting studies are hard to find. However, anecdotal evidence does suggest that most of the agile approaches, which have a disciplined focus on testing, improve on the 50% code test ratio. The agile approaches demand, correctly, automated tests that cost little to run. In eXtreme Programming, for instance, the rule is that the tests must exercise 100% of the system in 10 seconds or less. A nice goal, for sure, but impractical in most embedded systems. After all, someone has to watch the display and press the buttons.

The bottom line is that test is a huge problem. However, a well-run inspection program will eliminate 70 to 80% of the bugs before any testing starts.

Plenty of evidence suggest well-run inspections are some 20 times cheaper than debugging. That's a staggering number. Since the average project burns about half the schedule in the debugging phase, any technique with a 20X improvement nearly halves the schedule. I'm generally skeptical of any dramatic claim, from the Ginzu knives to politicians' promises to clean up Washington, but even if you cut that number by an order of magnitude inspections still shave a third from the schedule.

Higher quality code, for less money. What a deal!

Correctly run inspections will find few errors. That, of course, is exactly our goal. If I write software no one will see it's one kind of code. If, however, I know my colleagues will examine it in depth, the code will be very different. No one wants his errors pointed out. The inspections are an audit process that immediately raises the program's quality.

And that's what we want: correctness from the start, rather than trying to desperately pound quality in late in the project.

People complain they don't have time for inspections. Capers Jones studied 4000 software projects and found the number one cause of schedule problems was quality. (Jones, Capers. Assessment and Control of Software Risks. Englewood Cliffs, N.J.: Yourdon Press, 1994.) Bugs.

Inspections don't slow the project. Bugs do.

Apollo Resources

Saturday July 20 marks 50 years since Apollo 11's lunar module touched down on the moon. It's hard to believe that a half century has gone by. I was 16 at the time, working as an electronics technician during the summers on Apollo ground support equipment, and remember how excited all of us were about that mission. And the succeeding ones! Yet I also remember a later Apollo launch where the TV screen was 3/4 a football game, and the remaining quarter the launch. The nation quickly tired of these missions. We techies never did, and I am still an Apollo junkie with shelves full of books about the program.

Here are some of my favorites, in no particular order.

Digital Apollo (my review) - a must-read. David Mindell describes the evolution of the man-machine interface. The Apollo Guidance Computer (AGC) is credited with all sorts of successes, as it should be, but the engineers really designed an astronaut-computer-systems spacecraft where all of those components were integrated into a gestalt navigational system. Did you know the very first contract released for this program was to MIT's Instrumentation Lab for the AGC? Oh, it needed a bit of code, too, and the entire requirements document for that bit of work read: "The onboard guidance computer must be programmed to control the other guidance subsystems and to implement the various guidance schemes."

SUNBURST and LUMINARY (my review) - a book only a techie could love. Don Eyles was a programmer working on the lunar module's software. This is a mixed book with too much about his love life, but the only one I know of from a software insider. It's worth the fluff to gain some deep insight into the code.

Carrying the Fire - A lot of astronauts wrote memoirs, but Mike Collins' is by far the most literate. It appears there's a new edition available. Collins led an amazing life, most of which is not documented in this book, as it ends after Apollo 11. One thought that still lingers decades after I read this: he developed a spine condition that grounded him before Apollo. He complains that he had to fly commercial for some time, rather than in his NASA-supplied T-38. It's a tough life.

Apollo 11 (the movie) (my review) - OK, this is not a book, but is a recent (2019) movie about, well, you know what. There's no narration. There's essentially no new footage. It's all archival film and the sound of mission communications. It's stunning. I have no idea how to go about seeing it; we caught it in the theater. This is the best Apollo movie I've seen. It's very techie, tons of acronyms, and the jargon is everywhere as those involved were running a mission, not expecting their work to be movie-fied 50 years later. I feared Marybeth would be bored as she isn't technical, but she was entranced.

First Man - James Hansen wrote the only authorized biography of Neil Armstrong, and it's a page-turner. Don't watch the movie of the same name, which depicts this man as an emotionless robot. He was a deep individual. You've got to love a guy who said "I am, and ever will be, a white-socks, pocket-protector, nerdy engineer."

Stages to Saturn - Want to know how the Saturn rocket worked? Roger Bilstein's book will give you deep technical insight. So deep I only recommend this book for the ultimate Apollo junkie. I was fascinated.

Apollo: The Race to the Moon - Darn it, this book is out of print and even on Abebooks it's over a hundred bucks. Charles Murry and Catherine Cox give the complete back story to these missions. Did you know von Braun wanted to land the entire rocket (Saturn or Nova) on the moon and then bring it back? This book gives a good overview of those discussions and much else. It might be the best tome about the Missions Operation Control Room (which is being refurbished and should open to the public this year). There's a picture of a Mercury spacecraft being transported at the Cape on the back of a flatbed truck - on a mattress. From mattress to crawler-transporter in 4 years (the transporter became available in 1965). Astonishing.

Journey to the Moon: The History of the Apollo Guidance Computer - Eldon Hall gives the best description of the design and manufacturing of the AGC. Tons of diagrams and color photos which will answer most questions about this amazing bit of engineering.

Saturn V Owners' Workshop Manual - Remember the old Haynes workshop manuals for cars? They described everything about maintenance on a particular model of a vehicle, from brake work to rebuilding engines. This book is in the same vein. W. David Woods doesn't give repair information, but dives deep - very deep - into the design of the Saturn V. It's for major Apollo geeks only, but is a ton of fun. Every page is illustrated with pictures and diagrams. Need details of turbopump operation? Arrangement of the fuel lines? This is the manual for you.

And, in quirky but fun news, the Smithsonian will project an image of a Saturn V on the Washington Monument this week.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.


Joke For The Week

Note: These jokes are archived here.

A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and Tequila. -- Mitch Ratcliffe

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.