You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• On Getting it Right
• Thoughts on Code Inspections
• Apollo Resources
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

Matt Bennett pointed out that the last Muse, number 377, means this one is issue 0400. When was the last time you used octal? In the early days of the 8008 all of Intel's tools were octal, which was quite weird for an 8-bit machine. My collage had a Univac 1108, a 36-bit mainframe - those tools also used octal. That's also strange, as hex would have meant much shorter integers. My first exposure to hex was on an IBM 360 and I remember how very odd it seemed at the time. When someone close to me turned 64 last year I reassured her that she was actually only 40. 0x40.

Jack's latest blog: Fun definitions of embedded terms.

Latest video: A review of Siglent's SDS5000X

When you can measure what your are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind. Lord Kelvin

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Martin Ettl suggested:

A number of readers have sent in comments (responding to Muse 376 and 377) about how they use the assert() macro. Many report errors back to the mothership over the Internet. Percepio has codified this into an interesting product (full disclosure: I have participated in several advisory phone calls with them about this).

Mark Walter won the Joulescope current-measuring tool in last month's contest.

The folks at LitePlacer are offering a pick and place machine, in kit form, to Muse readers this month. This looks like a very cool robot:

Enter via this link.

George Farmer wrote about some new issues in th 737 MAX, and then had this to say:

When I went to engineering school ethics were never mentioned, but that was in the early 70s. IEEE's code of ethics includes this:

[We agree] 1. to hold paramount the safety, health, and welfare of the public, to strive to comply with ethical design and sustainable development practices, and to disclose promptly factors that might endanger the public or the environment;

That's a pretty good place to start.

What's the cheapest way to get rid of bugs?

Don't put them in in the first place.

The statistics are dramatic. Most code starts life with around five to ten bugs per hundred lines of code. Without inspections figure on finding about 95% of those pre-shipping. For a 100KLOC program that means shipping with hundreds of defects! (Note that some sources, such as Stan Rifkin, "The Business Case for Software Process Improvement", Fifth SEPG National Meeting, 26-29 April 1993, claim far worse numbers, on the order of 6 shipped bugs per hundred lines of code - six thousand in a 100KLOC project. My data,shows that firmware generally does not sink to that abysmal level.)

It's important to understand that testing, as implemented by most companies, just does not work. Many studies confirm that tests exercise only about half of the code. Exception handlers are notoriously problematic. Also consider nested conditionals: IFs nested five or six levels deep can yield hundreds of possible conditions. How many test regimes check all of those possibilities?

Cyclomatic complexity, among other things, tells us the minimum number of tests needed to completely exercise a function. I found that one function in the Apache web server scored a complexity index of 725. Who constructs nearly a thousand tests for a single function?

Another analysis of the testing problem is scarier. In "The Software Quality Challenge" (Crosstalk, June 2008) Watts Humphrey shows that a program with 100 decisions can have nearly 200,000 possible paths. With 400 decisions that explodes to 10¹¹ paths, each requiring a test! Though he describes a pathological worst-case situation, the implications are important.

In one example on real-world code, I ran a 10,000 line C program through LDRA's Unit tool, which auto-generates unit test code. It produced 50,000 lines of tests. How many of us write 5 lines of test code for each line of firmware?

Some approaches to test are much more effective. Code coverage, used in the avionics and some other industries, can approach 100% testing, albeit with huge costs. Test Driven Development advocates also claim good results, though reliable supporting studies are hard to find. However, anecdotal evidence does suggest that most of the agile approaches, which have a disciplined focus on testing, improve on the 50% code test ratio. The agile approaches demand, correctly, automated tests that cost little to run. In eXtreme Programming, for instance, the rule is that the tests must exercise 100% of the system in 10 seconds or less. A nice goal, for sure, but impractical in most embedded systems. After all, someone has to watch the display and press the buttons.

The bottom line is that test is a huge problem. However, a well-run inspection program will eliminate 70 to 80% of the bugs before any testing starts.

Plenty of evidence suggest well-run inspections are some 20 times cheaper than debugging. That's a staggering number. Since the average project burns about half the schedule in the debugging phase, any technique with a 20X improvement nearly halves the schedule. I'm generally skeptical of any dramatic claim, from the Ginzu knives to politicians' promises to clean up Washington, but even if you cut that number by an order of magnitude inspections still shave a third from the schedule.

Higher quality code, for less money. What a deal!

Correctly run inspections will find few errors. That, of course, is exactly our goal. If I write software no one will see it's one kind of code. If, however, I know my colleagues will examine it in depth, the code will be very different. No one wants his errors pointed out. The inspections are an audit process that immediately raises the program's quality.

And that's what we want: correctness from the start, rather than trying to desperately pound quality in late in the project.

People complain they don't have time for inspections. Capers Jones studied 4000 software projects and found the number one cause of schedule problems was quality. (Jones, Capers. Assessment and Control of Software Risks. Englewood Cliffs, N.J.: Yourdon Press, 1994.) Bugs.

Inspections don't slow the project. Bugs do.

Saturday July 20 marks 50 years since Apollo 11's lunar module touched down on the moon. It's hard to believe that a half century has gone by. I was 16 at the time, working as an electronics technician during the summers on Apollo ground support equipment, and remember how excited all of us were about that mission. And the succeeding ones! Yet I also remember a later Apollo launch where the TV screen was 3/4 a football game, and the remaining quarter the launch. The nation quickly tired of these missions. We techies never did, and I am still an Apollo junkie with shelves full of books about the program.

Here are some of my favorites, in no particular order.

Digital Apollo (my review) - a must-read. David Mindell describes the evolution of the man-machine interface. The Apollo Guidance Computer (AGC) is credited with all sorts of successes, as it should be, but the engineers really designed an astronaut-computer-systems spacecraft where all of those components were integrated into a gestalt navigational system. Did you know the very first contract released for this program was to MIT's Instrumentation Lab for the AGC? Oh, it needed a bit of code, too, and the entire requirements document for that bit of work read: "The onboard guidance computer must be programmed to control the other guidance subsystems and to implement the various guidance schemes."

SUNBURST and LUMINARY (my review) - a book only a techie could love. Don Eyles was a programmer working on the lunar module's software. This is a mixed book with too much about his love life, but the only one I know of from a software insider. It's worth the fluff to gain some deep insight into the code.

Carrying the Fire - A lot of astronauts wrote memoirs, but Mike Collins' is by far the most literate. It appears there's a new edition available. Collins led an amazing life, most of which is not documented in this book, as it ends after Apollo 11. One thought that still lingers decades after I read this: he developed a spine condition that grounded him before Apollo. He complains that he had to fly commercial for some time, rather than in his NASA-supplied T-38. It's a tough life.

Apollo 11 (the movie) (my review) - OK, this is not a book, but is a recent (2019) movie about, well, you know what. There's no narration. There's essentially no new footage. It's all archival film and the sound of mission communications. It's stunning. I have no idea how to go about seeing it; we caught it in the theater. This is the best Apollo movie I've seen. It's very techie, tons of acronyms, and the jargon is everywhere as those involved were running a mission, not expecting their work to be movie-fied 50 years later. I feared Marybeth would be bored as she isn't technical, but she was entranced.

First Man - James Hansen wrote the only authorized biography of Neil Armstrong, and it's a page-turner. Don't watch the movie of the same name, which depicts this man as an emotionless robot. He was a deep individual. You've got to love a guy who said "I am, and ever will be, a white-socks, pocket-protector, nerdy engineer."

Stages to Saturn - Want to know how the Saturn rocket worked? Roger Bilstein's book will give you deep technical insight. So deep I only recommend this book for the ultimate Apollo junkie. I was fascinated.

Apollo: The Race to the Moon - Darn it, this book is out of print and even on Abebooks it's over a hundred bucks. Charles Murry and Catherine Cox give the complete back story to these missions. Did you know von Braun wanted to land the entire rocket (Saturn or Nova) on the moon and then bring it back? This book gives a good overview of those discussions and much else. It might be the best tome about the Missions Operation Control Room (which is being refurbished and should open to the public this year). There's a picture of a Mercury spacecraft being transported at the Cape on the back of a flatbed truck - on a mattress. From mattress to crawler-transporter in 4 years (the transporter became available in 1965). Astonishing.

Journey to the Moon: The History of the Apollo Guidance Computer - Eldon Hall gives the best description of the design and manufacturing of the AGC. Tons of diagrams and color photos which will answer most questions about this amazing bit of engineering.

Saturn V Owners' Workshop Manual - Remember the old Haynes workshop manuals for cars? They described everything about maintenance on a particular model of a vehicle, from brake work to rebuilding engines. This book is in the same vein. W. David Woods doesn't give repair information, but dives deep - very deep - into the design of the Saturn V. It's for major Apollo geeks only, but is a ton of fun. Every page is illustrated with pictures and diagrams. Need details of turbopump operation? Arrangement of the fuel lines? This is the manual for you.

And, in quirky but fun news, the Smithsonian will project an image of a Saturn V on the Washington Monument this week.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Note: These jokes are archived here.

A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and Tequila. -- Mitch Ratcliffe

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.