Jack Ganssle, Editor of The Embedded Muse Jack Ganssle's Blog
 RSS Feed This is Jack's outlet for thoughts about designing and programming embedded systems. It's a complement to my bi-weekly newsletter The Embedded Muse. Contact me at jack@ganssle.com. I'm an old-timer engineer who still finds the field endlessly fascinating (bio).

2019's Most Important Lesson

November 1, 2019

My very first engineering class in college had some silly name like "Engineering Fundamentals." I don't remember much about it, other than that we were shown a video of the Takoma Narrows bridge failing in 1940, four months after it opened. At 18 years old I thought that was a very cool film. Decades later I'm struck that this disaster was used to educate hundreds of thousands of engineers about the perils of that particular failure mode.

We can learn from the disasters of other engineers.

We must learn from the disasters of other engineers.

I have a huge file of disasters in the embedded field and share some of those in my seminars, as only a fool would want to repeat mistakes made in the past. Your errors can teach others things to avoid.

2019 provided us with a truly remarkable example of this. The 737 Max crashes teach many lessons. To me, the most important is the peril of believing sensor data. The angle-of-attack sensor started giving truly bizarre readings, yet the computer accepted those as legit.

Sensors fail. That should be a meme burned brightly into our psyches.

A mistake we make is to believe sensor data. Or to smooth it, perhaps averaging some number of samples. An average of garbage is still garbage.

Consider this excerpt from the flight data recorder on the Ethiopian Airlines crash:

737 Max flight data recorder

The AOA sensor went from zero to 75 degrees in under one second. That in itself is unbelievable, and the software should have rejected it. But the altitude didn't change; if the aircraft were pointed up, it would probably go up. The airspeed remained mostly constant. At 75 degrees up angle airspeed would have bled off faster than a vampire relishing his first kill of the night.

The flight data recorder had 1790 channels of data fed to it. One gave an absurd value; many of the others contradicted that. Had the software evaluated the AOA sensor against the other channels it could have behaved more reasonably. Even absent the other data, comparing the 75 degree rotation in less than a second to a physical model of what a commercial aircraft can do would have sounded alarms.

The moral for all of us embedded people: Always compare sensor data to the physics and to other inputs. If it seems odd, respond appropriately. Here's a handful of other cases where crummy code gave stupid results. I have hundreds of other examples:

Two takeaways:

  • Check your sensor data for reasonableness
  • Check your outputs for sanity

Back to Jack's blog index page.

If you'd like to post a comment without logging in, click in the "Name" box under "Or sign up with Disqus" and click on "I'd rather post as a guest."

Recent blog postings: