|Jack Ganssle's Blog
This is Jack's outlet for thoughts about designing and programming embedded systems. It's a complement to my bi-weekly newsletter The Embedded Muse. Contact me at firstname.lastname@example.org. I'm an old-timer engineer who still finds the field endlessly fascinating (bio).
|For novel ideas about building embedded systems (both hardware and firmware), join the 35,000 engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype and no vendor PR. Click here to subscribe.
Lessons From a Failure
April 10, 2020
George Farmer reports on yet another embedded failure, and draws some lessons that should have been learned:
A few weeks back I was first in line for a touchless car wash at one of our local gas stations, waiting on a person who was already in the bay when I pulled up. Both the entrance and exit doors were closed, and it appeared to me the wash cycle was wrapping-up just as I arrived. I saw the green light turn on, signaling the driver to proceed forward through the dryers. This is where the story begins.
While I was waiting I dutifully slipped my prepaid carwash card into the reader and selected "Touchless." It is here that a red flag should have registered in my old, tired grey matter, but at 5:45 am and no caffeine in my system, I was an automaton. Had I been alert, at that moment no one was behind me at the time, and I could have backed up to a nearby exit point and simply left. But no! This is Wisconsin after all, and warmer, dryer weather in February is a rare thing, so I was determined to get several weeks and countless layers of crusty salinity off my wife's car. Besides, I don't like our cars being salt licks for the local deer population.
For starters, the kiosk didn't spit out a receipt like it normally does. "No Biggie," I thought, "This has happened before." Also, the "Soft Touch" and "Touchless" selection buttons were reversed; or, at least that's what the voice prompt indicated compared to their electronic display labels. I canceled my selection and started again. This time the selections and voice prompt lined up normally, so I chalked the previous faux pas up to mindless operator error and temporary brain death due to the lack of caffeine (there is no life before coffee).
OK, time to enter the car wash.
I'm waiting... and waiting... and... wait!
The person inside the bay isn't moving off the wheel chocks. Hmmm... Is this person stuck? Engine died? Not paying attention? Needs medical attention?
Normally the dryers sound like turbofan jet engines and are turned on just as the light turns green, but I didn't hear them. Finally it dawned on me the person was held captive in position because the robotic spray head boom suspended between two gantry stanchions hadn't retracted and was blocking the car from pulling forward. Not only that, but the left robotic gantry stanchion was positioned within about one foot of the driver's door. Both stanchions also have high-pressure spray heads, so I suspected the person understandably didn't want to roll down their window or try to get out of the car - a smart move indeed. I could not see the person through the fogged bay door and back car window, so I punched the Help button on the Kiosk several times to alert someone inside. I then beeped the horn quickly a couple of times and flashed my brights, just to see if this got the attention of the driver.
Nothing. Nada. Nor was there a response from pressing Help call button (it's an intercom).
At this point I wanted to back up a bit so at least I could get out of my car and go inside for help. Unfortunately a young driver had pulled up very close behind me, blasting music, and was staring down at her phone (I could see the eerie glow of the screen reflecting off her face). No amount of yelling and waving from my car window did any good, so now I'm stuck and can't seem to get anyone's attention.
Fortunately a Good Samaritan at one of the pumps could see the situation unfolding and went inside for help. A few minutes later the manager of the store came out to investigate.
First thing the manager tried to do was raise the entrance door via a keypad code. No luck. She went back inside for some keys and came back to enter a typical manual door at the side of the building. Successful entry this time (gotta love reliable, Old School technology).
After verifying the driver was OK, she went over to a large utility box on the wall - inside the wash bay - opened it, and must have flipped a switch or two. After about twenty seconds or so both doors opened, and the gantry moved to a Home position, releasing the driver from the clutches of The Robotic Beast. The driver departed with a sopping-wet car.
By this time several more cars had pulled up behind the young driver who was blocking me. I seriously doubt the young girl had even looked up from her phone to see what was happening. The manager came over to my car and asked if I had already paid - I answered in the affirmative. She told me the wash had been completely reset and should be fine now. Since the only way out of the jam was to move forward, I reluctantly decided to press my luck.
The initial underbody wash as I slowly drove in seemed to work fine, so I figured I was in the clear. The green light switched to red as my front wheels landed in the wheel chocks. "So far, so good," I thought.
BAD ASSUMPTION. And so the fun began.
Immediately the gantry and spray bar between the two stanchions began an unusual series of gyrations typically only seen in an improv dance studio: total chaos. What alarmed me most was the rotating spray bar that crosses over the car as the gantry stanchions move fore/aft: "Is the bar going to impact my car," I thought? Fortunately it didn't.
The amusing part of this spectacle was the spray bar rotated in the wrong direction most of the time, spraying soap and water all over the place, except onto my car. The gantry bounced back and forth as if totally confused, eventually stopping about mid-span of my car, fortunately with the spray bar well above the car's roof. The high-pressure jets on the stanchions turned on only briefly - certainly not long enough to clean the entire length of the car. I thank my lucky stars I didn't purchase the so-called Elite wash, as this also included a robotic hub cap cleaning using rotating brushes. I have no idea where these brushes would have landed on my car's side panels.
After only a short time on the chocks the light turned green, the dryers spun-up and, unlike the unfortunate driver before me, both doors opened. I zoomed off the chocks as quickly as possible, but upon arrival at the dryers, they shut off. I, too, exited with a mostly wet-but-not-so-clean car, happy to be clear from the clutches of a psychotic robot.
By the way, the young driver behind me must have finally pried her eyes away from her phone, as she pulled into the bay just as I was exiting.
After reporting the experience to the store manager and warning there may be another unsuspecting customer trapped in the bay, I headed home with a free carwash receipt in-hand, but with a 2-tone car (colors of dark blue and salt, in a random, tie-dye color pattern).
On my way home I found my mind slowly coming alive, and I began talking Shop to myself:
Who did the Failure Modes and Effects Analysis (FMEA) on the design of that robotic beast? Was there even an FMEA done on the design?
What are the fail-safes (if any)? Watchdogs? Sanity checks on the sensors? Level of redundancy?
Why was the utility box - which appeared large enough to contain the controller - located inside the bay!? It may be a sealed box, but certainly not hermetically sealed! Were the PC boards conformal-coated? Were Sealtights used?
Who inspected this installation?
Was anyone automatically alerted from inside the store? There should have at least been some type of Deadman alarm inside the store! Hey! How about a tri-color light outside the wash bay: Green = Ready, Yellow = in-use, Flashing Red = Alarm ("Danger, Will Robinson!"), Totally Off = Potentially Serious Trouble.
Who designed the drive-up? It's CAPTIVE! There should be a way to leave the kiosk by pulling forward, not backward!
Ok... The Wifey says I need to get out of the house more often. Keep your eyes on the road!
It's too easy to Armchair-Quarterback someone else's design. It's also virtually impossible for a single engineer - regardless of their formal training and cross-functional experience - to think of every possible failure scenario.
I also realize from first-hand experience that, more often than not, both engineers and their management are under extreme pressure to deliver the goods faster, better and, as always, cheaper. After all, someone high-up in the Food Chain must meet their quarterly figures in order to receive their annual BAB (Big-Ass Bonus), thereby keeping those buissjet companies in business.
Ok… That last one was a cynical cheap shot - my apologies. However, this leads me to a point: Greed and arrogance usually gets people into the trouble, and ultimately the company in the long run. As for greed, well, there are countless books out there on this subject, dealing with ethics, failed B-school business paradigms, etc., so I won't go down this path. As for arrogance, however, maybe not so much.
When I think of arrogance, the concept of Design Ownership comes to mind. A good engineer takes Design Ownership seriously. However, a very wise individual once said, "There's a fine line between pride and arrogance; the trick is to keep the pride and lose the arrogance." It's too easy to mistake arrogance in Design Ownership with pride.
It's often said that Pride is one of the seven deadly sins (along with Greed, Sloth, etc.), although I can't help but wonder if the original authors really meant Arrogance in place of Pride. I can truly understand why an engineer may feel a sense of pride in their designs - especially if they had solved some particular challenge along the way. However, that sense of pride should always be tempered with a healthy dose of reality in that all designs can and almost certainly will fail, given enough time or under the right circumstances. It's not a matter of if, but when. Without this sense of reality it's too easy for Pride to slip into Arrogance. There are WAY too many historical references where arrogance clouded even the finest engineering organization (The White Star Line for the Titanic, or NASA's Challenger and Columbia disasters come to mind, for example – there are thousands more).
Engineers and their management must realize that FMEAs play a vital and critical role in all aspects of design, as well as after design release, production begins, and product hits the streets. FMEAs are living entities that must be maintained throughout the product lifecycle, not simply put on the shelf as someone's checkbox schedule item. Additionally, FMEAs must be done in a cross-functional environment, with all major stakeholders present in one room. The engineers and their management need to check their weapons and arrogance (often mistaken for Pride) at the door! It's not an overstatement to say an FMEA - if done correctly - is a humbling experience.
Lastly, the concept of Design Ownership tends to get lost within a company's chain of command. Indeed, it's too often the case where culpability simply isn't even in the vernacular or business acumen above a certain level - although "Risk Taking" is often found in the acumen. Risk has become a virtue. Trouble is, many people simply don't understand or appreciate what Risk truly is, let alone how to calculate it. Indeed, engineers are often chastised for being too risk adverse. I say this is malarkey.
Engineers take risks on a daily basis, since there is no such thing as Design Perfection. I know I'm preaching when I say, for many reasons, there are ALWAYS design tradeoffs that must be managed, mitigated, and a design path chosen. The best we engineers can hope to achieve is an optimal design: one that is safe, meets all customer requirements at the lowest possible cost, is reliable, on-time, etc. However, there is a HUGE difference between taking a calculated risk - one that has been mitigated to as low as reasonably achievable - and taking a flippant risk based on arrogance or greed (or both). Regarding arrogance and greed, there are too many cases where people and corporations aren't held accountable. This needs to change.
For what it's worth, I have not been back since to use my free carwash :-O
Feel free to email me with comments.
Back to Jack's blog index page.
If you'd like to post a comment without logging in, click in the "Name" box under "Or sign up with Disqus" and click on "I'd rather post as a guest."
Recent blog postings:
- On Checklists - Relying on memory is a fool's errand. Effective people use checklists.
- Why Does Software Cost So Much? - An exploration of this nagging question.
- Is the Future All Linux and Raspberry Pi? - Will we stop slinging bits and diddling registers?
- Will Coronavirus Spell the End of Open Offices - How can we continue to work in these sorts of conditions?
- Problems in Ramping Up Ventilator Production - It's not as easy as some think.
- Lessons from a Failure - what we can learn when a car wash goes wrong.
- Life in the Time of Coronavirus - how are you faring?
- Superintelligence - A review of Nick Bostrom's book on AI.
- A Lack of Forethought - Y2K redux
- How Projects Get Out of Control - Think requirements churn is only for software?
- 2019's Most Important Lesson. The 737 Max disasters should teach us one lesson.
- On Retiring - It's not quite that time, but slowing down makes sense. For me.
- On Discipline - The one thing I think many teams need...
- Data Seems to Have No Value - At least, that's the way people treat it.
- Apollo 11 and Navigation - In 1969 the astronauts used a sextant. Some of us still do.
- Definitions Part 2 - More fun definitions of embedded systems terms.
- Definitions - A list of (funny) definitions of embedded systems terms.
- On Meta-Politics - Where has thoughtful discourse gone?
- Millennials and Tools - It seems that many millennials are unable to fix anything.
- Crappy Tech Journalism - The trade press is suffering from so much cost-cutting that it does a poor job of educating engineers.
- Tech and Us - I worry that our technology is more than our human nature can manage.
- On Cataracts - Cataract surgery isn't as awful as it sounds.
- Can AI Replace Firmware - A thought: instead of writing code, is the future training AIs?
- Customer non-Support - How to tick off your customers in one easy lesson.
- Learn to Code in 3 Weeks! - Firmware is not simply about coding.
- We Shoot For The Moon - a new and interesting book about the Apollo moon program.
- On Expert Witness Work - Expert work is fascinating but can be quite the hassle.
- Married To The Team - Working in a team is a lot like marriage.
- Will We Ever Get Quantum Computers - Despite the hype, some feel quantum computing may never be practical.
- Apollo 11, The Movie - A review of a great new movie.
- Goto Considered Necessary - Edsger Dijkstra recants on his seminal paper
- GPS Will Fail - In April GPS will have its own Y2K problem. Unbelievable.
- LIDAR in Cars - Really? - Maybe there are better ideas.
- Why Did You Become an Engineer? - This is the best career ever.
- Software Process Improvement for Firmware - What goes on in an SPI audit?
- 50 Years of Ham Radio - 2019 marks 50 years of ham radio for me.
- Medical Device Lawsuits - They're on the rise, and firmware is part of the problem.
- A retrospective on 2018 - My marketing data for 2018, including web traffic and TEM information.
- Remembering Circuit Theory - Electronics is fun, and reviewing a textbook is pretty interesting.
- R vs D - Too many of us conflate research and development
- Engineer or Scientist? - Which are you? John Q. Public has a hard time telling the difference.
- A New, Low-Tech, Use for Computers - I never would have imagined this use for computers.
- NASA's Lost Software Engineering Lessons - Lessons learned, lessons lost.
- The Cost of Firmware - A Scary Story! - A hallowean story to terrify.
- A Review of First Man, the Movie - The book was great. The movie? Nope.
- A Review of The Overstory - One of the most remarkable novels I've read in a long time.
- What I Learned About Successful Consulting - Lessons learned about successful consulting.
- Low Power Mischief - Ultra-low power systems are trickier to design than most realize.
- Thoughts on Firmware Seminars - Better Firmware Faster resonates with a lot of people.
- On Evil - The Internet has brought the worst out in many.
- My Toothbrush has Modes - What! A lousy toothbrush has a UI?
- Review of SUNBURST and LUMINARY: An Apollo Memoir - A good book about the LM's code.
- Fun With Transmission Lines - Generating a step with no electronics.
- On N-Version Programming - Can we improve reliability through redundancy? Maybe not.
- On USB v. Bench Scopes - USB scopes are nice, but I'll stick with bench models.