Jack Ganssle, Editor of The Embedded Muse Jack Ganssle's Blog
RSS Feed This is Jack's outlet for thoughts about designing and programming embedded systems. It's a complement to my bi-weekly newsletter The Embedded Muse. Contact me at jack@ganssle.com. I'm an old-timer engineer who still finds the field endlessly fascinating (bio).

For novel ideas about building embedded systems (both hardware and firmware), join the 40,000+ engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype and no vendor PR. Click here to subscribe.

Lessons From a Failure

April 10, 2020

George Farmer reports on yet another embedded failure, and draws some lessons that should have been learned:

A few weeks back I was first in line for a touchless car wash at one of our local gas stations, waiting on a person who was already in the bay when I pulled up.  Both the entrance and exit doors were closed, and it appeared to me the wash cycle was wrapping-up just as I arrived.  I saw the green light turn on, signaling the driver to proceed forward through the dryers.  This is where the story begins.

While I was waiting I dutifully slipped my prepaid carwash card into the reader and selected "Touchless."  It is here that a red flag should have registered in my old, tired grey matter, but at 5:45 am and no caffeine in my system, I was an automaton.  Had I been alert, at that moment no one was behind me at the time, and I could have backed up to a nearby exit point and simply left.  But no!  This is Wisconsin after all, and warmer, dryer weather in February is a rare thing, so I was determined to get several weeks and countless layers of crusty salinity off my wife's car.  Besides, I don't like our cars being salt licks for the local deer population.

For starters, the kiosk didn't spit out a receipt like it normally does.  "No Biggie," I thought, "This has happened before."  Also, the "Soft Touch"  and "Touchless" selection buttons were reversed;  or, at least that's what the voice prompt indicated compared to their electronic display labels.  I canceled my selection and started again.  This time the selections and voice prompt lined up normally, so I chalked the previous faux pas up to mindless operator error and temporary brain death due to the lack of caffeine (there is no life before coffee).

OK, time to enter the car wash.

I'm waiting... and waiting... and... wait! 

The person inside the bay isn't moving off the wheel chocks.  Hmmm...  Is this person stuck?  Engine died?  Not paying attention?  Needs medical attention?

No idea.

Normally the dryers sound like turbofan jet engines and are turned on just as the light turns green, but I didn't hear them.  Finally it dawned on me the person was held captive in position because the robotic spray head boom suspended between two gantry stanchions hadn't retracted and was blocking the car from pulling forward.  Not only that, but the left robotic gantry stanchion was positioned within about one foot of the driver's door.  Both stanchions also have high-pressure spray heads, so I suspected the person understandably didn't want to roll down their window or try to get out of the car - a smart move indeed.  I could not see the person through the fogged bay door and back car window, so I punched the Help button on the Kiosk several times to alert someone inside.  I then beeped the horn quickly a couple of times and flashed my brights, just to see if this got the attention of the driver.

Nothing.  Nada.  Nor was there a response from pressing Help call button (it's an intercom). 

At this point I wanted to back up a bit so at least I could get out of my car and go inside for help.  Unfortunately a young driver had pulled up very close behind me, blasting music, and was staring down at her phone (I could see the eerie glow of the screen reflecting off her face).  No amount of yelling and waving from my car window did any good, so now I'm stuck and can't seem to get anyone's attention.

Fortunately a Good Samaritan at one of the pumps could see the situation unfolding and went inside for help.  A few minutes later the manager of the store came out to investigate.

First thing the manager tried to do was raise the entrance door via a keypad code.  No luck.  She went back inside for some keys and came back to enter a typical manual door at the side of the building.  Successful entry this time (gotta love reliable, Old School technology).

After verifying the driver was OK, she went over to a large utility box on the wall - inside the wash bay - opened it, and must have flipped a switch or two.  After about twenty seconds or so both doors opened, and the gantry moved to a Home position, releasing the driver from the clutches of The Robotic Beast.  The driver departed with a sopping-wet car.

By this time several more cars had pulled up behind the young driver who was blocking me.  I seriously doubt the young girl had even looked up from her phone to see what was happening.  The manager came over to my car and asked if I had already paid - I answered in the affirmative.  She told me the wash had been completely reset and should be fine now.  Since the only way out of the jam was to move forward, I reluctantly decided to press my luck.

The initial underbody wash as I slowly drove in seemed to work fine, so I figured I was in the clear.  The green light switched to red as my front wheels landed in the wheel chocks.  "So far, so good," I thought.

BAD ASSUMPTION.  And so the fun began.

Immediately the gantry and spray bar between the two stanchions began an unusual series of gyrations typically only seen in an improv dance studio: total chaos.   What alarmed me most was the rotating spray bar that crosses over the car as the gantry stanchions move fore/aft: "Is the bar going to impact my car," I thought?  Fortunately it didn't.

The amusing part of this spectacle was the spray bar rotated in the wrong direction most of the time, spraying soap and water all over the place, except onto my car.   The gantry bounced back and forth as if totally confused, eventually stopping about mid-span of my car, fortunately with the spray bar well above the car's roof.  The high-pressure jets on the stanchions turned on only briefly - certainly not long enough to clean the entire length of the car.  I thank my lucky stars I didn't purchase the so-called Elite wash, as this also included a robotic hub cap cleaning using rotating brushes.  I have no idea where these brushes would have landed on my car's side panels. 

After only a short time on the chocks the light turned green, the dryers spun-up and, unlike the unfortunate driver before me, both doors opened.  I zoomed off the chocks as quickly as possible, but upon arrival at the dryers, they shut off.  I, too, exited with a mostly wet-but-not-so-clean car, happy to be clear from the clutches of a psychotic robot.

Murphy's Law!

By the way, the young driver behind me must have finally pried her eyes away from her phone, as she pulled into the bay just as I was exiting.

After reporting the experience to the store manager and warning there may be another unsuspecting customer trapped in the bay, I headed home with a free carwash receipt in-hand, but with a 2-tone car (colors of dark blue and salt, in a random, tie-dye color pattern).

On my way home I found my mind slowly coming alive, and I began talking Shop to myself:

Who did the Failure Modes and Effects Analysis (FMEA) on the design of that robotic beast?  Was there even an FMEA done on the design?

What are the fail-safes (if any)?  Watchdogs?  Sanity checks on the sensors?  Level of redundancy?

Why was the utility box - which appeared large enough to contain the controller - located inside the bay!?  It may be a sealed box, but certainly not hermetically sealed!  Were the PC boards conformal-coated?  Were Sealtights used?

Who inspected this installation?

Was anyone automatically alerted from inside the store?  There should have at least been some type of Deadman alarm inside the store!  Hey!  How about a tri-color light outside the wash bay:  Green = Ready, Yellow = in-use, Flashing Red = Alarm ("Danger, Will Robinson!"), Totally Off = Potentially Serious Trouble.

Who designed the drive-up?  It's CAPTIVE!  There should be a way to leave the kiosk by pulling forward, not backward!

Ok... The Wifey says I need to get out of the house more often.  Keep your eyes on the road!

It's too easy to Armchair-Quarterback someone else's design.  It's also virtually impossible for a single engineer - regardless of their formal training and cross-functional experience - to think of every possible failure scenario.

I also realize from first-hand experience that, more often than not, both engineers and their management are under extreme pressure to deliver the goods faster, better and, as always, cheaper.  After all, someone high-up in the Food Chain must meet their quarterly figures in order to receive their annual BAB (Big-Ass Bonus), thereby keeping those buissjet companies in business.

Ok… That last one was a cynical cheap shot - my apologies.  However, this leads me to a point:  Greed and arrogance usually gets people into the trouble, and ultimately the company in the long run.  As for greed, well, there are countless books out there on this subject, dealing with ethics, failed B-school business paradigms, etc., so I won't go down this path.  As for arrogance, however, maybe not so much. 

When I think of arrogance, the concept of Design Ownership comes to mind.  A good engineer takes Design Ownership seriously.  However, a very wise individual once said, "There's a fine line between pride and arrogance; the trick is to keep the pride and lose the arrogance."  It's too easy to mistake arrogance in Design Ownership with pride.

It's often said that Pride is one of the seven deadly sins (along with Greed, Sloth, etc.), although I can't help but wonder if the original authors really meant Arrogance in place of Pride.  I can truly understand why an engineer may feel a sense of pride in their designs - especially if they had solved some particular challenge along the way.  However, that sense of pride should always be tempered with a healthy dose of reality in that all designs can and almost certainly will fail, given enough time or under the right circumstances.  It's not a matter of if, but when.  Without this sense of reality it's too easy for Pride to slip into Arrogance.  There are WAY too many historical references where arrogance clouded even the finest engineering organization (The White Star Line for the Titanic, or NASA's Challenger and Columbia disasters come to mind, for example – there are thousands more).

Engineers and their management must realize that FMEAs play a vital and critical role in all aspects of design, as well as after design release, production begins, and product hits the streets.  FMEAs are living entities that must be maintained throughout the product lifecycle, not simply put on the shelf as someone's checkbox schedule item.  Additionally, FMEAs must be done in a cross-functional environment, with all major stakeholders present in one room.  The engineers and their management need to check their weapons and arrogance (often mistaken for Pride) at the door!  It's not an overstatement to say an FMEA - if done correctly - is a humbling experience.

Lastly, the concept of Design Ownership tends to get lost within a company's chain of command.  Indeed,  it's too often the case where culpability simply isn't even in the vernacular or business acumen above a certain level - although "Risk Taking" is often found in the acumen.  Risk has become a virtue.  Trouble is, many people simply don't understand or appreciate what Risk truly is, let alone how to calculate it.  Indeed, engineers are often chastised for being too risk adverse.  I say this is malarkey.

Engineers take risks on a daily basis, since there is no such thing as Design Perfection.  I know I'm preaching when I say, for many reasons, there are ALWAYS design tradeoffs that must be managed, mitigated, and a design path chosen.  The best we engineers can hope to achieve is an optimal design: one that is safe, meets all customer requirements at the lowest possible cost, is reliable, on-time, etc.  However, there is a HUGE difference between taking a calculated risk - one that has been mitigated to as low as reasonably achievable - and taking a flippant risk based on arrogance or greed (or both).  Regarding arrogance and greed, there are too many cases where people and corporations aren't held accountable. This needs to change.

For what it's worth, I have not been back since to use my free carwash :-O

Feel free to email me with comments.

Back to Jack's blog index page.

If you'd like to post a comment without logging in, click in the "Name" box under "Or sign up with Disqus" and click on "I'd rather post as a guest."

Recent blog postings: