Embedded Muse 58 Copyright 2000 TGG December 9, 2000
You may redistribute this newsletter for noncommercial purposes. For commercial use contact email@example.com.
EDITOR: Jack Ganssle, firstname.lastname@example.org
- Editor’s Note
- Ford Explorer
- Chinook Helicopter
- Thought for the Week
- About The Embedded Muse
In this issue I present three cases of products that suffered from serious design problems. Not to gloat. Rather, I think we embedded folks need to seriously look at failures in our systems, and find patterns that teach us better ways to produce products.
I actively collect embedded disaster stories. There are many lessons one can draw from the collection. Probably the three biggest problems that run through these disasters are:
- bad design that isn’t caught due to poor inspections
- inadequate testing
- poor error and exception management
So, enjoy the stories but do take the lessons to heart!
The Ford Explorer has received a certain notoriety due to problems with Firestone tires. The embedded system controlling this car is also a source of problems.
In a recent story (December 2000) the Baltimore Sun reports that engine controller’s code should limit the vehicle’s top speed to 106 miles an hour (wow!). A bug, though, allows the car to exceed 112 MPH. The problem? At that speed… the tires fail!
Software upgrades are available to all owners.
In June of 2000 some Explorers sold in Canada had another problem that causes the “Generic Electronic Module” to crash. This disables the air bags, windshield wipers, lighting, etc. The cure involves installing a resistor, to “prevent electronic noise”. A missing pull-up? I see dozens of systems a year where missing pull-up resistors cause erratic operation or crashes.
A fascinating report describes an audit conducted of the Chinook’s engine control software. EDS was contracted to review the code, but actually gave up after looking at less than 20% of the source. According to the report, EDS “abandoned the work because of what one executive said was the density of the anomalies found”. In other words, there were so many problems it made little sense to proceed.
This review was of released code; software that was in use in many flying Chinooks.
They found 485 defects of varying importance. According to EDS the fault levels were an order of magnitude or more than should be expected in “a rigorously developed safety critical system”. Unfortunately there’s no indication of program size; 485 defects in 10,000 lines of code would be appalling, in 10 million lines perhaps more understandable.
When a review board examines source code they often find many defects, as in the Chinook and several recent Mars missions. What strikes me so powerfully is that these boards use the same sorts of tools (like inspections) that proper software engineering should use. Why are we willing to skip proper engineering while building a product, and then resort to it to figure out what went wrong after disaster strikes?
Thought for the Week
The Man From Microsoft
There was a knock on the door. It was the man from Microsoft.
"Not you again," I said.
"Sorry," he said, a little sheepishly. "I guess you know why I'm here."
Indeed I did. Microsoft's $300 million campaign to promote the Windows95 operating system was meant to be universally effective, to convince every human being on the planet that Windows 95 was an essential, some would say integral, part of living. Problem was, not everyone had bought it. Specifically, I hadn't bought it. I was the Last Human Being Without Windows 95. And now this little man from Microsoft was at my door, and he wouldn't take no for an answer.
"No," I said.
"You know I can't take that," he said, pulling out a copy of Windows 95 from a briefcase. "Come on. Just one copy. That's all we ask."
"Not interested." I said. "Look, isn't there someone else you can go bother for a while? There's got to be someone else on the planet who doesn't have a copy."
"Well, no," The Microsoft man said. "You're the only one."
"You can't be serious. Not everyone on the planet has a computer," I said. "Hell, not everyone on the planet has a PC! Some people own Macintoshes, which run their own operating system. And some people who have PCs run OS/2, though I hear that's just a rumor. In short, there are some people who just have no use for Windows 95."
I'll present my Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. The early registration discount ends January 20.
"Well, I don't know anything about this 'use' thing you're going on about," The Microsoft man said. "All I know is that according to our records, everyone else on the planet has a copy."
"People without computers?"
"We had to get some malaria shots to go in, but yes."
"Oh, come on," I said. "They don't even wear BUTTONS. How did you get them to buy a computer operating system?"
"We told them there were actually 95 very small windows in the box," the Microsoft man admitted. "We sort of lied. Which means we are all going to Hell, every single employee of Microsoft." He was somber for a minute, but then perked right up. "But that's not the point!" he said. "The point is, EVERYONE has a copy. Except you." "So what?" I said. "If everyone else jumped off a cliff, would you expect me to do it, too?"
"If we spent $300 million advertising it? Absolutely."
"Jeez, back to that again," the Microsoft man said. "Hey. I'll tell you
what. I'll GIVE you a copy. For free. Just take it and install it on your computer." He waved the box in front of me.
"No," I said again. "No offense, pal. But I don't need it. And frankly, your whole advertising blitz has sort of offended me. I mean, it's a computer operating system! Great. Fine. Swell. Whatever. But you guys are advertising it like it creates world peace or something."
"World peace. It was part of the original design. Really. One button access. Click on it, poof, end to strife and hunger. Simple."
"So what happened?"
"Well, you know," he said. "It took up a lot of space on the hard drive. We had to decide between it or the Microsoft Network. Anyway, we couldn't figure out how to make a profit off of world peace."
"Go away," I said.
"I can't," he said. "I'll be killed if I fail."
"You have got to be kidding," I said.
"Look," the Microsoft man said, "We sold this to the AMISH. The Amish! Right now, they're opening the boxes and figuring out they've been had. We'll be pitchforked if we ever step into Western Pennsylvania again. But we did it. So to have YOU holding out, well, it's embarrassing. It's embarrassing to the company. It's embarrassing to the product. It's embarrassing to BILL."
"Bill Gates does not care about me," I said.
"He's watching right now," the Microsoft man said. "Borrowed one of those military spy satellites just for the purpose. It's also got one of those high-powered lasers. You close that door on me, zap, I'm a pile of gray ash."
"He wouldn't do that," I said, "He might hit that copy of Windows 95 by
"Oh, Bill's gotten pretty good with that laser," the Microsoft man said, nervously. "Okay. I wasn't supposed to do this, but you leave me no choice. If you take this copy of Windows 95, we will reward you handsomely. In fact, we'll give you your own Caribbean island! How does Montserrat sound?"
"Terrible. There's an active volcano there."
"It's only a small one," the Microsoft man said.
"Look," I said, "even if you DID convince me to take that copy of Windows 95, what would you do then? You'd have totally saturated the market. That would be it. No new worlds to conquer. What would you do then?"
The Microsoft man held up another box and gave it to me.
"'Windows 95....For Pets'?!?!?"
"There's a LOT of domestic animals out there," he said.
I shut the door quickly. There was a surprised yelp, the sound of a laser, and then nothing.