The Case of the Crashing 68000

Do you suffer from erratically crashing products? Feel like management is about ready to heave you to the wolves? Check out Jake's tale...

For hints, tricks and ideas about better ways to build embedded systems, subscribe to The Embedded Muse, a free biweekly e-newsletter. No hype, just down to earth embedded talk. 23,000 other engineers subscribe. It takes just a few seconds (all we need is your email address, which is shared with absolutely no one) to subscribe to the Embedded Muse.

It was a dark and stormy night in the port of Baltimore. Angel, my leggy receptionist, bleeped me on the squawk box: "Spike's here, Jake. And my paycheck bounced again. I've quit worse jobs than this!"

They call me Jake in the sleazy end of town, where the streets of the dirty city tell stories of pathos and passion. Edgar Allen Poe lived and died a dirty tramp in a gutter not far away. Old rotting buildings still testify to the port's seamier history, a history no amount of urban renewal can remove - nor should it.

"Cool it, Angel. Show the creep in and leave me alone."

Spike stumbled into the old armchair that looked like it went four rounds with an angry bear. A bottle of Captain Morgan fell out of his pocket and shattered on the floor, creating a spicy odor that overwhelmed the scent of mildew.

"It's bad, Jake," he mumbled, scratching the gray stubble on his face. "The Kid just can't make it work. If he doesn't get that thing fixed the customers will off him for sure."

Great - just after I duct taped the windows destroyed in the last round of tommy gun fire. At least they only nicked my monitor; once a round went right into the hard disk and ricocheted off a file of backup tapes.

I poured him a shot. He tossed it back, fast, his hands shaking as he slammed the now empty glass on the table. "Jake, this system is killing me. The last set of PC boards were no good. Now we're almost ready to ship it but every few hours the 68000 crashes. I just don't know what we're going to do." As he said this Spike hung his head down, elbows on his knees, a defeated man in a dirty business where the rules change daily and the spec sheets are mostly written in Japanese.

"Tell me more, Spike," I offered, "when does this little bastard crash?"

"Never in the main loop, Jake" he replied. "Only when the command interpreter goes out to service a packet, and then only sporadically. We can execute millions of cycles without a problem, then boom - it's history."

"Ya plan to execute the thing, huh?" I replied... and then realized he didn't mean it quite so literally. "I think it's time we call in Bruno."

"No! Never! I swore I'd never work with him again!" he screamed in a thin whine. "I hate those highly paid consultants! They always make us look so stupid!"

I placed the call. "$4000 a day, plus expenses. In advance," was the gruff reply before the phone slammed down. We waited for his white limo.

Bruno muscled his bulk though the door, a door wide enough for the usual slob but one that seemed pitifully narrow now. Bruno's face was a mass of scars that I knew he had won in a score of battles - the big one on his forehead was from IBM on the ATC project, that slash on his cheek came from the abandoned Sergeant York. Like a Teutonic swordsman's token Heidelberg scar, he wore these proudly. His leather briefcase banged on the floor.

"Hargmpfh!" Bruno was never much for words, but put him in front of a keyboard and his thick fingers were as graceful as Angel doing ballet.

"Well Bruno, you know the score. The system fails once in a while and the engineers have no idea why. It's a new design, but it looks pretty good for an outfit like this."

Bruno's hand crashed down on my desk, making the clip pop out of my .45. "Where's my money?" he rumbled. I scribbled a check which he held to the light for a minute before pocketing it and clumping off down to the lab.

We followed - Spike, slouching and mumbling, me with both hands on my wallet.

The door to the lab creaked open and we walked down the rickety stairs to its floor. Arcs of electricity flowed up Jacob's ladders, while over in one corner a team continued to try and reanimate Elvis's corpse. We ignored the stench and moved to a low slung table with a solitary engineer working feverishly over a small computer system.

"What's dis?" Bruno asked, pointing to a 68000-based factory controller connected via flat ribbon cables to a cage full of STD bus cards.

The Kid stared up at Bruno, fascinated and immobilized by fright, like a deer caught in the headlights of my Packard. His neck let out a sharp click as he looked down again and told Bruno that this was the source of all of his trouble. For three months the Kid had been debugging the hardware and software. He knew he was under the gun - literally - as we had hired him to replace an older engineer who made too much money. I saw the outcast just last week, hustling quarters in the street, and barely recognized him. His sign "can you spare a dime? I know calculus!" gave him away as one of those poor fools who had not learned to quit engineering by age 25.

Trying to be helpful I asked the Kid what he though the problem might be. "I dunno. I've looked at everything. The timing is perfect, the voltage levels are fine. It doesn't make sense!"

With a hand like a side of ham Bruno grabbed the scope probe from the Kid's trembling fingers and started looking at different test points.

"You ain't got no decent gear," Bruno complained. "How am I gonna work with dis lousy 100 MHz scope - it ain't even digital!" I made a quick note to have Spike hit the local test equipment store that night after it closed. Spike would have to be careful, because the cops were starting to get wise to us. They knew there was a pattern to the rash of test equipment capers on the south side.

Bruno pulled out a flip phone and ordered his driver to bring in the 1 GHz four channel beauty I just knew he had stashed in the limo. Then I started listening to Bruno mutterings. At $4000 a day, plus expenses, we had to learn a lot from the big brute.

"Them low frequency scopes with 100 MHz probes can't never really see what's on a high speed bus. I bet dis here sucker is oscillating, like real fast. Real fast," he muttered over and over.

Bruno propped his scope on the back of the liveried chauffeur. He probed the address lines - they looked pretty good to me, tristating as always as a processor enters a bus hold mode. The data lines were awful but data always is a mix of ones, zeroes, and tri-state conditions as the memories decode chip selects and output enables. Only the control lines - read, write, address strobe - seemed to present solid levels all of the time. Still, in a lifetime spent in the grungiest labs in the meanest cities this all looked pretty familiar to me. "What do you think?" I queried.

Bruno's head slowly revolved around to glare at me while the rest of his huge body remained motionless. "Don't rush a expert. I's a master artisan, and I need time to think."

"You have that scope set all wrong," the kid whined, "I always set the vertical channel on 2 v/cm. You've got it on 1 V/cm!"

"Shadup, Kid," Bruno said as he brutally shoved the Kid back onto his stool. "Yoose guys gotta understand that digital circuits are really analog. Ya can't see good enough on a 2V/cm setting to tell if a signal is above the legal minimum one level, or below the legal maximum zero. Hey I betcha don't even know what voltage a logic one is, anyway?"

"CMOS or TTL, HCT or HC?" shot back the Kid.

"Good answer, Kid. Each logic family has its own levels, and ya gotta make sure ya obey the rules. With the scope set to 1V/cm I can see in an instant if any of these levels fall outside of the legal range. Ya know how I hate being illegal. Besides, dat's just the sort of problem dat will cause dis kind of intermittent."

Spike spoke up: "if it's intermittent then maybe it's a broken track or something."

"Yeah. Lemme see." Bruno lifted the board and flexed it gently, all the time watching the monitor. The system kept running. Then he turned the board over and ran his fingers gently over all of the pins. I stared fascinated at the dozens of tiny scars on his fingertips.

"It's dem sharp through-hole leads. Dey cut me up, bad, like Roscoe da Razor did that time in Vegas." Later I learned that Roscoe was in the joint for passing counterfeit 486s. I also learned that Bruno was searching for unconnected input pins that may have drifted to the right state through luck, or 'cause someone up there watches over drunks, sailors, and swill like me.

"Da impedance of any digital output is pretty low. My paws shouldn't mess up the circuit at all," Bruno went on. Despite his almost sensual stroking of each pin the circuit continued to run without fail.

Now Bruno hooked the 1 GHz scope to one data line, and carefully connected the probe's very short ground lead to the IC's ground. He made sure the bandwidth limiter was off, triggered on the read line, and tuned the scope's trigger controls like an old-time ham pulling in a very weak signal. A low growl alerted me that something was up. He checked another data line, and then another.

"Dis ting is oscillating", Bruno roared. "What, yoose got amateurs designing dis stuff now? Watch - I triggered da scope on da read pulse. Look at the data line afterda read pulse goes away."

"No one does that," snapped the Kid. I didn't know what Bruno was getting at, but having seen him in a rage once before, I kept mum. I know for a fact that the pub owner now wishes he had too.

Bruno's neck started turning red. I backed slowly away. "Of course da signal's OK during read, you louse. After read, when da bus tristates, it's oscillating at about 450 MHz. Dat'll crash yoose system. Pull-up da data bus with some resistors in a SIP."

"No way, Bruno. The data books only talk about the data bus during read. Who cares what happens during a tri- state condition?"

"Add da SIPs," Bruno demanded. The Kid refused again. Bruno whipped out a 9mm and held it to the Kid's temple. "Da last someone dat messed with my ideas is pushin' up daisies," he rumbled.

I rushed forward to solder the SIPs in myself. If Bruno offed this poor kid the cops might hear the shot.

"Now we go to your office and wait," Bruno intoned, "I've set da system up to run all night to see if it is fixed. Look - the oscillations are gone, and I'll bet you a pair of brass knuckles it's OK now."

For 24 hours we sat, facing each other across the beat up oak conference table, Bruno's eyes boring into mine. He said nothing; I replied in kind. Each twitch escalated into a near shootout as we nervously watched each other's movements, my heater poised under the table, his likewise close at hand. For 24 hours we waited, wondering if this highly paid outsider was worth his $4000 a day, plus expenses.

The new day dawned. The squawk box reported success with the SIPs; a full report was on its way up.

Minutes later, carrying a stack of engineering notebooks, Elvis strode into the room.