Go here to sign up for The Embedded Muse.
logo The Embedded Muse
Issue Number 253, January 20, 2014
Copyright 2014 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com
   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for noncommercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go to https://www.ganssle.com/tem-subunsub.html or drop Jack an email.

Editor's Notes



Did you know it IS possible to create accurate schedules? Or that most projects consume 50% of the development time in debug and test, and that it’s not hard to slash that number drastically? Or that we know how to manage the quantitative relationship between complexity and bugs? Learn this and far more at my Better Firmware Faster class, presented at your facility. See https://www.ganssle.com/onsite.htm.

Quotes and Thoughts


"Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris." - Larry Wall

Tools and Tips

Please submit neat ideas or thoughts about tools, techniques and resources you love or hate.

A Little More on Licensing

Paul E. Bennett wrote:

I have been catching up on a few of the back-issues of the Muse as I hadn't seen it posted to me recently. You have an interesting discussion on licensing going on from which the comment by Vlad Zeylikman where he said "I also have a pet peeve: engineers should be taught history of engineering, from Chaldeans to Romans, to industrial revolution.  But that is not going to happen...."

I think that all engineers should be encouraged to read about engineering disasters in order to learn from the mistakes others have made already and ensure they do not repeat them. I have recently been going through the testimonies of Philip Koopman and Mike Barr regarding the Toyota UA cases. The big lesson to be learned from that is how to recognise when you have a potential single point of failure in your system.

However, the safety aspects are much wider than the engineering staffs in a company. The company should, itself, have to pass rigourous assessments to put something out to market that can impact the safety of people. So, focusing on how you prove your product is safe should be occupying the mind of everyone in the embedded systems field. It seems that something went terribly wrong with the process at Toyota. It may be some time before we gain more insight into what (given the secrecy over their embedded systems efforts).

I am not sure licensing of engineers would help. Technology moves on at such a pace these days I am not sure the licensing authorities could keep up. Better engineering education helps but that means good practical experience to temper the theoretical learning as well. I am one who came through an old fashioned apprenticeship. My IEng status was based on nearly 40 years of experience and exploration of my thinking and credentials in a review panel. Gaining a PE straight out of University is probably no real guarantee of excellence but one gained after some field experience might stand more credibility.

When is Four not Four?

John Carter responded to the joke in the last issue where the value of pi is set in a #define in case pi should ever change:

I feel guilty stealing a colleague’s story... but perhaps he wouldn't want to relate it, as his name may give away who did this....

Apparently he worked with a brilliant physicist, whose code was brilliant, but unmaintainable. Thus the software engineers gathered together and explained to him ideas like named constants etc. In the next version of the code there appeared this line:

#define FOUR 4

Sadly enough, when he looked at the code some months later, it read:

#define FOUR 5

Perhaps a little more detail and precision is required when advising people to name constants.

The Case of the Crashing 68000

It was a dark and stormy night in the port of Baltimore. Angel, my leggy receptionist, bleeped me on the squawk box: "Spike's here, Jake. And my paycheck bounced again. I've quit better jobs than this!"

They call me Jake in the sleazy end of town, where the streets of the dirty city tell stories of pathos and passion. Edgar Allen Poe lived and died a dirty tramp in a gutter not far away. Old rotting buildings still testify to the port's seamier history, a history no amount of urban renewal can remove.

"Cool it, Angel. Show the creep in and leave me alone."

Spike stumbled into the old armchair that looked like it went four rounds with an angry bear. A bottle of Captain Morgan fell out of his pocket and shattered on the floor, creating a spicy odor that overwhelmed the scent of mildew.

"It's bad, Jake," he mumbled, scratching the gray stubble on his face. "The Kid just can't make it work. If he doesn't get that thing fixed the customers will off him for sure."

Great - just after I duct taped the windows destroyed in the last round of tommy gun fire. At least they only nicked my monitor; once a round went right into the hard disk and ricocheted off a file of backup tapes.

I poured him a shot. He tossed it back, fast, his hands shaking as he slammed the now empty glass on the table. "Jake, this system is killing me. The last set of PC boards were no good. Now we're almost ready to ship it but every few hours the 68000 crashes. I just don't know what we're going to do." As he said this Spike hung his head down, elbows on his knees, a defeated man in a dirty business where the rules change daily and the spec sheets are mostly written in Japanese.

"Tell me more, Spike," I offered, "when does this little bastard crash?"

"Never in the main loop, Jake" he replied. "Only when the command interpreter goes out to service a packet, and then only sporadically. We can execute millions of cycles without a problem, then boom - it's history."

"Ya plan to execute the thing, huh?" I replied... and then realized he didn't mean it quite so literally. "I think it's time we call in Bruno."

"No! Never! I swore I'd never work with him again!" he screamed in a thin whine. "I hate those highly paid consultants! They always make us look so stupid!"

I placed the call. "$4000 a day, plus expenses. In advance," was the gruff reply before the phone slammed down. We waited for his white limo.

Bruno muscled his bulk though the door, a door wide enough for the usual slob but one that seemed pitifully narrow now. Bruno's face was a mass of scars that I knew he had won in a score of battles - the big one on his forehead was from IBM on the ATC project, that slash on his cheek came from the abandoned Sergeant York. Like a teutonic swordsman's token Heidelberg scar, he wore these proudly. His leather briefcase banged on the floor.

"Hargmpfh!" Bruno was never much for words, but put him in front of a keyboard and his thick fingers were as graceful as Angel doing ballet.

"Well Bruno, you know the score. The system fails once in a while and the engineers have no idea why. It's a new design, but it looks pretty good for an outfit like this."

Bruno's hand crashed down on my desk, making the clip pop out of my .45. "Where's my money?" he rumbled. I scribbled a check which he held to the light for a minute before pocketing it and clumping off down to the lab.

We followed - Spike, slouching and mumbling, me with both hands on my wallet.

The door to the lab creaked open and we walked down the rickety stairs to its floor. Arcs of electricity flowed up Jacob's ladders, while over in one corner a team continued to try and reanimate Elvis's corpse. We ignored the stench and moved to a low slung table with a solitary engineer working feverishly over a small computer system.

"What's dis?" Bruno asked, pointing to a 68000-based factory controller connected via flat ribbon cables to a cage full of STD bus cards.

The Kid stared up at Bruno, fascinated and immobilized by fright, like a deer caught in the headlights of my Packard. His necked let out a sharp click as he looked down again and told Bruno that this was the source of all of his trouble. For three months the Kid had been debugging the hardware and software. He knew he was under the gun - literally - as we had hired him to replace an older engineer who made too much money. I saw the outcast just last week, hustling quarters in the street, and barely recognized him. His sign "can you spare a dime? I know calculus!" gave him away as one of those poor fools who had not learned to quit engineering by age 25.

Trying to be helpful I asked the Kid what he thought the problem might be. "I dunno. I've looked at everything. The timing is perfect, the voltage levels are fine. It doesn't make sense!"

With a hand like a side of ham Bruno grabbed the scope probe from the Kid's trembling fingers and started looking at different test points.

"You ain't got no decent gear," Bruno complained. "How am I gonna work with dis lousy 100 Mhz scope - it ain't even digital!" I made a quick note to have Spike hit the local test equipment store that night after it closed. Spike would have to be careful, because the cops were starting to get wise to us. They knew there was a pattern to the rash of test equipment capers on the south side.

Bruno pulled out a phone and ordered his driver to bring in the 1 GHz four channel beauty I just knew he had stashed in the limo. Then I started listening to Bruno mutterings. At $4000 a day, plus expenses, we had to learn a lot from the big brute.

"Them low frequency scopes with 100 Mhz probes can't never really see what's on a high speed bus. I bet dis here sucker is oscillating, like real fast. Real fast," he muttered over and over.

Bruno propped his scope on the back of the liveried chauffeur. He probed the address lines - they looked pretty good to me, tristating as always as a processor enters a bus hold mode. The data lines were awful but data always is a mix of ones, zeroes, and tri-state conditions as the memories decode chip selects and output enables. Only the control lines - read, write, address strobe - seemed to present solid levels all of the time. Still, in a lifetime spent in the grungiest labs in the meanest cities this all looked pretty familiar to me. "What do you think?" I queried.

Bruno's head slowly revolved around to glare at me while the rest of his huge body remained motionless. "Don't rush a expert. I's a master artisan, and I need time to think."

"You have that scope set all wrong," the kid whined, "I always set the vertical channel on 2 v/cm. You've got it on 1 V/cm!" "Shadup, Kid," Bruno said as he brutally shoved the Kid back onto his stool. "Yoose guys gotta understand that digital circuits are really analog. Ya can't see good enough on a 2V/cm setting to tell if a signal is above the legal minimum one level, or below the legal maximum zero. Hey I betcha don't even know what voltage a logic one is, anyway?"

"CMOS or TTL, HCT or AUC?" shot back the Kid.

"Good answer, Kid. Each logic family has its own levels, and ya gotta make sure ya obey the rules. With the scope set to 1V/cm I can see in an instant if any of these levels fall outside of the legal range. Ya know how I hate being illegal. Besides, dat's just the sort of problem dat will cause dis kind of intermittent."

Spike spoke up: "if it's intermittent then maybe it's a broken track or something."

"Yeah. Lemme see." Bruno lifted the board and flexed it gently, all the time watching the monitor. The system kept running. Then he turned the board over and ran his fingers gently over all of the pins. I stared fascinated at the dozens of tiny scars on his fingertips.

"It's dem sharp through-hole leads. Dey cut me up, bad, like Roscoe da Razor did that time in Vegas." Later I learned that Roscoe was in the joint for passing counterfeit Pentiums. I also learned that Bruno was searching for unconnected input pins that may have drifted to the right state through luck, or 'cause someone up there watches over drunks, sailors, and swill like me.

"Da impedance of any digital output is pretty low. My paws shouldn't mess up the circuit at all," Bruno went on. Despite his almost sensual stroking of each pin the circuit continued to run without fail.

Now Bruno hooked the 1 GHz scope to one data line, and carefully connected the probe's very short ground lead to the IC's ground. He made sure the bandwidth limiter was off, triggered on the read line, and tuned the scope's trigger controls like an old-time ham pulling in a very weak signal. A low growl alerted me that something was up. He checked another data line, and then another.

"Dis ting is oscillating", Bruno roared. "What, yoose got amateurs designing dis stuff now? Watch - I triggered da scope on da read pulse. Look at the data line after da read pulse goes away."

"No one does that," snapped the Kid. I didn't know what Bruno was getting at, but having seen him in a rage before, I kept mum. I know for a fact that the pub owner now wishes he had too.

Bruno's neck started turning red. I backed slowly away. "Of course da signal's OK during read, you louse. After read, when da bus tristates, it's oscillating at about 450 Mhz. Dat'll crash yoose system. Pull-up da data bus with some resistors in a SIP."

"No way, Bruno. The data books only talk about the data bus during read. Who cares what happens during a tri-state condition?"

"Add da SIPs," Bruno demanded. The Kid refused again. Bruno whipped out a 9mm and held it to the Kid's temple. "Da last someone dat messed with my ideas is pushin' up daisies," he rumbled.

I rushed forward to solder the SIPs in myself. The lab was already a mess, but if Bruno offed this poor kid the cops might hear the shot. "Now we go to your office and wait," Bruno intoned, "I've set da system up to run all night to see if it is fixed. Look - the oscillations are gone, and I'll bet you a pair of brass knuckles it's OK now."

For 24 hours we sat, facing each other across the beat up oak conference table, Bruno's eyes boring into mine. He said nothing; I replied in kind. Each twitch escalated into a near shootout as we nervously watched each other's movements, my heater poised under the table, his likewise close at hand. For 24 hours we waited, wondering if this highly paid outsider was worth his $4000 a day, plus expenses.

The new day dawned; the squawk box reported success with the SIPs; a full report was on its way up.

Minutes later, carrying a stack of engineering notebooks, Elvis strode into the room.

Ongoing Education

Software engineering is not a discipline. Its practitioners cannot systematically make and fulfill promises to deliver software systems judged by their customers as usable and dependable, on time and fairly priced. The illusion that software engineers possess a discipline has produced the major breakdown called the software crisis.

This quote, from a 1992 paper written by Peter Denning and Pamela Dargan ticked me off the first time I read it. The authors essentially claim that we're neither capable nor professionals.

Then I read it again. Well, yeah, we can't actually fulfill promises, the software usually isn't on-time, yup, and it's often not all that useable.

Maybe they have a point. Are we professionals?

Dictionary.com defines the noun form of "professional" as a person following a profession, especially a learned profession. Though the definition is annoyingly recursive, the sense is that professionals are the highly educated wizards who routinely solve tough problems. Like an engineer, right?

Well, maybe not. Compare us to other professionals. Doctors, for instance -- most people endow those authorities with the "professional" moniker. Sure, they have an enormous amount of education, but perhaps the defining aspect of their career, the one that truly merits the word "professional," is their on-going education. Somehow doctors manage to incorporate the vast amounts of new research into their daily practices. Can you imagine being a doctor who learned medicine in the 1940s, before penicillin, who continued to practice through the 50s, 60s, 70s and 80s using the same skills he learned so long ago?

He'd be considered a butcher.

How about teachers? I've always considered them professionals. In most states they must take a certain number of continuing education credits every year or two to maintain their certifications.

Plumbers, hair stylists, electricians and numerous blue collar jobs require certification and licensure.

We design nuke controllers, avionics, implanted medical devices, and if you can spell C, you've got a job.

Embedded is a stealth industry today. Consumers might be vaguely aware of the buried processor but give it little mind-share. As electronics becomes ever more pervasive, and as critical systems suffer more failures (for instance, pacemaker recalls due to firmware problems are going up) legislators will wield the usual blunt ax.

We can -- we must - stave off this impending revolution that will hit all of our careers, but only by acting more professionally, by actively soaking up new ideas, concepts and approaches to building systems.

All of us are smart folks who are experts at learning new skills, like how to use a different processor or building faster ISRs. But we too often fail at finding and embracing new processes that can help us make better products.

For example -- are you familiar with the CMM? In my opinion every professional software developer should understand the Capability Maturity Model. The usual chorus will loudly protest here. Sure, the CMM is bloated and flawed. But it contains useful and essential ideas that we can incorporate into our daily work.

Have you read Watts Humphrey's A Discipline for Software Engineering... and done the homework? It's a tough slog but the benefits are amazing. Though no silver bullet, the mechanisms Humphrey diagrams give big benefits to individual programmers.

How about any of the books about agile methods? Have you adopted any of the concepts? Though a few in the agile community makes whacky claims, and I remain unconvinced about some of the practices, there's a wealth of valuable ideas that will lead to better systems.

Are you a member of the ACM or IEEE? Both offer a wide range of useful (and some deadly dull) journals.

Books (here's my reviews of some), classes, conferences, magazines and on-line resources abound. Are you truly engaged with your profession?

If not... why not?


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intents of this newsletter. Please keep it to 100 words.

Joke For The Week

Note: These jokes are archived at www.ganssle.com/jokes.htm.

This is not exactly a joke; this is a Black Friday Deal that Zach Smith ran across recently:



Advertise With Us

Advertise in The Embedded Muse! Over 23,000 embedded developers get this twice-monthly publication. .

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.