You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Tin Whiskers
• Are Bugs a Problem - Given Quick Fixes? (Redux #2)
• The Vanishing Embedded Engineer?
• This Week's Cool Product
• Jobs!
• Joke for the Week
• Advertise with us
• About The Embedded Muse

Years ago a reader wrote in to tell about how he was on a plane waiting to come home. Maintenance was iteratively coming on the plane, and going off of it. Clearly there was a problem. Finally the captain came on the intercom and announced they were going to turn the aircraft off for thirty seconds, and then turn it back on.

It worked. They flew.

The past few issues of The Embedded Muse have included, at first my comments on the pursuit of firmware quality, and then readers' responses. Perfection is hard and even impossible to achieve. But it is, in my opinion, a goal to which we should aspire. Somehow the electronics industry has created a world where we toggle circuit breakers on planes to fix problems, and where every three-year-old knows if something electronical does something weird, cycle power. It seems getting a product to market is more important than getting it right.

Yet there's a tsunami of data that shows writing high-quality code shortens schedules. Why? Correct software drastically shortens debug and test sessions.

That's what my Better Firmware Faster seminar is all about. Find out how you can bring this class to your facility, to help your engineers achieve world-class code on a shorter schedule. More info is here.

On June 27-28 NIST will host the Sound Static Analysis for Security Workshop at NIST's facility in Gaithersburg, MD. To quote from their communications "this two-day workshop is focused on decreasing software security vulnerabilities by orders of magnitude, using the strong guarantees that only sound static analysis can provide. The workshop is aimed at developers, managers and evaluators of security-critical projects, as well as researchers in cybersecurity." I plan to attend. It's free, and there's more info here. (This link is corrected from the one I posted last issue).

Clyde Shappee sent a link to a number of (mostly) spaceflight-related quotes: http://spacecraft.ssl.umd.edu/akins_laws.html

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Tony Mactutis, KM7J, wrote:

Circuit Cellar just released a nice compendium of PCB vendors' design and quoting tools.

In the last issue I wrote: "A couple of interesting factoids he gave include that the "4" in FR4 PCB material means the speed of a wave on that PCB is about one quarter that of c in a vacuum." A number of readers disputed this. Doing some research it's pretty clear "FR" stands for "fire resistance," but the "4" remains, to me, obscure. It appears to be a grade of fire resistance, as suggested by this from Wikipedia: "FR-4 does not specify specific material, but instead a grade of material, as defined by NEMA LI 1-1998 specification". But nearly everyone pointed out that FR-4 has a permittivity of about 4, and the speed of light varies as the square root of that, so runs at about one-half c on an FR-4 PCB, not one-quarter. Carl Van Wormer linked to an EDN article which states:

This month, thanks to the folks at Expresslogic, we're giving away 8 copies of Real-Time Multithreading, a great book about RTOSes in general and ThreadX in particular.

Enter via this link.

Earlier this month I met with engineers at Goddard Space Flight Center in Greenbelt Maryland to talk about tin whiskers. It was a fascinating and scary discussion. What's even more alarming is that so few engineers are aware of the phenomena.

Tin whiskers (TW) are tiny filaments that grow from tin-plated surfaces. But it's not just tin: zinc, cadmium and perhaps other metals also form whiskers.

Whiskers were first discovered in the 1940s on cadmium-plated surfaces on air-gapped variable capacitors, like the ones once used to tune radios. They would grow long enough to short out plates, suddenly changing the capacitance and thus the frequency the radio was tuned to. Engineers were advised to avoid cadmium-plated surfaces and components, and to use tin or zinc instead.

Alas, later findings implicated those two materials as well.

These whiskers can grow from a tinned surface to short out a circuit. For instance:

The NASA folks have a rogues' gallery of TW-infected components.

TW shorting a connector pin to the connector's shell

TWs are an increasing problem in electronics today because of the EU's RoHS (Reduction of Hazardous Substances) directive. Leaded solder is largely not allowed any more, so tin-based solders are generally used. Older solders were a mixture of lead and tin, and the lead seems to drastically mitigate (though does not entirely eliminate) whiskers. Remove the Pb, as we've done to comply with RoHS, and whiskers can grow. Modern components are so small, and SMT parts have such a tiny lead pitch, that TW shorts are even more likely.

Whiskers are thin. They're generally between 0.5 and 10 um. So if enough current flows they may arc out, like a fuse, so the circuit might be self-healing. On the other hand, a lot of circuits operate with mere mA or uA, which won't be enough to zap the TW.

Worse, NASA showed examples of high-powered relays on AWACs planes where a tiny whisker arced and created a plasma, causing other TWs to arc, leading to catastrophic failures where hundreds of amps flowed before the relay's metal box burned through.

AWAC relay destroyed by TW

Whiskers can be long - over a centimeter. So they can bridge even between ICs.

The NASA people showed me a veritable horror show of TW. Like a 2 x 2 foot section of raised flooring from a server room with a zinc bottom. They figure that one panel has over 10 million whiskers growing. Normally a floor panel isn't much of a problem for electronics, except these get removed for service; maintenance workers unknowingly brush them off. Air currents can waft them into the electronics. Oddly, an adjacent panel was whisker-free.

Eleven space missions have had computer failures, and in some cases complete loss of the spacecraft, due to suspected TW problems.

The Space Shuttle program was almost canceled 4 years before its scheduled shut-down because of TW. PCB hold-downs on flight hardware were tin-plated. Enormous numbers of TW were growing. In a shake test (on the ground) a TW dislodged, creating a short and causing a critical control unit to fail. The Shuttle's program manager, beset with yet another potentially catastrophic problem, considered ending the entire program. Ultimately thousands of these hold-downs were replaced in all of the vehicles with tin-free versions.

Shuttle card guides with TW

The mechanism behind their formation isn't understood, but tin atoms migrate between microscopic crystalline grains (called grain boundary diffusion) over surprisingly long distances. These appear to aggregate on one grain, growing the whisker.

No one can predict when or if they will form. Adding at least 0.5% of lead to the tin greatly reduces their formation (while creating RoHS problems), but TW have been observed even on Pb/Sn solders. Conformal coatings help, but don't eliminate them.

Sometimes they are quite visible but often one needs an LED or fiber-optic light to illuminate a sample at different angles to see the whiskers.

What is the solution? There is none known at this time. This is what NASA recommends (from their web site):

NASA has a web site dedicated to whisker research. I thought The Conjuring was scary, but the site will really give engineers nightmares.

In Muse 348 I mused about a meme that is circulating that if one can fix bugs quickly, bugs don't matter. Muse 349 had replies from readers who disagreed. That sparked a lot of email from readers who disagreed with the disagreements. Here are some responses:

Paul Carpenter wrote:

James Thayer posited:

Like me, John Grant comes from the olden days of EPROMs:

I appreciate the feedback, both for and against my (uncompromising) take on quality code.

The always-interesting Jacob Beningo wrote an article in Design News where he speculates that the firmware developer of yore is going extinct. He contends that in the near future developers will be abstracted from the hardware to an unprecedented degree. No longer will we manipulate registers; instead we'll call APIs, rather like Windows programmers. He attributes this to the increasing complexity of, well, everything. We're using all sorts of complex comm protocols, graphics, and resources more akin to desktop development than low-level bit pushing.

I think the picture will be more mixed. It's good news that we're increasingly relying on third-party packages to handle these tasks. That's the whole idea behind reuse.

When I was a young engineer, 45 years ago, we wrote everything in assembly language. We wrote our own RTOSes and communications stacks (such as they were). Today we operate at a higher level of abstraction, and other than nostalgia, no one really wants to go back to those days.

I think Jacob is right in that more firmware people will be distant from the hardware. Managers will want this as deeply-embedded people are expensive, and for years some have told me they use a high-end CPU that will support Linux so their pool of engineers is wider... and cheaper.

But the hardware is a real thing. Our code will run on it. Worse, sometimes it won't run properly, and a deeply-embedded engineer will have to break out a logic analyzer, scope and other tools to probe nodes. To watch comm links. To monitor the PWM to figure out why that brushless DC motor stalls.

There will always be a huge class of products using smaller microcontrollers that can't run Linux or big stacks. Special problems, like building ultra-low power systems, will demand engineers that can write code and understand hardware nuances.

Digi-Key currently lists 72,000 distinct part numbers for MCUs. 64,000 of those have under 1 MB of program memory. 30,000 have 32 KB of program memory or less. The demand for small amounts of intelligent electronics, programmed at a low level, is overwhelming. I don't see that changing for a very long time, if ever.

Those 64,000 MCUs will be programmed by deeply-embedded people who can't conceive of being abstracted from the hardware.

I'm not sure when this came out, but ARM's DesignStart gives engineers access to the Cortex M0 and M3 IP. Want to design your own SoC? With DesignStart you can get started - for free. Plop your custom design into an FPGA. Or, commercialize a product paying no up-front fees, just royalties.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Note: These jokes are archived at www.ganssle.com/jokes.htm.

From Taylor Hillegeist:

Q: What is the fastest way to write spaghetti code?

A: Copy and Pasta.

Advertise in The Embedded Muse! Over 27,000 embedded developers get this twice-monthly publication. .

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.