Go here to sign up for The Embedded Muse.
TEM Logo The Embedded Muse
Issue Number 316, November 7, 2016
Copyright 2016 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com.

Editor's Notes

Did you know it IS possible to create accurate schedules? Or that most projects consume 50% of the development time in debug and test, and that it's not hard to slash that number drastically? Or that we know how to manage the quantitative relationship between complexity and bugs? Learn this and far more at my Better Firmware Faster class, presented at your facility. More info here.

Quotes and Thoughts

"Inspections are one of the most effective defect removal techniques. Testing is one of the least effective defect removal techniques. We know this from measurements; it isn't speculation. And yet, what do most organizations do? They test. What do most organizations not do? They don't inspect. Why is this?" Jeff De Luca

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Row hammer can cause unwanted bit flips in DDR3 memory. We've known about this for some time, but were assured that the problem would go away once we migrated to DDR4. Turns out, that might not be the case.

Chris Deckard had a calculator suggestion:

I just read Jon Titus' recommendation for an Android graphing app. Completely agree with his statement, "I'd use for a while and then forget how to use" regarding hardware calculators. I'd like to recommend www.desmos.com, It's an online graphing calculator loosely based on OS X's Grapher. It's quite intuitive, and allows for saving graphs if you are logged in. Being able to copy+paste data from Excel is great since I use it quite a bit for designing transfer functions before I implement them in code. Also supports sliders so you can interactively explore how a parameter affects a system.

JP wrote:

Thought I'd make a quick comment on the "tools" post that you included for one of your readers. Although this app seems cool, it would be better for high school aged students to get a good graphing calculator and learn to use it. Why, one might ask: they can take a calculator with them for act/sat testing (tablets are not allowed). It's also important to know that only certain calculators are allowed on the tests. The ACT seems to be more restrictive as it doesn't allow the TI89.


I just bought my son the "color" version of the TI-84 .. some of us were born too early to have graphing calculators in high school and were glad to have a state-of-the-art TI-30. :)

Grzegorz Mazurkiewicz suggests a plotting tool:

Take a look at this great open source software - Kst https://kst-plot.kde.org/. It allows you to visualize your data on 2D plots. It has also some analysis capabilities (histograms, power spectrum). It supports many file formats, but I think the most useful is of course CSV (or maybe the simplest to use).

A very interesting feature of this tool is that it can show the data in real time. It means that it updates the plots when the new data is available in the input file.

The simplest usage scenario for us, embedded developers, is to print the data using UART on a device side, transfer it to PC (FTDI or similar), save to CSV file and use it in Kst.

I did it when I was working on a project where accelerometers and other sensors were used. All the data from the sensors plus some variables were visualized using this tool. Very useful as a debug method.

Worth to mention is that this tool is really fast and handles lots of data without any lags. Tutorials: https://www.youtube.com/channel/UCuYlIZjLCWsDS6d3ghVZWrA

Freebies and Discounts

I'm a big proponent of the Ada language, so this month will give away a copy of Building Parallel, Embedded, and Real-Time Applications with Ada, by John McCormick, Frank Singhoff and Jerome Hugues. This is excellent introduction to the language though at $114 (on Amazon) is quite overpriced. I reviewed it here. (The GNAT Ada compiler is free and open source).

Enter via this link.

The Vote

(This is a story I wrote in 2004 that seems appropriate this week - Jack).

President-elect Bubba "the can man" Jones expressed surprise at his unexpected win in last night's quadrennial elections. With 100% of the precincts reporting in, Jones overwhelmed the opposition by acquiring an astonishing 65.536% of the vote. His opponents garnered a total of just 1.024%, with the rest strangely going to an as-yet unidentified candidate named "hckergrrl".

Seeming confused by the crush of reporters and well-wishers Mr. Jones continued to hover protectively over his shopping cart. Suspiciously eying members of his new Secret Service detail he was heard complaining about attempts to take his collection of aluminum cans and old clothes. "Nobody ain't gonna rip off my stuff," he muttered. "Dem cans worth 2, mebbee 3 bucks."

Hustled into the waiting limo he expressed delight at the prospect of sampling the car's mini-bar. Tonight the president-elect remains in seclusion in the Blair House, hastily-recruited aides telling reporters he was "sleeping off the effects of a trying campaign."

Though it remains unclear just how Mr. Jones won the presidential election despite his not having been on the ballot, this reporter has seldom seen an election run in such an efficient and forthright manner. Contrast last night's speedy decision with the rancorous climate of just four years ago, where slim margins and problems with paper ballots led to a court's decision that disenfranchised voters nationwide.

Modern technology has eliminated the age of hand-counted ballots. "Hanging chad" will be nothing more than a colorful phrase recalled in history books. Recounts now take seconds. Never again will the courts decide an election. Call 2004 "The Year of the Electronic Vote."

Thank the citizens of Florida for our electronic elections. Misplaced ballots, poorly-punched selections, and perhaps even outright corruption held the 2000 presidential race in suspense for weeks. Determined to avoid the scandals, the Sunshine State replaced thousands of antiquated manual machines with the latest of electronic vote counters. High tech touch screens instantly record each voter's decision, transmitting the results to a national database when the polls close. Recounts involve nothing more than a retransmission of the data, since we know the computers themselves are deterministic, immune from fraud, and cannot make mistakes.

Initial trials in the 2002 Florida primaries seemed less-than-promising. System crashes, locked up touch screens, and confusing instructions held the McBride/Reno contest in abeyance. Yet a winner did emerge, to tackle the incumbent in November's general election. Call 2002 a prototype of success, one that blazed a trail to this new and better form of e-Democracy that our children will inherit.

The Critics

Luddites suggest that we rely too much on software for critical systems, sometimes referring to the Navy's unfortunate Smart Ship program. Recall, though, that after the sixth fleet attacked Palau in Micronesia earlier this year, the review board showed the logic of that autonomous decision: the computer realized that Palau lacked the ability to shoot back. "This clearly shows the efficacy of a system designed to win wars," Admiral Decobocker noted, "the system created a war that it could not lose. The subsequent divide overflow that crashed the flagship's propulsion system was entirely unanticipated."

(Reports indicating the ship was overwhelmed by an unruly band of Caroline Islanders who turned it into a floating casino bar remain unconfirmed).

Others mentioned last month's recall of 600,000 Internet-ready automobiles when hackers found they could deflate the cars' tires, reprogram the engine controllers to emit constant backfires, and lock the windows up and doors closed with the heat on "high". Detroit reluctantly admitted to a buffer overflow problem but stated these were minor problems blown out of proportion by the press. Spokesman Graeme Kirchner once again urged passage of the National Anti-Hacking Act, saying "these bloody unsupervised latch-key kids run rampant through the computer networks. Parents should keep them bloody well away from the bloody home computer."

To get to the truth of software risks I interviewed Tom Thorten, lead software engineer of the SmartVote system employed so successfully in this week's elections.

Me: "Mr. Thorton, isn't this device what you folks call a safety-critical system?"

TT: "Nah, this isn't a safety-critical app. That's for avionics and nuke plants. No one's gonna die if this thing crashes. Hey, if there's a problem with this box ya just reset and carry on. Pretty much like any chunk a code."

Me: "How can you ensure the vote is tabulated correctly?"

TT: "Vote? Is that what this thing does? They just told me what to put on the LCD and how to log the data. The spec was pretty light, know what I mean?"

Me: "No."

TT: "Well, it's like any system. The boss is wrapped up with making promises to customers. Her boss is usually off testifying somewhere, and the Big Boss is still awaiting extradition from Barbados. It's pretty much up to me what this puppy does."

Me: "How was this tested?"

TT: "Oh, the usual, ya know? We pounded on it some, then Bob's kid (ya know the one? Cute little tyke?) played with it for a while. It was hell getting the ice cream outa the PCB. Then we stuck it down in Florida in 2002. Worked pretty good, what I hear. Few probs, but as soon as we get some more developers we'll work out those kinks. We figured most of those old folks wouldn't hit the screen too fast, but when they did, man that ole priority inversion thing hit bad, know what I mean?"

Me: "No. What happens to the data?"

TT: "Oh, we toss it into one of those embedded DBs, a cool relational thing. Sure wish it was reentrant; those corruption problems are killing us. Then at the end of the day we TCP it to Republican National Headquarters. Or was it the Dems? One of them, they keep changing the IP address on us. Little do they know - we stuck in a back door that lets us set a new IP at will. Saves a recompile every time they change their freakin' minds. Those dudes then dump it all into the National Database. But our box logs the data and ships it out. After that it's not my problem."

TT: "Well, I guess some folks might want to hack their way in, but it's pretty unlikely. We're running Embedded Win eXtraP, the most secure Win OS ever. It's so good it keeps downloading patches and fixes; hey, the system we ship evolves to something else over the course of a year. It's like, organic, man. No one really even knows what the code base looks like now. Pretty awesome, know what I mean?"

Me: "No. So no one can break in?"

TT: "Nah, probably not. Well, not at least if the users had a brain. I can't figure out why our customers never seem to set up a password like the read.me tells 'em to. Half them systems are wide-open! You know users, though. It's like my cable modem, you know? I see all sorts of open systems on the cable. My kid even ripped off some lawyer's Amex numbers from a Word doc on this dude's exposed C drive. Sure am proud of that little rug rat."

Me: "In Florida some people struggled with system lockups. What happens then?"

TT: "Oh, jeez, customers complain about every little thing. Just reset it, for Pete's sake, and don't get your panties in a wad."

Me: "But what happens to all of that data?"

TT: "Bit bucket, I guess."

Me: "Huh?"

TT: "/dev/null. You know."

Well, the arcane language of the computer professional was something we English majors didn't learn at Princeton. But I feel it's only proper to delegate the complexities of encryption, computers, and corporate accounting to those professionals who know these things best. Proper government oversight, as mandated in the Public Security and Trusted Computing Act of 2003, insures the nation's interest will be met. Mr. Thorten assured me that the Transportation Security Administration will start examining his code base "real soon now".

One Person, One Vote

President-elect Jones inherits a fractious constituency, divided in part by the nature of the very vote that gave him the nation's highest office. Perhaps the largest issue brought to the fore by various commentators in this post-election day is that of universal suffrage. Just what does "one person one vote" mean in a high tech age? Does your vote matter?

The 2000 presidential contest resulted in a razor-thin margin. Some contend that it makes no difference who had won that election, since the difference in votes was in the noise. Either candidate would have had the support of half of the electorate.

After the next mid-term elections a slew of contests won by the narrowest of margins raised questions only recently resolved by the Supreme Court. In a unique move the nine Justices deferred to the analysis of statisticians, who argued that uncertainty is part and parcel of every data gathering exercise. They pointed to the Gallop Polls which have long had an associated error band.

In Florida and other states an automatic recount is mandated when a contest results in only 0.5% difference or less between the candidates. The statisticians argued that such a small delta is meaningless, that no amount of recounting or runoff elections would express the will of the people with any more certainty. Justices bought this argument, resulting in the famous 3 sigma election rule: if the error does not exceed 3 sigma, why worry?

As the Chief Justice wrote in his majority opinion, "hey, this is the government. You just can't expect nine 9s."

After that landmark decision the liberal press was taken to task for promulgating the idea that voting was a quaint but pointless exercise. One wag suggested, for instance, that a Republican in Maryland (there are a few) shouldn't bother casting a vote in state-wide elections. The long and overwhelming history of Democratic successes there suggests that Democrats waste their time if they vote, and Republicans waste their spirit. To a first approximation the outcome seems preordained. The old saw "my vote negates yours" now reads "my vote counts as little as yours."

This reporter would argue that every vote counts, just as it did when the Founding Fathers brought forth this great nation more than 200 years ago. The Supreme Court's unfortunate decision did not factor in the power of technology to solve most ills. I see no reason why our scientists can't employ the same technology that so accurately guided the Mars Climate Orbiter to the surface of Mars, or that enabled Armstrong's triumphant Apollo 13 landing, to ensure every vote counts.

Don't be misled by critics who contend that, in the absence of paper records, an e-recount is nothing more than another database download. Does a ballot whose chad-clinginess is interpreted by an army of well-intentioned but exhausted observers better represent the will of the people?

The new technology of electronic voting insures every person is properly represented. Your touch-screen selection instantly tips the balance in favor of your candidate. The margin for error, due to the supreme number-crunching power of the computer, is surely zero. I look forward to the day I can vote from home using the power of the Internet and the security inherent in the most popular operating systems.

The Rest of the Story

In other news, this reporter was heartened to see the Dow Jones sharply rebound when Red Hat (the latest addition to the esteemed ranks of the 30 industrials) released their Open Vote package. At 4096, up from yesterday's close of 2048, the Dow promises release from this long-lived recession.

More Tips for Aspiring Developers

In Muse 315 readers sent in tips for aspiring developers. Since then a number of readers have sent more.

Andy Kunz wrote:

On the subject of advice for young engineers:


I work with a team making consumer products (electronics for radio control airplanes).  The most productive engineers are those who take what they're making at work home with them and try to use it as a customer would.  That means following the manual (and marking it up for the Manuals team), using it in the expected ways, using it in unexpected ways, and looking for ways to break it.  Passion brings excellence.


Since model airplanes are also my hobby for nearly 50 years, I use for fun what we make in a daily way.  If I won't use one of our products in my own planes, how can I expect my customers to?  Hanging on my shop wall is one of my airplanes that crashed because of a fault in my code.  There is only one of them, and hopefully it stays that way.  Customers never had that code in their radios, by the way.

Scott Winder has some great points:

  1. Read Study other peoples' code. You learn how to code in school, but not necessarily how to code efficiently. When cycles count, the difference between "/4" and ">>2" is gargantuan. Not every seasoned engineer is going to be writing code worth reading, but a lot of them are. (Hint: look for the code that has comments explaining a shortcut when it's not immediately obvious where the shortcut leads. Unless you're solving a debugging puzzle, your time is best spent learning tricks where there's a built-in manual.)
  2. "Don't be clever" is bad advice on its own, but the spirit of the message is sound: avoid cleverness for cleverness' sake. On my whiteboard is a (possibly apocryphal) quote by Brian Kernighan: Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.
  3. Think outside the box. Before you discount this as a tired cliché, let me explain: consider your gazintas and gazoutas (system inputs and outputs) when root-causing an issue. One of the most valuable lessons I learned as a newly minted engineer came when I was working with a colleague on an airborne system that wouldn't POST. The system was essentially a PC, and we knew that any number of things could keep the system from coming up, so we replaced memory, the video card, and even the power supply. At some point, one of us said "it's almost like it's not even getting power." So we replaced the power cord, and then self-administered forehead smacks. The vibrations in the airborne environment had broken a connection in the cable. We were so focused on finding the problem inside the system that we hadn't considered that the problem might be outside. This applies just as well to unexpected inputs, RF interference, ground loops, etc.: if your system interacts in any way with the outside world (all of them do), then the outside world should be considered as a possible source of problems.

Ray Keefe thinks:

Jeff Beck was once asked how he became such a good guitarist. His answer was that he was in a series of bands where he was never the best guitarist. I think the idea of getting yourself into a position where there are better people around you who you can learn from, makes a of of sense. So I agree with Dave Telling on career development. And when you are that better person, be prepared to tech others and help them be even better than you.  The world needs better code than it has now. And that requires us all to be better designers and coders.

At our office I have a cup with the slogan on it "While awake { write code();}". It is humour. Because the actual writing is the low hanging fruit. It is the design that makes all the difference. Yes, the code also needs to be written well. But a poor design written well is still usually a bad outcome. 

Caron Williams contributed:

A lot of sage advice, but I was somewhat surprised that no-one suggested reading Fred Brooks' Mythical Man Month. Young software developers - if they have heard of it at all - loudly put it down as being old, and outdated. These same developers are, at the same time, making at least 50% of the errors that it describes so well. They are then surprised to find their project is over budget, seriously time slipped, and not meeting requirements. And then they invoke another MMM problem by adding more people in an attempt to get it back on track.

As a contract design engineer often brought in to fix things, I have seen more clients' projects fail - and continue to see current projects fail - because of the 'Second System Effect' than from any other cause. If your readers are unfamiliar with this, they should get a copy of the book and read it. It's not big; it is well written, surprisingly non-technical, and an easy read.

Harold Kraus has some outside-the-box suggestions:

I note that while the heading is "Tips for Aspiring Developers" ,  what the person asked was "What are the disciplines of becoming a great embedded programmer?"

IMO, a programmer is not necessarily a developer.

If you focus on algorithms as the fount of greatness, I think you could be widely misled.  Now, I am talking here specifically about the classic instructional algorithms; the series of sorting functions where the competition was to be the fastest and smallest. Such algorithms serve as exercise to teach the language, but offer no instruction into architecture or process.

Numerical Methods (programming is just a field of mathematics?)
That said, an interesting aspect to DO-178 (that has been depreciated in ISO 26262?) is the specific concern with algorithms (finite set of well-defined rules that give a sequence of operations for performing a specific task). Now, I am talking here specifically about more complex, numerical algorithms (e.g., root finding, Phong shading).  The FAA's concern with algorithms it that the developers have the mathematical competence to avoid safety pitfalls of "algorithms", knowing with proven certainty that the accuracy of the selected algorithm is sufficient and correct for the application and that the known failure modes of the selected algorithm have been full contained.

A key point to understand here: the FAA is not only concerned with whether you have coded the selected algorithm correctly and error free (this is to be a given for all of your code), but that you have selected, correctly designed, and validated the correct configuration of the correct algorithm.  

Language Arts 
Recently, I've experimented with referring to myself as an English engineer, rather than as a software engineer, since 80 percent of what I do is technical communication (coding is the easier 20 percent).  Any embedded project is a social project and you must communicate effectively to be a part of it.

Linear Systems/Control Systems
A significant application of embedded systems is control; even in application-oriented embedded systems, there are still control functions.  Major control design and code generation tools (Matlab/Simulink, SCADE, LabVIEW, and even some minor ones) have, at the core, the transform block/channel architecture.  My observation is that people taught conventional programming classes are ill-prepared for the concept of concurrent, synchronous blocks.

Athletics / Anatomy and Physiology
I can't remember where I heard this, but the answer to "what is the best exercise" is "whatever you like doing (enough to do it)".  Anything that get you out and doing things, you should do. What is important is that your brain is interacting with as much of your senses, muscles, and gut (the intestinal nervous system is almost as massive as the brain) as fits your interests.  Like muscles, bones, mitochondria, etc; any nerve that is unused is eventually dissolved.  Any nerve lost in the brain results in a reduction of robustness, at best.  In adult life, brain mass is preserved only by rigorous physical exercise.

Moreover, experience the mental exercise of interpreting your body as an embedded system; sensors, signal filtering and processing, actuators, power management, thermal management, maintenance.  The very concept of feedback came from study of the cholesterol homeostasis system. Look as the vision system; more than 5 stages of feedback-controlled attenuation functions, some electrical, some chemical, some mechanical.

Appreciate the social developmental value of athletics, even if yours is not a team sport -- if the activity is competitive, you are at least socially interacting with the opponent.

The most painful thing I deal with as a youth coach is knowing that the kids that struggle and drop out of athletics are the ones that need it the most.  No, kids (and adults) don't need to be champions (that is unrealistic anyway); they need to be out, active, challenged, and growing.

This Week's Cool Product

Maxim max17201 block diagram

Battery-operated devices are becoming more and more important, but it's notoriously difficult to accurately measure a cell's state of charge. Generally voltage alone is not a good indicator of reserve capacity as the discharge curve of most cells isn't linear. Maxim's new 172xx ICs are fuel gauges that promise to increase battery capacity accuracy.

Traditionally we've measured capacity by coulomb counting. Integrate current while charging and subtract the amount used in the discharge cycle. It's easy to do but provides only a rough idea of remaining capacity, and suffers from drift. Small accumulated errors compound. Battery capacity degrades with age. Maxim combines coulomb counting with a very complex proprietary algorithm. The parts reportedly achieve better than 3% accuracy 94% of the time, and better than 10% 99.5% of the time.

The devices sport a lot of features, including prediction of battery end-of-life, computed by accumulating capacity loss over each charge/discharge cycle. It supports multiple cells with cell balancing. Interestingly, instead of a sense resistor one can use a PCB track to form 0.0035 ohms of resistance.

It's designed to annoy end users as an encryption engine can be enabled to prevent battery cloning, sort of like how some printers won't take third-party ink cartridges.

On the down side, the part seems to be meant for batteries with lithium chemistries only - the data sheet is somewhat confusing about this. And with 40 uA of active current it's not for ultra-low power systems. The vast array of features comes at a price of a dizzying array of control registers.

According to Digi-Key these are $1.59 in quantities of 2500.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

Note: These jokes are archived at www.ganssle.com/jokes.htm.

When your hammer is C++, everything begins to look like a thumb.

Advertise With Us

Advertise in The Embedded Muse! Over 27,000 embedded developers get this twice-monthly publication. .

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.