Go here to sign up for The Embedded Muse.
TEM Logo The Embedded Muse
Issue Number 347, April 1, 2018
Copyright 2018 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

Ad for HCC-Embedded

David Sudolcan won last month's contest for a copy of Jean Labrosse's uC/OS-III.

Steve Leibson published a short(ish) but fascinating history of computing, with some thoughts on where we're headed as Moore's Law becomes, well, history.

A lot of prospective entrepreneurs contact me for advice. I always recommend they read the first half of The E-Myth by Michael Gerber (the second half is pretty dreadful). Gerber's contention, which I've observed countless times, is that too many owners of small businesses are completely focused on working in the business - that is, doing the consulting, making the products, and generally engaging in the main business activity. Instead, one should (at least some of the time) focus on working on this business. That might include finding ways to increase productivity. Reduce taxes owed. Deliver faster. But unless one is actively making the business better, decay, entropy and the competition will prevail.

This is what my Better Firmware Faster seminar is about. Take a day with your team to learn to be more efficient and deliver measurably-better code. It's fast-paced, fun, and uniquely covers the issues faced by embedded developers. Information here shows how your team can benefit by having this seminar presented at your facility.

Quotes and Thoughts

"1. Good enough is good enough. 2. Good enough always beats perfect. 3. The really hard part is determining what is good enough." - Ravi Sandhu

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Paul Carpenter has written an article with tips on stack usage.

Salary Survey

In February I ran a salary survey. There were 1142 useful responses (i.e., properly-filled out form, non-hobbyists, etc.). Here's a summary of the results. This is not a scientific survey! No tests were performed to ensure the accuracy of the data, and one would be wise to understand that participants may have confused local currency with US dollars, or to make other mistakes. Further, I've combined salary and benefits into one number, which may be confusing due to local laws. For instance, in the USA we, or our employer, pay for health insurance, while in some other countries that is a State benefit. And, oddly, the average listed value for benefits is $413, which seems so low as to be unlikely.

Still, the data is interesting and paints at least a broad picture.

Salary Survey geographical distribution

We're getting older, and this trend has been significant since I started doing this survey in 2006. Here's this year's data and that for 2014:

embedded salary survey ages

In the USA, respondents reported the following figures for combined salary and benefits. Note the, in many cases, fairly significant increase over 2014:

embedded salary survey USA compensation

Regionally, a few interesting averages:

USA Average $115,454
CA $140,541
CO $116,245
FL $105,571
IL $119,298
MA $121,942
MI $112,018
PA $103,074
TX $124,701
WA $127,453

In Canada, the figures look like this:

embedded salary survey Canada compensation

In Western Europe:

embedded salary survey Western Europe compensation


Australia and New Zealand:

embedded salary survey Australia/NZ compensation

embedded salary survey India compensation

Paid vacation days per year:

embedded salary survey vacation days

Hours worked per week:

embedded salary survey hours worked per week

9.6% of respondents work as consultants. Their hourly rates are all over the map, so this chart shows the average and the standard deviation:

embedded salary survey consulting rates

Here's the distribution of consultants' rates in the USA:

embedded salary survey consultants rates in USA

6.6% of respondents feel business is about the same as last year. 2.9% feel it's better. An astonishing 82% thinks business is "much worse than a year ago" and an additional 0.08% feel it's "worse than a year ago." Recessions seem to follow a ten-year cycle; could this be an early indicator of trouble?

Yet 29% feel the demand for engineers is about the same as last year; only 5.6% feel it will diminish. 54% expect a strong demand for engineers while only 3.5% expect their work to be offshored.

223 people left comments. A few highlights are:

  • Ageism is starting to catch up with me - opportunities seem to evaporate despite exact qualification.

  • At my job we are developing embedded software using C++11 and there are exceedingly few resources on how to develop embedded systems using modern C++. We try to stick to industry best practices, but a lot of the time there is a lack of standards (or even discussion) regarding parts of C++ that are now 7+ years old. I would love to see more discussion and use of modern C++ in the embedded community.

  • Canada still has a huge brain drain problem across the tech sector, embedded is no different. If a new grad can almost double their salary by heading down to the US on a NAFTA visa - well, why wouldn't they? There is also a far better VC ecosystem in the US for tech. Until those circumstances change, I don't really see growth potential for capital-intensive startups like hardware and embedded in Canada.

  • Employer has done three layoff in previous ten years; where each reduction ends with more responsibilities and less pay. And no one in San Diego County is hiring permanent employees that are over age 55. Should have quit and migrated to the east coast during early 00s to escape this 'cult of youth'. Perfect 20/20 hind-sight...

  • Engineering these days seems to consist of working out the right things to search for and then knowing enough to tell the rubbish answers from the good ones.

  • Everyone wants connected Cloud/IoT.

  • Everything Changing So Fast Fast, Super Competition At Home And From Outside, Cheap Products From China.

  • Have made significant salary sacrifice to be able to work more from home office. It's worth every penny and employers should offer it more. Also gave up any kind of managerial responsibility. Just because you can do something well doesn't mean you should, when you don't enjoy it. Does not mean I don't mentor junior engineers. That's just something you owe your younger self.

  • I generally enjoy my job, but get frustrated trying to find resources to learn more about software and hardware design philosophies. I wish that I had more exposure to other microcontroller suppliers beyond the one we use at work to possibly become a consultant some day.

  • I have been trying to fill an embedded engineer job at my company for the last 3 months. I think there is a real shortage, based on the lack of good candidates and higher salary requirements that some candidates wanted, but we could not justify. Good news, we interviewed a person yesterday who hopefully we can get to join our team.

  • I have cut down on staying up to date with technology. My management is not interested in new technology in our products.

  • I'm lucky to be working for a good company. ,

  • In the UK I expect the Brexit schizophrenia* to escalate, with employers screaming that they can't recruit foreign talent, yet completely ignoring the talent on their doorstep within the country. *Schizophrenia: abnormal social behavior and failure to understand reality, including false beliefs, unclear or confused thinking. Pretty well describes the UK employment/Brexit situation.

  • Love what I do, but the work/life balance has really gone to hell, over the last fifteen years: business expectations have become increasingly unrealistic.

  • More and More Hardware Design Engineer jobs available here in Belgium which seem not to get filled in easily. Seems the market is in search for more Electronic Hardware design engineers here in Belgium.

  • My knowledge doesn't matter, only my age. I'm being pushed out, but not kicked out yet because my employer has more work than personnel. I'll be let go as soon as the staffing matches the work.

  • Other countries teach computer science and technology much better than the USA. I have 2 bright sons who absolutely think what I do is boring. They have no interest in software or technical details of hardware. I do mostly embedded software, but not hardware design although I interface with plenty of hardware. Most of the new kids out of school know little about how a computer actually works. Only the high level interfaces and languages.

  • Still see strong demand for embedded software & hardware in the US. IoT hype and positive market conditions have certainly helped.

  • The field is evolving too fast. I once had time to keep abreast of trends and changes. To experiment. Now I am focused on completing projects with two to three week turn around times (or at least that is how much time I get as I hopscotch from one job to the next). Heaven help me if I get something wrong. Some customers are not forgiving of missed dead lines.

  • There is NO quality left in the world!!
April's News

Dateline: April 1, 2018.

Some wild news items have come across my desk recently. I was surprised to learn of a C# 8051 compiler. Another press release explained why all those servers smoked when Slashdot slashdotted itself. Then Google's mesh of servers became self-aware and returned the single word "MORON" in 72 point type to all searches on anything Kardashian related.

Now it seems Linus has ported Internet Explorer and Outlook to Linux. Why? Well, he figures that the only way to combat Microsoft's hegemony in operating systems is to invite the same sorts of virus attacks that make Windows so popular. Once hackers see Linux as a viable target, the same brain-dead PC users who don't upgrade their virus definitions and who never use a firewall will be drawn to the OS. Then they'll see how "ls -la" is so much more intuitive than Windows Explorer. Apparently spam messages (URGENT AND CONFIDENTIAL) directed to Linux machines already advise users to immediately issue "rm -r" to get the $27 million that's sequestered in some bank in Nigeria.

Here are a few more oddities you may find interesting.

Agile Methods

The agile community has come up with a new methodology that builds upon the ideas and successes of SCRUM, TDD, FDD, and XP. EXtreme saboTage dOoms softwaRe pronTo (EXTORT) defines four practices a healthy software organization can use to improve their software processes.

Tom Stochastic's book "EXtreme saboTage dOoms softwaRe pronTo Explained: Embrace Chaos" expands XP's philosophy of provoking change early to provoking failure as soon as possible. From the book: "Software costs are spiraling out of control. Every year projects get bigger and later, yet studies show few large projects are ever delivered. By provoking failure early companies can save billions which would have been squandered on gigantic IT projects that are inevitably doomed to fail."

The author dissects Chrysler's famed C3 project, which has often been touted as a poster child for eXtreme Programming. After the original project became a complete debacle, Kent Beck was called in to use his XP practices to save the program. Four years later, again over budget and behind schedule, C3 was canceled.

By employing EXTORT on Ford's similar payroll system Mr. Stochastic was able to achieve complete program failure in only two years.

Instead of XP's "hard to remember" 12 practices, EXTORT needs only four. The first is a 15 hour work week. Tired people make mistakes, and the brutal 40 hours demanded by XP advocates tends to intrude on web surfing, laundry and long lunches. As Mr. Stochastic writes, "It's exhausting to even think about programming."

EXTORT requires team- instead of pair-programming. A single developer sits at a computer, whose output is projected on a 2500 square foot monitor in a stadium. 40,000 other developers in the stands heckle and kibitz. "Sure, it doesn't sound all that productive," says Mr. Stochastic, "but if two heads are better than one, think about the power of a small city all working together on a 50 line function. And stadiums already have the infrastructure to provide nutritious programming food, like hot dogs and beer."

Shops using EXTORT unfactor relentlessly. Code that can be obfuscated must be obfuscated. Obviously, software that clearly communicates its intent invites change. "Developers must be very afraid of the code," Mr. Stochastic writes, "otherwise they're likely to bow to pressure for bug fixes and enhancements."

(An unfactoring tool is available that removes white space, adds random carriage returns, renames all variables to Ixxxx, where xxxx are sequential numbers starting at 0001, and replaces comments with quotes from the Kama Sutra.)

Finally, EXTORT practitioners never construct a real test, which might break the code and lead to 20 hour work weeks. Limited unit tests are fine as long as none check extreme values or other, hard to debug conditions. Integration testing is required, but only by the customer. Users expect bugs; why disappoint them?

Caffeine Deprecated

Wet Planet Beverages recalled 14 million cases of Jolt Cola today after researchers discovered the drink leads to bug-infested code and violent workplace behavior. Long a favorite for late night programming stints, the caffeine-rich drink is a staple for 20-something developers working under crushing schedules.

Dr. James Wired explained that stimulants such as caffeine keep one awake; awake one can write programs. And all bugs stem from code. As he put it "no code, no bugs."

Dr. Wired also found a strong link (correlation coefficient of 0.999) between crazed sleep-deprived developers chasing masses of bugs with the deadline hours away, and crime. "You ever see a meditating Buddhist monk go postal?" he asked. "These uber-caffeinated drinks are like PCP. Just last night I saw a guy with eyes the size of the fabled mega-squid bashing his boss's Beamer with a Dvorak keyboard."

Wet Planet plans to release a new drink for programmers called "Slumber." Said to contain 20 mg of Phenobarbital, the product promises an immediate decrease in software defect rates. "Three or four cans of this stuff and you'll have the highest quality code in town," a PR hack promised.

Starbucks countered with "Cafe ~Expresso," a decaf blend with three times the amount of Phenobarbital in Slumber. To encourage developers to continue to use their stores as meeting and work places all hotspots will be downgraded to dial-up.

Pillows will be provided.

GoTo News

Edsger Dijkstra has recanted his 1968 paper "Go To Considered Harmful." (Communications of the ACM, Vol. 11, No. 3, March 1968, pp. 147-148). That work was considered seminal at the time and spawned a wave of structured languages like C and Ada.

Recently, though, the National Enquirer quoted Dijkstra as saying "at the time most software was infested with gotos. Hell, look at Dartmouth Basic. The tiniest bit of code requires dozens of gotos. And FORTRAN! Did you ever see such convoluted mess? Yeah, they eventually tidied that language up, but look what's happened since.

"C, C++, Python, they're all the same. I need a bleedin' 62 inch HDTV monitor to see the code pushed a yard to the right by all those indentations! Remember when 'white space' was used to illustrate structure? Well now all we've got is white space. The code in a deeply-nested construct starts on column 1823! And it reads "while while while while while' like some perseverating autistic kid. You see 'ifs' nested so deep I need a bloody periscope to find the closing braces.

"And they call this stuff structured?

"I dunno what I was smoking when I wrote that paper. It was the 60s, after all. Somehow I figured the love and peace thing would translate into beautiful snippets of structure, bits of poetry crafted by long-haired programmers. Ha! Now we're all part of a machine, adding patch after patch to a rickety structure to get yet another product out the door.

"So now I'm thinking FORTRAN got it right in 1957.

"My new approach is called Outrageously Obfuscated Programming Slang, or OOPS. Structured programming often disguises code which is in fact a convoluted muddle. Looks good at first glance but when you dive into the stuff you realize the developer had tied himself up into a Gordian knot of do-while loops. With OOPS we embrace gotos; our special editor draws lines between each goto and its target. The spaghetti mess fairly screams at you off the page, making it easy to figure out what to refactor."

I visited Professor Dijkstra's lab for more background information on this story, but his assistant told me "goto" someplace as hot as a marketing guy when told there's been yet another schedule slip.

SCO Wins

The long-running feud between SCO and, well, the entire world, ended last week when the Supreme Court forced IBM to publish all of their source code to every product they've ever released. After digging through hundreds of millions of lines of Autocoder, Cobol, Fortran, APL, C and more the court found a single comma they ruled was stolen from SCO Unix.

In their 682 page opinion, which used the "⇑" character instead of commas, the Justices wrote: "This brazen and egregious theft of a much smaller company's intellectual property is intolerable. Some might complain that a single character does not deserve copyright protection⇑ but in consultation with our panel of illustrious computer experts we found that the comma is composed of 8 whole bits⇑ each of which was crafted and invented by engineers at SCO. IBM's wholesale pilferage extended to all of their products released over the course of generations."

SCO's chief executive Darl McBride released a statement saying: "Now that we've proven the Linux community is using our proprietary software it's our intent to go after every other company and individual infringing on our copyrights. Illegal comma users beware: pay up or face litigation!"

The Justices' opinion concluded with the statement: "We grepped the Constitution and found 585 commas therein. The United States will therefore pay SCO a yearly licensing fee of 585 billion dollars⇑ with additional reparations backdated to 1789."

Reportedly Microsoft has offered their own comma character ("$") for a license fee of "merely half what those pirates at SCO are charging. And ours comes with an automatic update service."

In related news the legal firm Botts, Butts, Botts & Butts, LLP, SP, LLC, Inc, GMbH, AB, Ltd, recently was awarded a software patent for the loop. 29 claims cover all conceivable forms, including the infinite loop. A press release on their web site advises all companies providing, using, modifying or maintaining any form of software to scour the code for loops, and to remove those immediately. The release goes on to say "in those very few percent of cases where it's either impossible or too costly to excise all loops, Botts, Butts, Botts & Butts, LLP, SP, LLC, Inc, GMbH, AB, Ltd offers a reasonable licensing plan enforced with just one dongle per loop used."

And that's the news for this April 1, 2018.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

Note: These jokes are archived at www.ganssle.com/jokes.htm.

Featured prominently on one vendor's booth at Embedded World, this sign pretty much sums up many of our experiences with customer support:

Advertise With Us

Advertise in The Embedded Muse! Over 27,000 embedded developers get this twice-monthly publication. .

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.