Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 386, November 18, 2019
Copyright 2019 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

Express Logic

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Jack's latest blog: How Projects Get Out of Control

Martin Keppler sent a link to a video about rebuilding an Apollo Guidance Computer.

Quotes and Thoughts

Techniques proven and routine in other engineering disciplines are considered radical innovations in software engineering. Fred Brooks

Tools and Tips

SEGGER Embedded Studio The leading cross platform IDE

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

In the last issue I referenced an interesting article about bi-di communications over an LED by reverse biasing the LED. Siva Govindarajan wrote:

On reverse biasing LED subject -  standard LEDs (GaN and GaAs types) are not designed for reverse biasing and if we give continuous reverse bias, in a long run with high humidity conditions, silver electro migration in LED die will happen causing LED to fail. In the automotive world, LEDs will never be subjected to reverse bias and there are lot of lessons learned in auto industry on Ag migration. For small home projects, reverse biasing technique might be OK but for high volume and nasty environment conditions it is not advisable.

Freebies and Discounts

This month's giveaway is a 30 V 10 A lab power supply:

Free lab power supply

Enter via this link.

On Honesty in Scheduling

In the last Muse I mused about honesty in scheduling, expecting to get a lot of negative feedback along the lines of "you're just an idealist." Mostly that didn't happen. Many people complained that schedules never seems to consider unexpected interrupts. Did you know that most people are engaged in their main project only 55% of their work hours (a study in the sadly-deceased Crosstalk magazine [2014] pegs that at 50%)? Talk to a grizzled old embedded developer about how he creates a schedule and you're likely to hear about doubling the estimate, which correlates closely with the 55/50% number.

Oh, and a schedule is an "estimate". It's never an "exact-imate". That implies a probability distribution function. Only a fool believes a distribution like this:

Here are some of the replies.

Lars Pötter wrote:

I witnessed colleagues doing that. It worked for them and it is hard to argue against something that works. It was in a situation where the relationship between them and the manager that demanded the schedule was dysfunctional. I still don't like this because it is a cover up of the real problem to avoid escalating the issue.

When getting asked for a planing I often asked the manager to what level of detail he wanted in the planning. And they always requested less detail than I was willing to give. But they always wanted it fast, sometimes directly on the spot in a meeting.

Engineers often don't think like a manager in such situations. I have learned to first think about what the manager wants to achieve.
Sometimes it is just about the decision to choose option A or Plan B. In this situation just stating the effort relation between the two ("A is much more effort than B") is all the manager needs.

Other times the manager thinks about a deadline. So the real question is probably can we finish this before the Event that is coming up. Here again answering that question will be better than detail planning of each step for all possible variations. ("we might have a demo for A in 3 weeks, but nothing to show if we go with B")

A detailed plan is a way to communicate the work packages in the process and the estimated size of the packages. Managers might only think with half a brain and only inside their tiny box, but they are not dumb. Most managers know how to weight the estimates coming from their engineers.
Therefore the inflation that the engineer puts in might just as well be cut out by the manager knowing that this engineer always inflates that much.

Then if a manager cuts the estimate and the engineer then fails to achieve the new estimate isn't that a "told you so" moment. Whereas if the engineer is able to achieve the new deadline, then the manager was right to cut the estimate, right?

In the end everybody needs to be truthful to oneself. Estimates can be wrong. We are doing stuff we never did before (otherwise we would reuse the code) therefore we never know for sure how much effort there will be. And in the end it is still the job of the manager to motivate his engineers to work harder,...

Peter Heath had an analogy:

I once met a mechanic who did everything he could not to repair my car. By that I mean he would do no repair before its time. Period.

One day after having become a bit exasperated by his reluctance to replace a fuel filter with a petrified connection that he would not fix until the filter started to cause performance issues, I asked him about this philosophy. His answer was simple and because of that it stuck with me ever since.

If I tell you something needs to be done when it doesn't, I'm lying to you. Then, I have to remember my lies. If I just tell the truth, I don't have to think about it anymore.

Needless to say, he remained my mechanic until I moved 2500mi away and could no longer reasonably have my vehicle serviced at his location any more. As I get older and my memory begins to, well, get older, his wisdom becomes all the more apparent.

BTW, I drove with that fuel filter for another 3 years and it never did give me any problems.

Harley Burton has had his battles:

I really enjoyed your article on honesty in scheduling.  I fought that as an engineer and engineering manager my entire career.  My most significant experienced with that was when DOD-STD-2167a had just been released.  I was group leader on a project for the Air Force to develop an automated test machine.  Our company won the contract and we began to develop the system.  My engineers and I developed a very complete work breakdown, task assignments and fully develop the schedule.  Our final analysis was nearly 1200 pages, totaling approximately $10 million, a staff varying from 8 to 32 people and a total just less than 4 years.  The Air Force did the same analysis independently with nearly the same results.  They arrived at nearly the same results.

When I presented that to my management, they immediately said that we would develop it in 1 year.  Since this was a cost plus contract, they did not question the dollar amount, although they cautioned me that the program would be audited and therefore we should be very careful to justify every expense.  Additionally, I was advised that we would not be hiring that many people.  They naturally assumed that I had padded the project to make sure I would get it done.  This practice was common for many military contractors in those days.  Schedules, costs and manpower was padded a little so management could cut it to some pre-defined number that was somehow in their mind.

We built the system and delivered it to the Air Force in approximately 40 months, just less than 4 years.  My people hit the target cost within $10K.  I staffed with full-time engineers in my group, engineers from other departments as needed and as many as 20 contract software developers.  The product met 100% of the design goals.  Our documentation, developed in full compliance with DOD-STD-2167a.  The software integration, testing and verification was the easiest I ever encountered in over 40 professional career years.

The up-front work was very difficult and the learning curve was tough, but since it was a requirement for the contract, in retrospect, I wish I had been able to force the issue on all future developments.  I keep telling myself I should do this in my personal (hobby) work as well since I am still actively developing products for my own benefit.  However, I don't do it.  I'm probably too lazy, even though I know better.

I just wanted to affirm what you were discussing.  It's been going on for years and if we all would learn to be more honest about the development and structured about our development process and our managers would learn to trust their developers, I think we would all do a better job with less failures, lower cost and fewer ulcers.

Mike Davis also opts for honesty:

Thanks for the good reminder in "Honesty in Scheduling."  I whole-heartedly agree with you.  I'll even go a bit further and assert that Engineers that lie or guess schedules are part of the problem.  As a consequence, not only to be ethical but part of the solution, we *must* capture each estimate and how we computed it.  That way we can catch invalid assumptions soonest and help our managers do their jobs, and we can learn to do better estimates. 

Bill Smithem said:

It's been a couple of decades since I've done any embedded systems development (other than for my own curiosity and personal use), but I've never been blessed with upper level management that dealt with the truth about schedules. My immediate managers, sure, but they were engineers, and they always lied to upper management.

As team lead at the last embedded design job I had, I was required to attend the weekly status meetings where we dealt with the after effects of management's decisions from the previous status meeting. At one point, I pointed out that this was the third or fourth week in a row they'd decided to deal with an emergency by putting the same engineer on the problem for the next three or four months, and they'd committed about 700% of his time already.

The problem was fixed by informing me that my presence was no longer required at the weekly status meetings.

Not so surprisingly, this company no longer exists.

Herb Lacey has had trouble with managers:

'Honesty in Scheduling' is great, reinforces the obligation engineers have to be honest with their bosses, and should be oft-repeated and always followed, however in my experience of twenty-five years as a developer, every single manager's response to an honest schedule breakdown with detailed task analysis, was to threaten the developers. Most managers lied and said they would at least advocate for additional resources (time) if the engineers provided a detailed task analysis, however when such an analysis was presented, they would say the analysis wasn't persuasive, and then, we would be further in the hole due to the time spent doing the task analysis. If someone out there has an actual management structure that responds ethically to developer schedule concerns, they should value that management very high (scarcity).

Bob Dunlop writes from a different perspective:

My experience as a hospital nurse is that there is something of a similar self-defeating work ethic:  Working off the clock.  It's not unusual at all for there to be more work to do than can be accomplished in the shift. Sadly, many RNs exacerbate the situation by clocking-out and then continuing to chart, or worse, do patient related tasks. Why is this a "bad" practice?  First, there is no real record of how many person-hours of care were actually required to accomplish the requisite tasks, so management gets to continue with the fantasy that staffing and tasks have been appropriately allocated.  Second, this work is not compensated.  Third, here in California, the umbrella liability/malpractice coverage for the employee disappears the moment they clock-out. There have been cases of malpractice charged against RNs who worked off the clock.

As with engineering (I did software for some 15+ years before entering nursing), deluding ones self, or management about the actual effort and time required is self-destructive. Further, in healthcare, there can be some deadly or financially devastating outcomes.  I always told my students, as well as coworkers: Don't Do That!

Survey Results

VDC did a survey of the embedded space; the results came out a month ago. VDC has allowed me to quote some of the interesting statistics. As always, take the data with a grain (or a tub) of salt. In my opinion, many of the questions asked are ones engineers really don't have the data to answer. I think surveys are like impressionistic paintings; they give an interesting though vague sense of reality.

Schedules are longer than one might think. The mean is 41 months, median 12.

Mean source lines of code is 371,063, with a median of 12,500. As is typical of these surveys, the results are quoted to an absurd level of precisions (371,063?) When I ask teams how much source they work with the answer is generally accurate to +/- 50% or so. And some people will include only code written in-house, while others might fold in acquired components (Linux?).

Why are projects late? Here's the survey's results:

Processors use in embedded projects

Total cost of engineering for the current project: Mean is $5.5 million, median $250K. I'd sure love to see the distribution on this one given the huge disparity between mean and median!

Where do those dollars go?

Engineering time

Processors used in current project:

Embedded processors used

To me, the most interesting results were about shipped bugs. Respondents report a (mean/median) 79/5 patches in the first year of deployment of the most current project. Cost to fix in people-hours is (mean/median) 136/24 per patch. If an engineer's loaded cost is $150k/year, that's about $2000 to $10,000 per fix. With up to 79 shipped bugs/year we're looking at hundreds of thousands to almost a megabuck in support costs due to quality problems! And that's only in the first year.

The survey is gigantic, having 6200 lines and 335 exhibits in the summary spreadsheet.

This Week's Cool Product

It's possible to measure temperature using nothing more than a diode as the sensor, as these devices have a temperature coefficient of about -2 mV/°C. But if accuracy is important, other options make more sense.

Maxim's new MAX30208 temperature sensor is accurate to 0.15% over 0 to 70°. Digi-Key quotes $2.23 in singles and half that in quantity.

Interestingly, the datasheet specs the power supply rejection ratio as 0.006°C/V, instead of the more common dB. I like the °C/V rating as it's so easy to interpret.

Max operating current is 100 µA, and only 1 µA in standby.

With a 2x2 mm package self-heating could be a problem, but somehow they've conquered this:

MAX30208 self-heating

It has an I2C interface, and includes two GPIO pins.

The part is advertised for wearable devices, but I can see a lot of use for it in all sorts of sensor applications. While the IC is available today, the eval board has a six-week lead time.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

These jokes are archived here.

The C Programming Language -- A language which combines the flexibility of assembly language with the power of assembly language.

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.