Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 419, April 5, 2021
Copyright 2021 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Embedded Studio cross platform IDE

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

I'll be speaking at the Embedded Online Conference, which takes place from May 17-20. One talk that sounds particularly interesting is by Steve Scandore, lead software person on the Mars Perseverance Rover. There are a lot of well-known folks presenting papers.

Fabrizio Bernardini notes a double anniversary, which I had forgotten. Those of us of a certain age can remember the hysteria in the USA when Gagarin made his flight. I've read that the control panel was padlocked as they did not want him messing with the controls! Here's Fabrizio's email:

on the 12th of April some people will celebrate the 60th Cosmonautics Day, because of the anniversary of Yuri Gagarin's flight, while others will celebrate the 40th anniversary of STS-1, the first flight of the Space Shuttle. Some will celebrate both (like me).

Thinking of STS-1, perhaps you remember that the coincidence of the first Shuttle flight with such a historical date was due to a software "issue", as the flight was originally scheduled for the 10th.

The Space Shuttle is probably one of the most incredible, most complex, most everything that humankind ever built. And in that flight there were so many new technologies (shall we talk about the MIL-STD-1553, first time in space with 26! different buses on-board) that the first flight was considered one of the boldest step EVER undertaken by NASA. And the first flight was also crewed.

I can write for hours about this, but to cut the long story short, I found on the Internet this wonderful piece by Jack Garman (here), one of the engineers behind Apollo Shuttle software, that explained the incredible glitch that went undiscovered until the very first countdown that scrubbed the 10th launch attempt.

It is not only for the detailed description, but also the wonderful insights you will find in the last two pages on software role and software development and … all the usual topics we read in your beautiful newsletter: so nothing new under the Sun, but a precious memory!

Jack's latest blog: Over-Reliance on GPS

Quotes and Thoughts

An ounce of design is worth a pound of refactoring. Karl Wiegers

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Kakob Engblom's blog is often interesting. He recently posted a history of the pursuit of measuring worst-case execution time. While some successes have been had the article lays out some of the difficulties involved. I have always been frustrated that this metric, which can be so important, is so elusive.

Freebies and Discounts

April's giveaway is a digital oscilloscope kit - assemble it yourself, though the SMT part is per-soldered.

Enter via this link.

Also, Percepio is offering anyone who registers for the Embedded Online Conference by April 30 with the promo code PERCEPIO90 an extra USD100 off the registration fee. That's in addition to the early-bird discount, which also expires at the end of April. This discount combo brings the cost down to a very affordable USD90.

The Pandemic Survey Results

In the last issue I asked how your work is being affected by the pandemic. Thanks for the responses! Here are the results:

Where we're working in the pandemic

Interestingly, everyone working in the office is happy to be there, 82% of those working in a mix of the office and home are happy, and 78% of those working from home like that situation. Only a few have lost their jobs as a result of the pandemic, though 5% report having reduced wages or lost consulting work.

Here are some of the comments from readers, anonymized:

I heard an "Expert" on the radio today describing the fact that she had meetings with people from Sweden, another European country, and a group in the US in the last two days.  She claimed that this new way of meeting was impossible a year ago, and that what we used to think of as work travel for business meetings is dead and will never return.

I think she is an idiot.

I can't wait for in-person meetings to resume.

In my aerospace environment wise industries and all agencies are reporting improved efficiency, up to 30%. Not the same if you are into production, but still stuff is being delivered, and deadlines met, invoices paid.

We are currently working primarily from home with limited access to the office.  It has worked out well enough that there is a high probability that following the pandemic we may switch to a combination of work from home/ work from office based on project needs.

Working from home is GREAT.  It's much easier to get focused time at home and control my work environment.  I do think some of the 'serendipity' of ad-hoc discussions is harder at home, but -- for me -- it's been great overall.

[Working from home] has some serious drawbacks. The post-meeting meeting's don't happen anymore.  Since my job is very social, talking to someone after a meeting or meeting them in a hallway was a way to lots of work done without management interference. 

Last March I began working from home almost exclusively, going in to the office only when necessary – usually to get my hands on hardware. Then last September I began going in to the office on a full time daily basis, again due to the need to work with the electronic hardware.  Everyone here conscientiously observes all of the company mandated pandemic related precautions – masks when moving about the facility or when in close proximity to others, social distancing, limited group size for gatherings, body temperature measurement upon entering the facility, cubicle surfaces and common areas are sanitized each weekend. Many people here are working mostly from home, with a few coming in once or twice a week.

Our business was not negatively impacted tangibly by the pandemic. In fact, the pandemic afforded us the opportunity to develop a new product to aid in diagnosing COVID-19. We donated a bunch of them and so helped save lives and gain some goodwill in the community as a result.

Pre-Covid, I worked one day a week from home.  I would like to resume that after things become safe - maybe twice a week from home. I do miss having chats in the hallway and/or going out to lunch with my coworkers. My work environment at home could use some improvement, but usable. 

Are you happy with your current work environment? Yes. This is great. I'm able to pick up my laptop, monitors, and a "roll up" camping table, and setup anywhere there's internet. For example, I rented a condo for two weeks in Tahoe City, skied every morning, and still got in a full day of work every day.

The cat is out of the bag: It turns out that we don't have to be "in attendance" for most of our work. A key enabler for me is Windows Remote Desktop over a VPN. Occasionally, I need to reach out to an onsite coworker to move a scope probe or reboot something: Slack and Zoom. Finally, some things are enabled by test code, for example code to inject UI events remotely.

I miss face-to-face interactions with my co-workers, scratching out ideas on white boards, overhearing conversations, …

All of my coworkers, myself, and my wife (who works in a different field for a different company) feels like we're putting in a lot more hours. For example, we now have quite a few night time meetings, take really quick lunches, and in general are just non-stop compared to before COVID.

Pandemics remind me of one thing daily.... Open Plan offices are doomed. They were a Bad Idea to start with, and a total disaster in a pandemic.

I'm working from home (my shop, actually. A little separate metal building in the country) but that was my business model all along.

Up until the pandemic, companies were, "We don't do that. You have to come into the office." and not as interested in contracting with us.

Now we are all the rage. As busy as we can stand (there are three of us in our little LLC).

I guess it's a really ill wind that blows no one any good.

I am working from home still. Love it! More productive. Less hours for me away from home and the kids than before, yet still more hours logged at work. 
I do hope to continue this way and I think our company is game to continue. 

We found a way at the very beginning to staff the lab with a couple people to make necessary swapping of equipment so we can remote to test benches.  About 50 embedded developers at home and 2 at the lab. 

I do need to go into the lab maybe  twice a month. We don't have more than 5 in the lab at a time. 

It's like a ghost yard at work. A building for 400 and only about 20 people there each day. Mostly its the environmental test labs. 

The pandemic did not have an effect on our job security, rather we have a high demand for projects to be finished, but we are missing PCBs we design and parts for manufacturing.

We had to let go one of our engineers in embedded SW group due to discipline. Finding a replacement was really hard!
We barely had applications (is everybody staying put? Waiting for the passing of the pandemic?).

I hear most other IT companies have the same issue. Due to the increase of web marketing, web shopping, every engineer with a brain could find a job in these fields after a few weeks of learning.

Recruiting 6-8 years ago was much easier! Tons of applicants to filter, to choose from. 2-3 years ago we saw a decline of applicants. I think our profession may be running low on resupply from the universities. (at least compared to tempo all the western companies trying take a foothold on the cheap market)

No problems working from home except the home is open-plan - with my wife also working, conference calls just echo throughout the house (even with headsets).

Setup a radio bridge to bring internet out to the barn (finished space) for my Home Lab.  Now my wife likes that space better!

I currently work in Tokyo for a medical device company, well... designing medical devices. Japan has handled the pandemic in a very different way when compared to North America or Europe. In the Japanese constitution there is a weird law that the government can not interfere with people personal or business freedoms. This let do a country which did insanely well for COVID when there was no lock down, no mask mandates, no forceful shortening of business hours. Especially when you consider how much denser the population is across Japan compared to NA or EU. The main reason for this success was that everything is voluntary by both people and companies and most of them followed the guidelines. The most the government could do was to ask very nicely for people to follow their regulations and give money to companies/businesses who did what they asked. I think that is very interesting when compared to my family and friends' experiences in NA or the EU.

Also, throughout 2020 I found myself with extra time for many different reasons. So, I invested time that into improve my engineering skills and learning skills outside of engineering. Such as investing, personal finance, marketing, content creation and more. This was to allow me to be able to step out of engineering directly if needed and/or not rely on a company for my income. Because of how many people around the world lost their jobs and how something like this is likely to happen again in the future. I wanted to be prepared and not have to worry about supporting my wife and future children.

This is the largest social experiment ever done, without a proper preparation, without control groups, without deep thinking. that will affect people dramatically without really knowing it. Like kids that lost their social interaction while school is closed, till marriages relations, managed by politicians instead of by engineers & doctors. Many PHDs will be written about it even 100 years from now. I am willing to bet on it.

I run a small electronics design and manufacturing company.  At the start of the Covid lockdowns we placed several employees on furlough, but we qualified for a PPP loan and brought everyone back within 2 weeks.  We have been working in the office ever since.

I set up an office in our sound-proof music room at the rear of our house, so distractions weren't a major issue.  I found myself working longer hours, but much less productively.  Those longer hours were tempered by the fact that my wife and I would usually take a one-hour walk around our neighbourhood most lunchtimes.

Some people say working from home is more efficient. I don't agree. As an embedded engineer to work with ASIC designers the ability to have f2f discussions and draw diagrams on a white board is very important to make quick and sound tradeoff decisions etc. It makes a big difference when you can have an ad hoc discussion with your teammate when you run into some challenges. It helps by just talking to someone when you may realize what the issue is. I do feel that it is awkward to call someone who works from home. Normally I text the person first to see if she or he can accommodate a call even though I am their manager. I do agree that some people spend more time on work when working from home but at less efficiency. For example, if I have a spectrum analyzer at hand I can see the energy spike easily. Without it I will have to use another way to get what I want to see. This will take more time and I may not be 100% comfortable with the results. In any case I do believe that it is important for embedded engineers working at the office unless one does the whole project by herself/himself. 

Happy with WFH - a million times better than traveling to office

I changed jobs a few months ago.  Still sitting in the spare room, looking out into the same garden watching the same chickens walk past.  Weird, but lovely.

Prior to the pandemic, I always took every opportunity available to work from home, so I am happier, and I think more productive, than ever, since I have no commute eating into my personal time.

We use Microsoft Teams for video chat and screen sharing. It is amazing. Productivity is 5x-10x higher than before the pandemic. Doing a remote screen share to pop up some source code or document while discussing something is so awesome. Also traffic is much better than before. Also, now I get to have breakfast and lunch with my kids, which before would have been unheard of. I'm never going "back to the way things were", regardless of covid or no covid. 

A year from now, I do hope we're back to *much* more in person work.  I've been employed at the same company for 32 years and am seeing a slow destruction of the creativity and comradery that happens with the daily, random interactions with other people.  It will have long term damaging effects but management seems less than concerned about it. 

The loss of knowledge transition and mentoring to new hires is also an issue.  That takes time and a lost year+ is a long gap. 

Earlier career engineers need to be aware that staying at home makes you invisible and will increase the chances you'll be laid off in bad times. 

Despite most of us being engineers – who usually prefer working in solitude, I think my colleagues and I miss the personal interaction of work discussions and some meetings.  There's something to be said for being able to stop by someone's desk to quickly hammer out a solution to a problem or gain concurrence on a path forward.  And despite being able to multiplex (effective multi-tasking is a myth in my opinion) while on telecons when working from home, the interaction and focus of meeting around a table usually leads to a better discussion and greater understanding of each other's take on the issues.

On Software Reuse

Also in the last Muse, there were comments about problems and opportunities using off-the-shelf software. That generated a blizzard of responses.

Kevin Towers wrote:

Many years ago we used a small OTS RTOS (CMX) for a AVR based radio modem with a serial port.  During testing, we found that that if we forced more data into the modem than we knew it could handle, it would lock up and not recover.  Debugging this we found that the stack was overflowing.  In our serial interrupt routine, we had interrupts turned off while handling queues and an event signal to the RTOS.  We discovered that the call to the RTOS signal function turned interrupts back on!  This caused the next character in the UART FIFO to generate another interrupt before the current interrupt function had finished.  This caused the stack to overflow. 

I contacted the developers of CMX and told them of the problem.  Their answer was basically, they felt latency was more important and they wanted interrupts turned off for the shortest period of time.  The problem therefore was with our system and we should tell our customers not to send so much data!  Luckily, CMX shipped with the source code, so I fixed it myself.  Now when overloaded, our modem would happily drop characters but it would not crash.

Since then, I've always insisted in choosing third party tools based on whether source code is provided.  Most of the time, I never need it, but the option is there if I do.

Tovah Whitesell sent this:

First my company/group have known the frustration of an RTOS vendor who wasn't fixing discovered bugs fast enough to be of use to us. The vendor was QNX and upon trying to adopt it as our future exclusive RTOS (we currently use ThreadX) we ran into to many instances of major bugs arising that they told us they'd get fixed  … eventually. After three years they still hadn't addressed our issues and we had product to ship. Needless to say we haven't shipped with QNX and we've instead started on adapting FreeRTOS to our needs. It is frustrating to not have an off the shelf solution for an RTOS. We would demonstrate the bug via a lengthy and detailed (sometimes with the fix!) explanation but it didn't get on their list to officially fix. We had a handful of fixes patched to us but most were deferred until the end of the year … which kept getting deferred every three months or so until it became a running joke. We would normally create our own work arounds and not rely on the supplier to obtain it. In the end we abandoned QNX and will continue to utilize ThreadX and our FreeRTOS variant going forward.

Pete Friedrichs (I will soon review his latest book) wrote:

Daniel Way's contention is that software programming is "engineering" in the same way that building a bridge is, does not necessarily comport with my own experience.
Yes, there are engineering aspects to it… just as there are engineering aspects to the production of a hard-cover book. But the real work in producing the book does not lie in its mass, physical dimensions, paper composition, or font size. Its value lies in its symbolic content. In essence what any programmer is really doing is attempting to write detailed work instructions that will be unconditionally understandable for an incredibly simple-minded, overly-literal, moron… otherwise known as a computer. That's it. Thus, software "engineering" is as much a creative and literary endeavor—an art, if you will—as an engineering exercise.
Furthermore, Way's example of the hydraulic piston company and their seal/o-ring vendor glosses over some of the hazards of using canned software in your projects. Consider:

  • When a piston manufacturer acquires O-rings from a vendor, they *purchase* them. Most likely they will not be paying recurring license or royalty fees.
  • I am not aware of an O-ring manufacturer who requires a purchaser of their parts to sign a "Terms of Service" agreement to use their product.
  • O-ring vendors cannot push remote "updates" that might cripple a perfectly functioning piston in the field.
  • An O-ring vendor cannot cripple functioning pistons in the field because their corporate office has made an arbitrary "End-of-life" decision.
  • If an O-ring becomes unavailable from one manufacturer, it will mostly likely be available through another one… either off the shelf, or fabricated by order. The Piston manufacturer does not have to discard his entire tool chain and start from scratch in order to embrace an O-ring from a new supplier.
  • There is ZERO possibility that the O-ring manufacturer can insert hidden materials into their product so as to allow unauthorized control of the piston in which they are installed, nor is there any way for a 3rd party to exploit the O-ring to trigger a failure on demand.

To be clear: I am not arguing that one should homebrew code when it can be purchased, or vice versa, and I'm not arguing either approach is "best." Engineering is never about "the best." It's always about choosing between good enough or better, as measured by the constraints imposed by (usually) mutually-exclusive and conflicting requirements.
Parting shot: "Fear, uncertainty, and doubt" is not necessarily irrational. In some circles, it's also known as "due diligence."

From Steve Johnson:

Regarding bug fixes for RTOS, you may remember the "Find a bug, get a Bug" promotion for the VRTX RTOS from Hunter & Ready (before they were purchased by VxWorks or Wind River Systems, not sure which).  I didn't find a bug, but I had numerous conversations with Jim Ready about adding features (like a TCP/IP stack, and making TCP/IP communications accessible via in-circuit emulator, so I could communicate with my real time application via SSH, etc.  This was really novel at the time (1987-1991), and made my job designing the hardware control for the cardiac imaging ultrasound medical device (that we've corresponded about before) incredibly easier.  At least they seemed to be interested in fixing bugs at the time...

John Carter isn't a fan of proprietary code:

In my career dealing with proprietary software, especially black box, has been a uniform negative.

I'd _never_ recommend it, I'd always argue very strongly against it.

Dealing with open source software, yup has it's ups and downs. Sometimes I fix the downs myself, sometimes I produce a good bug report and drop it into the appropriate mailing list... and often I get a bug fix within a day.

Personally I strongly recommend using existing Open Source software... but do due diligence.

Is it maintained? Is it's quality good? Is there an active community around it? Are the developers responsive to bug reports?

I have seen companies get into trouble by skipping that step.

That said, an existing package that _doesn't_ meet those requirements,  may _still_ be better that doing it all yourself.... but the understanding must be it's almost as much work as doing it all your self. ie. If you are going to be delivering a quality product... you may have to become a primary dev on that package. Even if you have to fork it.

The advantage is you then may grow a community of users and testers and devs around that part of your product.

Dan Daly's thoughts on reuse:

Waving off a preponderance of bespoke software and a hesitance to use reusable software components as NIH or FUD (Daniel Way, TEM 418) is an oversimplification.  So-called reusable software has many problems yet to be overcome before it is as reliable as reusable hardware (ICs) has been for decades.  Until most of those problems are solved, software engineering will continue to plod through a prolonged adolescence.

I could speak at length about a number of those problems (interface design and source code that is readable and consistent come to mind as examples), but a big one is documentation: ICs come with datasheets that plainly state electrical and timing characteristics, relevant minimums and maximums, pin functions, and a wealth of other interesting and useful information - everything I need to know to use the IC.  Silicon vendors invest real money, time, and effort into the generation of these documents.

Analogously, I need detailed information about the functions that make up the interface to a piece of reusable software, and I would like to know something about preconditions postconditions, ranges of valid inputs, expected failure modes, and many other aspects.  I seriously doubt that there is anywhere near the level of investment in documentation on the software side that there has been on the hardware side for as far back as I can recall.

When "reusable" software pieces are consistently documented as well and beautifully as integrated circuits are, I might be closer to buying the assertion that too much software is bespoke.  Until then, it's a crapshoot, and in too many cases, I trust software written by myself or others close to me much more than something thrown over the wall.

This is an interesting point. At least two RTOS vendors have done a good job with this. Both Micrium and Express Logic have published substantial books detailing the operation and use of their products. (Though Micrium's software is now open source).

From Dave Smart:

As can happen, I've had an OS adoption experience, perhaps rather dated nowadays, but I think relatable by some. First, we have to roll back the calendar to the early to mid-90's. Before that point in time, individual projects (controllers on mobile equipment) used just about every 8-bit microcontroller that was available on the market – usually it was up to the individual responsible engineer to choose. All code was assembly on "bare-metal", and only pioneering projects had advanced to C. The time was right to bring more networking and major control system performance improvements, and a 16-bit part (Infineon C167) was chosen to replace the variety of 8-bitters. This was still a rather new processor architecture at the time, which meant all new hardware designs, and a new compiler. Not surprisingly, we regularly detected bugs in the compiler, and applied workarounds until updates were available. The compiler vendor was quite responsive to our needs at the time.

I had the mission to research commercially available embedded operating systems for this processor, and together with a small team, select, acquire, train, and then develop a suite of standard drivers and middleware. The plan was then to deploy this across the company for new hardware designs using the C167. With the help of Embedded Systems Programming (and other) magazine(s), I found something like 24 choices advertised! After working to down-select (I eliminated most as marketing vaporware, and in particular one I recall where the person that answered the phone was busy in the kitchen and needed to call her husband in from the backyard). Soon I had it narrowed to two choices. Both OS Vendors had been around for quite a few years supporting a variety of micros from 8 to 32-bit. For both companies, the C167 was a totally new "port". Some company internal experts had good experience with one of the two OSes, but for a different micro, so with no other discernible differences, that decided our direction and proceeded forward.

Going from bare-metal programming to using an OS was clearly a big change. Just a few years before we also held mass training in C programming, but only a few of us had applied it to product. So, we started with a small team getting trained by the OS vendor. They had an OS build that ran on PC hardware, which easily supported the training experience and learning the APIs. That all went well, and we committed the work to an upcoming major project, where about a half-dozen networked ECUs would be built from this work.

Now, with training behind us, the target compiler installed, newly designed target hardware in hand, and big-iron emulators (those $20K to $50K models from back then), we set to work. The simple hello-world worked. We added serial IO, ok. Wiggle some pins, sure. Port over an interrupt driven CAN communications driver, along with concurrent hardware capture-compare interrupts, and analog end-of-conversion interrupts, and uh, things weren't working… We usually found the code in various illegal instruction traps. Simplify the code, it works. Build it back up, trapped again.

If you could wear out the processor traps, we would have. And contacting the vendor didn't go well for way too long. We were new to OSes, and most calls ended with us convinced that we just didn't know how to properly use their OS. So, we toughed it out, week after week and into the next couple months. The overall project schedule started out relaxed, then held some risk, and approached dire jeopardy. We paid the OS vendor to send an engineer to our site to get us past our blockage. If that turned out to be elbow-to-elbow training, it would still be worth it to us.

A couple of Monday's later, the engineer (responsible for the port to this micro) arrived onsite. I think I recall he scheduled his return ticket for Tuesday afternoon. But we kept him past that. All week long, we kept finding defects that were major! We had removed our "useful" code, cutting it down to where we basically had an OS stress test with a bunch of tasks and ISRs pounding on it. With the engineer we fixed a few easy defects in the OS code. He didn't fly home until Friday afternoon, with a promise to "prioritize" them by Tuesday. We scheduled a conference call between my managers and his for a week after that, to understand when they would be fixed. At that call they offered no projected timeline.

What about the project that was in jeopardy? We couldn't wait an indeterminate amount of time, so we jettisoned that OS. Now what? Do we start fresh with the 2nd OS supplier, and the lead time to acquire the OS, training, and knowledge to get to the point where we could then assess it?

But by then the project schedule could not support starting the next research project. We ended up leveraging some internal work from the year before, which was rather primitive, but the "kernel" of which was a deterministically real-time, preemptive, priority- based task scheduler. With a rapid build-up of more drivers, all tuned for our reference hardware architecture, it was just barely successful in supporting that first products schedule. Over time, that internal work grew to become a company proprietary OS that is only just now seeing its sunset – with the mass migration to 32-bit.

It never became an "OS" in the sense of most commercial versions. On the other hand, the more restrictive architecture of this kernel, device drivers, and middleware, tuned for the company needs, forced consistent software design patterns, which led to great successes. As the lead for that team for a few years, I boasted to adopting engineering teams that day 1 would be training, day 2 would be configuring the OS (drivers mostly) to their [compatible] hardware, and we'd see it run before the end of that day on their hardware. On day 3, they could start focusing on their application as they finished the configuration of the drivers. And we supported them as they needed to develop new custom drivers and folded them into the common code. And that model succeeded for years, even as it slowly grew more and more "capable" [complex]. But who back then would have imagined a 25+ year life from their work?

Bob Japenga (you may have read his many columns in Circuit Cellar) wrote:

Why is software reuse such a hurdle when the lessons of specialization in other disciplines are clear?

Executive Summary

  1. I think that the industry re-uses software more than you think
  2. I think that re-using portions of application software is very difficult unless the requirements are similar and management is aware of the similarities
  3. Specifying different applications at the same time can increase the designer's ability to create software that can be re-used

Details - We re-use more than you think

After more than 40 years of embedded software engineering (the last 30 with my own company where lowering the cost of software development was critical) we did everything we could to try and re-use existing software. Let me tell you that it is not easy. That said - let's look are where we and the industry do re-use software:

  1. Every time you use the C run time library, you are re-using code
  2. Every time you use a proprietary boot loader, you are re-using code
  3. Every time you choose and use an off-the-shelf RTOS, you are re-using code
  4. Every time you use a Java package or Linux package (ftp, sftp, ntp, etc) or FreeRTOS extension you are re-using code
  5. Every time you use a graphics library - like Microchip often supplies, you are re-using code
  6. Every time you use a reference design (FreeRTOS or Linux based) you are re-using code
  7. Every time you use a crypto library  or wear leveling file system, you are re-using code

I could go on. I think the industry does re-use software for a large portion of all of the code we ship. We are re-using other peoples code.

Re-using Application Code

The problem comes with re-using your own software. We developed a re-use library for some low level functions - but did not use them as much as we hoped. They were either provided in the new tool chain or not needed or not as sophisticated as the new project required.

When we were successful in re-use at the application level, it was with similar products. We developed a C++ graphical user interface that we were able to reuse with 5 different products with totally different hardware interfaces. But 4 of the 5 had the requirements defined at the beginning - so our interface was designed with all of them in mind. And the fifth product was specified later but based on the same graphical paradigm - so re-use was easy. But when the 6th product came along, the interface requirements were so different that we were unable to reuse. And I think  that is the crux of the problem and also a pointer to where to go in the future. I am not convinced that in the actual development of embedded systems, what we would call the application will have much re-use except where the applications are similar. We should use off-the-shelf (re-use) software for as much of the low level components as possible and then ignore re-use for the application. Designing for re-use for an unknown future application is difficult if not impossible

Let me give you one example. Early on we needed a wear-leveling memory interface for our flash. To create a generic re-usable wear-leveling library would have been quite costly in time and dollars. But now, wear-leveling is built in - in the chip or in the file system or an off-the-shelf library. So that extra effort would have been wasted. Luckily, in that case, we recognized that and just created enough wear leveling to meet that project's requirements.  This code was not re-usable.

Other issues

Clearly there are many places that could benefit from more re-use. But to compare the application domain space of software projects to that of building a bridge is not, in my opinion, worth comparing. I think they are vastly different and software is much more complex. But that could be because I haven't designed a bridge since I was nine and we used milk cartons and 2 x 6's across a creek.

From Jon Daley:

I arrived at a company as an embedded software engineer, and the company had just switched to Windows CE from a custom OS, and they had budgeted a week to get the bootloader written and two weeks to get the rest of the OS up and running. They were 18 months late on a 6 month project (the embedded software leader is one of your readers - HI Jeff! and I'm sure he still has fond memories of those years...)

While there are lots of things that are good about off-the-shelf software, sometimes people treat it as a miracle cure.

One part of that project dealt with IrDA printers and apparently HP had a bug in some of their printers and there was a snarky comment in the Microsoft workaround code. But the reason I was looking into it was that HP had fixed their bug and Microsoft's workaround hadn't considered that HP might fix the bug some day. It was a trivial one or two line fix to work both for broken and fixed printers and I sent in the patch to Microsoft and they ignored it - I checked two major version releases later and the snarky comment was still in the source.

Another time, VMWare had an incompatibility with the latest kernel, and so I fixed it and sent it to them and published it on my website. I was an early adopter of that particular kernel but for years, my website was the only place to download a working copy of vmware.  I would dutifully fix it when there were breaking changes in the kernel, but it eventually needed more knowledge of the kernel and vmware drivers then I was interested in learning, so I switched to Virtualbox and left vmware behind.  People continued to download it for a long time because vmware didn't feel like fixing it (the original changes were just headers, not even any "real" code.

Contrast that with open source software, where I've submitted patches (or even just bug reports) and it often gets fixed that day, and the authors are thrilled when I submit a patch, so they can check and approve it, rather than have to figure it out on their own.

Somewhat related is in the web/PHP/HTML world, I have some simple "libraries" that I wrote decades ago and they are so simple there aren't any bugs. Occasionally I add a feature for a new project, but I've used that code numerous times successfully. I can't think of a PHP project that has worked for that long without changes (it was written in the PHP3 days, and still runs fine on PHP7).

Failure of the Week

From Tilak Rajesh. Note that the addressee is "null" and the salutation is "Dear ,":


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.


Joke For The Week

These jokes are archived here.

The great mathematician, John Von Neumann, was consulted by a group who was building a rocket ship.
When he saw the incomplete structure, he asked, "Where did you get the plans for this ship?"

He was told, "We have our own staff of engineers."
He disdainfully replied: "Engineers! Vy, I have complete sewn up the whole mathematical
theory of rocketry. See my paper of 1952."

Well, the group consulted the 1952 paper, completely scrapped their 10-million-dollar
structure, and rebuilt the rocket exactly according to Von Neumann's plans.

The minute they launched it, the entire structure blew up.
They angrily called Von Neumann back and said: "We followed your instructions to the
letter. Yet when we started it, it blew up! Why?"
Von Neumann replied, "Ah, yes; that is technically known as the blow-up problem ...
... I treated that in my paper of 1954."

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.