Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 404, August 17, 2020
Copyright 2020 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Editor's Notes

SEGGER Embedded Studio cross platform IDE

Last issue's articles about Knowledge Capture and Hardware, Software or Hardware/Software generated a flood of correspondence. It wasn't possible to include all of your replies, but so many had so much to say that this issue of the Muse is focused on some of those replies. Thanks for all of the feedback, which I always appreciate.

Quotes and Thoughts

Design is an iterative process. The necessary number of iterations is one more than the number you have currently done. This is true at any point in time. David Akin

Tools and Tips

HCC Embedded Seooc

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Referring to a note in the last Muse, David Strip wrote:

I've ordered several PCBs from these Chinese services over the years and have been pleased with the results. Most use an interface nearly identical to Seeed's. (The last couple of times I used JLCPCB.)  Just be warned that the shipping is expensive (at least compared to the PCBs themselves). That 100mm x 100mm board, 10 units, may be only $4.90, but shipping will typically push the total to $15-20. Still cheap, just not the amazing price that appears in the quote. (JLCPCB lets you see the shipping without uploading Gerbers).

Also note that 100mm x 100mm, 10 units, is definitely the sweet spot. Bump that to 200mm x 100mm and the price jumps 10X. Shrink to 50mm x 100mm and the price stays $4.90 Drop the quantity to 5 and the price remains $4.90 (though I suppose the shipping cost drops). Bump the quantity to 20 boards and the price jumps to $44.41. Odd.

While you can panelize with a single design repeated, separated by V-grooves, some houses detect multiple independent designs and bump the price.

Most of these houses seem to accept layers using the default KiCAD conventions.

All said, these are a good deal in the sweet spot of 100 x 100 2 layer boards. Not dirt cheap with other choices.

Jerry Mulchin responded to thoughts on cheap PCBs:

Here is a company, I believe in Shenzhen, China, that I use for PCB's. Have purchased several PCB's from them in the past and have not had any problems with the boards or the company. Delivery is with DHL and I usually get the boards in about 2 weeks after placing the order. Pricing is instant as well for both 2 and 4 layer boards. They are very responsive to EMail questions too. I just order five 4-layer boards, 9" by 7" for $115.74. That includes shipping by DHL, which I think is about $30.

The link for Bok Technology is https://www.boktech.cc/

I use Eagle PCB for all by boards, and BokTech has a script that generates all of the required Gerber files they require to make a PCB. They also do a flying Probe test for free on the boards. Just a happy customer.

And Dave Smart wrote:

Another site I came across is http://pcbshopper.com, which let's you define a few simple metrics and then it rounds up the pricing from 20 to 30 different PCB manufacturers (in different parts of the world) and gives you pricing based on the metrics and your desired delivery schedule. Even as a hobbyist, I don't always have the patience to go with the lowest bidder - so I've used several different manufacturers. I try to not go as small as 0402 parts, unless I forget because I'm zoomed in so much during design and they look each to install.

Sergio Caprile responded to my thoughts about state machines:

Well, I can't provide experience but some vague info. I've played with a demo of Visual State (it is (was?) really expensive for my southamerican budget). Looks very pro, and the generated code looked readable. More palatable to the OO programmer perhaps.

I've tried some examples from Pragmadev Studio http://www.pragmadev.com/product/studio.html ; didn't have the opportunity to actually use it yet. Looks great for designing comms stuff, if a bit complex to grasp the whole picture. It lets you work on the application perspective but I lost all view on what was actually going on in the hardware.

I've also toyed a bit with TASTE ( https://taste.tools ), just to have a taste of it. Hard to grasp what it actually does from my angle of view, looks like the free version of Pragmadev Studio for highly concurrent big projects. The video on using it with Simulink is perhaps the clearest explanation of it.

For hardware state machines there is Qfsm, simple and to the point http://qfsm.sourceforge.net/about.html

My state machines are still hand written C switches. Medium to low complexity and code reusability have kept me from switching so far.

Francois Baldassari also had a useful link for finite state machines:

My friends (and early Memfault customers) at Skip Scooters have released a nifty library to implement FSMs on embedded systems. It is open source on Github https://github.com/rideskip/anchor/tree/master/fsm. Embedded software engineers do not share their code nearly as often as they should, so I'm heartened to see Skip lead the way.

Freebies and Discounts

Dainius Stankevicius won the Firia Labs Explorer Kit in the July contest.

Firia Labs is donating one of their CodeBots (a Python-enabled robot with a host of sensors) for this month's giveaway.

Enter via this link.

More on Knowledge Capture

Anton Ivanovsky was one of many responding to the article on capturing knowledge:

To answer your question (How do you preserve your design notes and other important bits of information that exist outside of normal tool flow?)

At my last work I was taught to use a txt file to log snippets of information. To add a bit of redundancy I also work out of google drive which is synced between two PC's (so if one goes down I have the other PC plus online storage as a backup)

I find that using a simple txt file works quite well for me and have begun to use the same methodology for all my projects (big & small). Anything these days can read a txt file natively, plus they are easy enough to search

Ted wrote:

Enjoyed reading your summary of the "Knowledge Capture" paper. I've worked as a direct employee, contract employee, and independent consulting engineer for over four decades in Ocean, Mechanical, Electrical, and Software Engineering. I've found that shifting gears this big can be painful and time consuming.

Fortunately interdisciplinary OE (Ocean) ties many, sometimes all of these together into a single project. Primarily when I'm the sole inventor of some underwater device. In ME I've done design, but primarily Finite Element Analysis (FEA). In SE, C and C++ programming for engineering applications, and real-time embedded applications. In EE, primarily design of custom single board computers for my own OE inventions.

Knowledge capture has been something I've focused on for many decades. Starting with "crib sheets" of pertinent engineering: equations, properties, concepts for open book exams in  undergraduate & graduate engineering courses. While being self-employed I learned that any confusion over what specifically you are supposed to do on the project - will come out of my hide. So, I take notes on all pertinent phone or email conversations. I also ask a lot of questions, which I've found to be very annoying to engineering managers.

I have also created over 1000 pages of notes (not an exaggeration) on using engineering design and analysis tools, and software engineering. The notes were created with Word, but I've used Adobe Acrobat (the creator) to create gigantic PDF compilations of: my notes, excel spreadsheets, Mathcad docs, online PDF docs, relevant magazine articles, etc., etc. Separate docs for each field: OE, ME, EE & SE. These can hold virtually any file type, but view PDF's directly, and can open Office docs. You can create a multi-level Table of Contents, and can search a single doc, or across the entire PDF.

I've found this to work so well, that I expanded it to Manufacturing Engineering, Photography, Image Processing, and Outdoor Activities. I have thirty PDF docs ranging in size from 400MB to 4 GB. This actually works pretty well, but my solution has been so successful that it has become its own problem! Too many files that are too big, and making it hard to find things! I thought you might find this amusing.

Eric Geller corrected my reference to OneNote:

I also get value from MS OneNote (you called it Microsoft Notes).   I've used it for over 5 years.  In the last 3 years I committed fully to MS OneNote by forcing myself to stop taking notes with pen & paper.  The results have been excellent.

My experience in a corporate setting is that OneNote is also very useful for knowledge capture.   But even better was our initiative to create a knowledgebase using a wiki.  We chose Atlassian Confluence as the platform.  The degree of user commitment to providing input and relying on other's output is the biggest predictor for value - it creates a chain of trust.

Graham wrote:

I was involved with several, sometimes extensive, trials with tools that stored data in manufacturer-specific formats. One trial was a sophisticated multi-user text database which my whole team used to store technical information for support purposes. After I left the firm, and after several years of putting data into the tool, the software company abandoned the product. The firm I had worked for went out of business not long after.

After all these trials, where each tool 'orphaned' some of the input, I decided that freedom from the control of any software manufacturer was a primary requirement.

I migrated to WikidPad about ten years ago, and across several jobs and my private life I now have -wow- close to 18,000 short pages, each of which is a text file- copy/paste is an easy way to populate it, for all the material snagged from outside to support my decisions. It's usually the first tool I start in the morning and one of the last to close at night.

All the data I store in WikidPad is in plain text files. There is some markup to make it more functional (links to external files, some formatting), but that markup would be easy to migrate to another markup style using just a search and replace utility. WikidPad can render my pages as web pages so I can see embedded pictures, tables etc in place.

I can easily import and export, and edit the files through the WikidPad user interface or just with a text editor, it makes no difference.

WikidPad uses a database to store and search index information and wiki links- Each page (file) name is automatically a Wiki word, and WikidPad automatically highlights it as a link anywhere it finds it in a file it retrieves. The database is kept completely separate from my text, so my text is not changed by it and remains 100% portable. I synchronise a copy of all my wiki text files to my mobile phone, where I use a text search tool to access them on the phone, not WikidPad. I can equally synchronise the files to a corporate server or internet service- which I have done to share subsets with colleagues.

WikidPad works on Windows, Mac and Linux.

George sent this:

You asked for feedback on how others capture engineering knowledge.   I've been doing this ever since 1987  when a product called Tornado , a free form database that lived on a 3.5" floppy came out.

Search on this was lightning fast.  a series of square dots filled a Hercules Graphics screen. Every dot representing a note. As you typed your query  ( some word  on a note you wanted) , Notes without this  key word would melt away leaving you with a visual representation of  only the number of notes that contained your search word. It was fabulous.   For the next  20+ years  I used its successor called Infoselect.  It used a Tree outline note keeping structure , I could link to files on my hard disk and  web URLs.  It had a calendar for reminders,  flags  . Everything I could wish for for data organization both for home and work. Even Encryption of  my notes for password keeping.  Although I could store images of circuit diagrams, notes,   addresses, passwords,   Images of invoices,  I stopped using at version 9 when it became apparent it had occasional problems with data loss (unforgivable).  The program  had been using temporary  app data areas of Windows, which would vanish with data purges, along with some of my data.    

Since then I  have tried many , many  free form dAta bases. mainly trying different programs suggested by  

The product that stood out head and shoulders above the rest  of the outlining, freeform data bases ( personal organiser )  was 
The Journal  by RM David  http://www.davidrm.com/ 

Its been rock solid and bullet proof  for holding my work and home notes.  Searches are lightning fast and efficient. 
I have never lost a note or a circuit image. 

Best of all is that the  back end is based on SQL lite - a thoroughly tested product that can handle many GB of data  without getting slow or getting itself tangled up.

I can also  select which volumes get encrypted for password and circuit note keeping.  It uses Tags  and links  if you like using them, and it easily captures web pages of notes and images 

The developer is super responsive and has made many changes based on . requests of his users 

I have nothing to do with the software other than being a very satisfied engineer based user. 

Carl Palmgrenhas a couple of suggestions:

A program that I have used for a long time (since 2006) is UltraRecall offered by Kinook Software.  It runs on computers using Microsoft Windows: https://www.kinook.com/UltraRecall/

UltraRecall allows storing data in a logical format defined by the user in one, or multiple, databases according to how the user decides to segment and categorize their information.  It contains a large set of features, including an extensive and useful search function.  The user manual is available online here:

One drawback for me is that it is only available for computers running Microsoft Windows, I wish it also ran on other platforms.

Also, see KeepIt by Reinvented Software for the Mac OS.  KeepIt is not as feature-rich as UltraRecall but it is quite usable and useful.  The website for KeepIt is: http://reinventedsoftware.com/keepit/

Neil Cherry is in DevOps:

My job, DevOps (this becomes important for this and the next section). I still keep hand notes also but prefer electronic format because I can search (regexp). Drawings are done on paper or white boards and photographed. That's a bit tricky on some things as I'm not allowed to take pictures of some material.

I use Microsoft Notes though I don't like it. That's because I don't know how to use it. It's still useful but not as useful as Emacs and Org-Mode. I've been a long time Emacs user (since 1978) so I am quite familiar with the keystrokes. Seldom do I use the mouse and if I do it's in conjunction with the keyboard, not quite as good as Apple and the Mac.

One feature that's immensely useful is the literal programming. I do a lot of work with bash, Power Shell, Python, Perl, Java, Javascript, HTML/CSS and Emacs Lisp (don't ask ;-) ). I can create test cases and run them from inside Emacs, grab the results and then later manually post them to the Jira (User Story and Test Case tracking system). While Org-Mode is tricky to learn, it is incredibly powerful. It helps reduce the time I interrupt my train of thought. It even helps me to keep track of events (appointments, etc.) and allows me to put my notes in somewhat of logical order. That last part is tricky as I'm generating gigabytes of notes (with URLs, images and links to objects kept in other applications). I've been using this for the last year and it has helped in keeping me sane and helping me to keep up with the ever changing environment. I work in the Software Define Network DevOps as Quality Assurance.

Frequent contributor Luca Matteini wrote:

Now, this is hard, part maybe cause I'm in my fifties as you mention, but even because there is a lot of data to keep up with, with many designs. As I add more projects to my portfolio (and in half of them I'm not even one of the original designers) the knowledge base on them expands exponentially.

After some time investigating solutions I found a possible way to "brain books" with outliner software. Since an outline is an organized tree, where you can categorize, subdivide, and move data, it has become in my thoughts a good scheme to
list "everything" that can be described.

I have to admit that I haven't found yet the "definitive" tool, and I start feeling I suffer from the "CRIMP" syndrome

With an old projects' base, for a few reasons, I chose "KeyNote NF"https://github.com/dpradov/keynote-nf . This has a now 'ancient' user interface, but it still works as expected even on very old systems, has a small footprint and it's free.

I tried other free/open-source alternatives, but none of them satisfied me, some are too much "cloudy", others have a price tag, nearly all have an installer bigger than the office suite I use...

In Robert Lange's submission, (DFMEA = Design Failure Mode and Effects Analysis, CAR = Corrective Action Report, typically a root cause analysis of a failure:

At my previous company toward the end of each project we would update the product line DFMEA document based on the lessons learned review. Ultimately these DFMEA items would be reviewed at the start of the next project(s) and then baked into the DVP test plan. Some line items became low scores due to being designed out of or in to the product and some items required specific tests to reduce the score.

Granted, our lessons learned did not span into decades and we didn't have to capture information for many product types but as long as there is a DFMEA library that gets reviewed and added to every project it will function as a good historical archive of lessons learned.

Perhaps tying each "lessons learned" based DFMEA line item to a brief summary or CAR could wrap up the details years later. That way as associate first hand knowledge is lost due to the sands of time the core knowledge itself is still there.

Some of these DFMEA documents became quite lengthy as we added line items to cover hardware issues, software issues, wireless issues, etc. but a periodic review of the DFMEA document before each major project "gate" would help to ensure solved issues do not come back from the grave.

Daniel McBrearty likes those who can look at a problem from 10,000 meters:

Like your piece on hardware/software and systems people. I don't know if they are a dying breed but I do find there is often a lack of one on many teams. The tendency towards over-specialisation and "guru culture" in, especially, the software industry has really exacerbated this. I think it can have a bad effect on systems design and projects because it encourages over-complication. Specialists who lack system level awareness can have a tendency to want to use the latest smart technique they learned just because they learned it. The question "is this really worth the complexity and risk" or (better) "is there a simpler, cleaner way" is often not asked. And no one wants to "kill their babies", or risk the political/social consequences of killing someone else's. This is a big enemy of simplicity, just the right amount of which is key to reliability. And sometimes the only way to fix a design that is spiraling out of control is to take the scissors to it.

A good systems guy is the one person with the perspective to say "this looks like too much" or "we are fixing the wrong problem" here, or to recognise that things are becoming overly complex. I have a rule of thumb : if you can't describe what a system does (from a high level perspective, without going into details of subsystems) in 5 or 6 sides of A4, you might already have a problem. If you need 40+ and lots of indecipherable diagrams, you do. Projects I have worked on that went well could generally pass this test. There have been one or two that didn't ...

Martin Glunz is a systems person:

 From my personal history I'd consider myself a "system person". Started my career in the 90's, having done all the usual stuff you mentioned in the Muse, I do acknowledge "That was a lot of fun!".

And I want it back.

Today, after having spent many years in the hardware department I've seen a lot of shortcomings with having people specialized to their field of experience:

- It's quite hard to get something to do out of your "proven field of experience". If you don't work constantly and hard on it, you'd always be assigned to the same class of tasks. Looks just like managers think "Someone who's doing frobnitz all the time is best occupied with more frobnitz to do." They don't see the worthiness of experience in other fields of engineering.

- Thinking and acting on system level often is discouraged by others (managers and even some engineers) though they say "system level thinking is good" when directly asked.

For example, a hardware guy prefers to build an elaborate discrete multiplexer to route a larger number of analog signals to a few ADC inputs of a MCU. If a system guy proposes a cheaper solution that might involve some cross-department action, this solution often is declined for the sake of "too much effort to implement".

- Managers tend to make decisions that look wrong from a system guy's view, driven by short-sighted goals. In the long term, I've seen these decisions fail or end up in broken schedules and  significantly raised total NRE cost.

J.G.Harston was slightly off topic but brings up an important point:

> Ford learned at great expense how to position gas tanks due to the
> Pinto fiasco, but do young engineers who weren't even alive at the
> time know those lessons learned so painfully, so long ago?

This was exactly my thoughts a few days ago when I frustratedly administered percussive punishment to my tablet. Why oh Why oh Why oh Why does every generation have to be retaught that the absolute immediate first priority of any user interactive system is to immediately AT ONCE respond AT ONCE to indicate that the user has done something. Think about what to do later, give feedback RIGHT NOW THIS INSTANCE. Why oh Why oh Why does this even have to be taught in the first place, how do these people have such little sense of the universe around them that they haven't learned this lesson within the first few milliseconds of programming their first code?

With normal mechanical user interfaces you get feedback from the mechanics of the mechanism, you feel the button move, you hear the button hitting the end of the travel mechanism. With touch screens there is no mechanical feedback which means the software must give feedback.

But all too many systems only give feedback when they get around it.
But no feedback is an explicit signal to the user that the user hasn't pressed the button correctly, it is an explicit instruction to the user to try again.

You reach for the light switch, you don't feel it move, you try again.
You push the door latch on the washing machine, you don't hear it click into place, you push it again.

You press a button on your tablet. The tablet doesn't respond. You press it again. It doesn't respond. Clearly, you aren't pressing the button correctly. You try to carefully aim your non-transparent finger that is larger than the target, there is still no response.
You press harder. Still no response. Press again even harder. And harder. And harder, until your finger pops through the display. You then administer percussive punishment across the steering wheel of your car.

(rant over!)

More on Hardware, Software, or Hardware/Software?

John Sloan and many others had thoughts about this article in the last Muse:

As a software engineer, the projects I've worked on over the past couple of decades have been based on RTOSes like VxWorks and more recently Linux. True, a lot of the software developers on those projects didn't have to know that much about the hardware. But in my experience that has to be at least one software engineer that surprises the hardware team by insisting on getting a copy of the hardware schematics and bill of materials so they can look up most of the data sheets. And that person will inevitably end up not only writing most of the device drivers and other low level software, but will also be crucial in assisting in hardware/firmware/software integration, systems testing, and even sometimes debugging when the higher level developers get caught by some low level weirdness like a bug that sends you down to the assembly code to figure it out. Sure, those sorts of systems people are expensive in the short run. But in my experience they are highly cost effective in the long run. (Yeah, I'm typically that sort of systems person.)

I like to say "I'm not a hardware person, but I have a hardware person on speed dial."

Boi Okken wrote:

After reading your interesting article on "hardware, Software or Hardware/Software" with great interest, I noticed that you came up with the job category 'systems person'. As far as I know, it already has its own term!

They are system engineers, and are generally more experienced engineers that grew out of their original field to specialize in the management, integration and macro scale design of big projects. More and more this is also turning into its own specialization, with courses in universities for just this job. System engineers are usually highly experienced in a multitude of fields, and they are usually highly sought after and paid a pretty penny for their work.

So I wouldn't say they are a dying breed, rather morphing into its own specialization.

Steve Taylor made an important point about the Socratic Method::

On systems people: Yes, they didn't exist in the company I emigrated from the UK to join. Not sure if we are making systems people anymore, but its been lucrative for me. There was very much a silo mentality and I'm afraid it showed in our engineering and blame culture. Right now,  I manage (or perhaps "consult" in all  design decisions in mechanical, electronics and firmware, as well as negotiating manufacturing contracts. I've found that Socratic interventions eventually sink in, and my team are actually writing stuff into Confluence pages at least. The boys have found that actually written notes too save their asses when the inevitable "Did you test X ?" and "How did you test X", so the idea is gaining popularity. Socratic methods means they are now alive to thinking "Oh shit, what did he mean by that "- and reanalysing their problems" .

I'm a fan of the Socratic Method, which is basically a process of helping people figure things out by asking good questions. It's far better to mentor a young engineer by using questioning to help that person gain understanding than simply tossing off an answer.

John Kougoulos wrote:

We call them generalists in IT operations. In many cases they are underestimated because everything works so management does not even realize how important it is to have someone to put the puzzle in order, or, and perhaps even more important, to translate things from one specialist to the other. Some others call them devops. And some others "site reliability engineers".

Outside operations, some consulting companies understand their value, they label them something like "Delivery Architects", but mostly in more application level projects.

Typically they need something like 15 years of experience and many of them have grown in parallel with the technology, so they managed to absorb the growth of information over the years.. eg I started as a developer, became a systems administrator, then network admin, then security and so on.

But there are also the "hackers". These are these young ones that learn everything in dt/2 and they want to know everything in detail to be able to exploit the bugs/design issues. In the last few years they have also learned hardware things, with amazing  results.

Are they a dying breed? Probably no, startups tend to have multi skilled people in order to survive. 

Are they enough though? Probably no. Demand seems to grow. A friend of mine was referring to these people as "self-oscillating"; they don't need someone to tell them what to do all the time, they find ways to keep themselves busy by learning new things and they are more on the hands-on side of things. There is a certain percentage of people thinking and working this way.

Can we have more in our field? Maybe, but it sounds difficult because it needs long term planning. If the companies pay for specialization, available immediately, how do you convince someone to grow his knowledge wide but deep enough instead of very deep (that probably pays more)? And when is it wide enough to hire him as a "systems person" and more importantly how do you interview them. Again this guy was telling me.. the most important interview question to hire one of these guys eg in the IT field, is to ask them if they have a car and know how a 4-stroke engine works. This would give you a clear hint that the guy was curious enough to learn how something works, despite his enthusiasm for digital things. But it does not sound like a legitimate interview question and it does not mean that he is ready yet.

Perhaps job rotation for the ones that show the skills with a good level of job assurance?

John's comments on specialization resonated with me. My dad was a mechanical engineer who worked on Apollo at Grumman. He told me one of the engineers there was the world's expert in designing wheels for lunar roving vehicles. I've always wondered what happened to that person post-Apollo? Specialization can be useful, but remember that technology moves on and that certain skill can be obsolete overnight.

Francois Baldassari figures it's all about seniority:

In my experience this is just a matter of seniority. Excellent senior EE's can write some firmware, excellent senior FWE can rework and debug a board.

Senior engineers are not disappearing, but they are rarer than junior ones: many are lost along the way to retirement, promotions into management, career pivots, and newsletter writing ;-).

I was lucky to work alongside a fabulously talented team at Oculus (the makers of virtual reality headsets), where I could point to 8 or so "systems people" who could do it all.

Failure of the Week

S. Stewart sent this. When the Wyze watch Android app doesn't sync on Blue Tooth it seems to fill some data fields with 72123:


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.


Joke For The Week

These jokes are archived here.

Outage (as in Internet outage): Isn't it interesting how close that is in spelling to "outrage"?

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.