The Embedded Muse Logo The Embedded Muse
 Issue Number 443, April 4, 2022
Copyright 2022 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com
   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Contents
Editor's Notes

SEGGER Embedded Studio cross platform IDE

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

The Embedded Online Conference will take place next month. I'm giving a talk titled "Learning From Disaster". The cost for the entire conference is $190, but if you register here with the promo code GANSSLE it goes down to $149. This event was a huge success last year.
Quotes and Thoughts

Vannevar Bush felt that the success of any engineer's plan often depends on good writing:

"Once the plan has been decided upon, he must convince his superiors that the plan should be carried out. This convincing requires that the engineer write brief clear English, which adequately and concisely conveys the meaning in a convincing way. Good proposals have been turned down because the engineers who drafted them could not present them in convincing form."
Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Are you using MLCCs to decouple the power rails? If so, everything you know about selecting decoupling capacitors might be wrong. This article is enlightening.

Here's an interesting article about debugging analog signals on bare metal.
Freebies and Discounts

April's giveaway is a DMM.

Enter via this link.
Convincing "Them"

The quote (above) from Vannevar Bush is more important than a lot of engineers realize.

In a sense, engineers are the interface between the technology and the rest of the organization. We communicate with the non-techies via the written and spoken word. As such, documents are written with the readers' needs in mind.

Technical documents are typically either descriptive or persuasive. Descriptive docs (e.g., requirements, customer needs, code or design explanations) are, as Joe Friday would say, are "just the facts, ma'am." These are accurate and concise enumeration of what is needed or how things work. Bullets are often useful. They need to be pruned of everything not relevant. The reader needs to understand what is going on. These are educational materials.

Persuasive writing, on the other hand, is designed to convince someone that some as-yet-unmet objective needs to go our way. Want $50k for that bit of test equipment? Your goal is to prove to the reader, perhaps your boss or a higher-level boss, that the company's bottom line (or some other important metric, one that is critical to someone other than yourself) will be improved by more than the cost of the gear you want. Cite numbers, ones that are believable and substantiated. You're an engineer; your domain is numbers. Give those, but also ameliorate their sometimes daunting impact (some readers hate math) with convincing prose. But make sure the doc is tight and as short as possible, starting with an executive summary that makes your point in a few sentences.

Technical writing is not difficult. We're not creating a novel with a plot and interesting characters. Tech writing is about constructing short, declarative sentences. Using active voice. ("Meter A displays the current voltage", not "The voltage that is currently flowing will be displayed by meter A".) Paragraphs should address a single subject and be relentlessly on-point. Jargon and acronyms are often fine, but never use an acronym that either: is not industry-standard, or not previously defined. But avoid them if the target reader won't understand your point.

In many ways tech writing is writing like we speak.

Edit your work mercilessly. Some writing friends call this "baby killing", as a red pen slashing relentlessly through your prized work might be very painful, but can greatly improve the text. Be sure the document addresses the readers' needs in a clear fashion with language they will understand.

Resumes are a form of technical writing. Far too many are poorly written. They don't address the reader's need, which is simply "is this person worth interviewing?" (And, sad to say, when confronting a stack of CVs the reader's first job is to quickly find ways to cull them. Trash as many as possible to avoid wasteful interviews.) For more on them see this.

I often get writing that is incomprehensible. Or resumes that stress the irrelevant (e.g., "Mental Health: Improving, but..."). We need to take as much care with the written word as we do with crafting our code.
More on BIST

The article in the last issue about Built-In Self Test spawned some interesting email.

J.G. Harston wrote:

On 21-03-2022 13:10, Jack Ganssle wrote:
> Today I see plenty of products that run RAM tests. But a RAM failure,
> especially in an MCU, will almost certainly be massive.
...
> But your run-of-the-mill RAM test is generally pretty silly.

Just as frustrating and futile is when I try to count the available RAM on startup - trying to avoid reflections of existing RAM in blocks with partial addressing, trying to avoid memory-mapped I/O, trying to avoid any side effects of writing to ROM (write to too many locations in your EEPROM and suddenly you've written to it), trying to avoid overwriting the test code. I usually cheat and just have a couple of links on the board that the installer is required to set to select the memory size.

From Jonathan Morton:

My take on BIST is as follows:

0: If the device is in full working order, the BIST should pass, and the device should then operate normally.

1: If there is a fault in the device hardware that is sufficient to prevent the device from operating normally, the BIST should detect this fault with a reasonable degree of likelihood, report the fault to the operator (turning on a fault lamp probably suffices), then go into a quiescent "fail-safe" state.  The precise nature of this state will depend on the application of the device; if in doubt, switching things off, applying brakes, and generally "doing nothing" are *usually* the right choices.

2: As a corollary, if the BIST passes, that should offer some confidence that the device is in fact operating correctly.  This implies that the inputs appear to read plausible values, the processing hardware has not thrown any obvious faults, and - if the output functionality can be sensed somehow - that it is also responding more-or-less as expected.  Sometimes a cross-check with a nominally identical device is available; such a cross-check indicating mutual consistency should also offer some confidence in correct operation.

3: The BIST should be performed at power-up or very shortly afterwards, and should take a correspondingly brief amount of time.  Optionally, some or all of the BIST may also be performed periodically or as a continuous monitoring during operation.  This is an engineering decision, to be based on the criticality of a failure during operation.

> I think there's another aspect to BIST: the datasheet. All too often marketing demands some unexplained form of BIST so it can appear on the product's datasheet. The tests may mean nothing, but are seen as yet another selling point.
>
> When the first IBM PC came out the manual included the BIOS's source code. It started by testing the functionality of each machine instruction, halting if a failure was found. The logic of this baffled me: if instructions were to fail the code would halt or go off into the weeds. Either way the result would be a hung machine. The tests proved nothing.
>
> Today I see plenty of products that run RAM tests. But a RAM failure, especially in an MCU, will almost certainly be massive. A push/pop or call/return in the code that reports the problem will fail. The implication is that these tests have to take place practically at the startup vector, be written in assembly, and avoid all RAM use.
>
> In some instances it makes sense to look for a single event upset, such as from a cosmic ray, or to check the integrity of a data structure on boot. But your run-of-the-mill RAM test is generally pretty silly.

There's a time and place to check the basic functionality of a CPU at the instruction level.  I've seen and worked with code which actually did that for the 6502, with the intent of discovering and prompting fixes for faulty CPU *simulators*.  I can actually show you a short instruction sequence which can distinguish between an NMOS 6502, a 65SC02, a Rockwell/WDC 65C02, a 65C816, and even a common type of simulator which aims to simulate an NMOS 6502 but do so imperfectly.  But doing so on every boot is indeed pretty silly.  It makes you wonder about IBM's opinion of Intel's competence in building reliable CPUs at the time.

A quick check of the device RAM, however, is not silly.  Most CPUs have enough registers to run a basic self-addressing and walking-ones test without "trusting trust" outside of the CPU core and the ROM holding the test code.  From there, communication with sensors and actuators can be established, initialising them into safe quiescent states.  Good error handling here should form part of a BIST.

The BIST should also check at this stage that analogue inputs are not pegged at a hardware limit, as that tends to indicate a hardware fault (open or short circuit).  Inputs which should have regular and frequent activity should be monitored for such activity, and if absent or otherwise obviously abnormal, a fault condition should be inferred.

Otherwise, the BIST should stay out of the way of normal operations.

Here's Sergio Caprile's take:

So far I've managed to channel those requests into a routine that can be manually triggered (preferably only at startup with a key combination or a special plug) and serves factory testing purposes only. The code will show something at the display (if present) and wait to show key press information (if present), while cycling the outputs and lighting LEDs when inputs take the active state; if present. Analog channels may show raw input voltage at the display. That's all, just a manufacturing helper, I'll deliver a test procedure document and if something fails it is the manufacturer's responsibility to check what is wrong at the assembly facility.

Proper code being flashed is checked by the flashing tool. The core is expected to behave properly with just that, these are microcontrollers.

RAM is usually not checked, massive RAM failures most likely will manifest themselves as funny stuff during the test procedure, subtle failures will probably pass unnoticed. No reports of that so far, I guess the manufacturer would discard the board or replace the microcontroller should he find one of these; I'm usually not involved at the production level except for the very first and pilot batches.

The same routine can be used for testing in the field by installers, when in doubt.
Abstraction Redux

Muse 441 had an article on abstractions. Bogdan Baudis replied:

I think people forget that main reason to employ abstraction is to reduce complexity, not hide it.

Einstein was one of the masters of abstraction, reducing the problem to essentials only, in order to grasp the solution.
But that is from him:
"It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience." (this quote is usually shorted to:  "Everything should be made as simple as possible, but no simpler", but there is no proof Einstein ever said that).

Frequently when we mean "abstract" we really mean "encapsulate".

If we find that a particular solutions end up with similar interfaces then we may entertain the abstraction.

Too often OOP evangelist insist on finding the "true core nature of the objects" right from the start in order to define the abstract class and insist then done "the right way" will result in reusability and general happiness forever.

But in reality too often the team does NOT know really the true scope or the bugbears laying in the forests to be crossed.

I saw once a library for the data acquisition devices: the base class was covering all the types of analog interfaces, number of channels, modes of triggering, data sizes, anti-alias filter settings, communication interfaces etc. etc. in general, they were trying to describe a generic ADC-based system. If I understand this did not survive introduction of USB.

I bet that OOP experts will argue that that was because "the design was not correct". Maybe so, but I would also argue that it was simply unnecessarily overreaching.

Quotes from:
https://www.nature.com/articles/d41586-018-05004-4

Eric Smith wrote:

Around 2000 I was working for a company that was building a health care related electronic device that a doctor or clinic would send home with a patient. The device had an LCD display, a few buttons, and a telephone modem. It would ask the user questions about their health, and report the answers back to a central server. Pretty simple, right?

The first prototype had been built before I joined the company, and used a Microchip PIC16C74 8-bit microcontroller, with all of 7.5KB of program memory and 192 bytes of RAM, and a character LCD. The prototype hardware (wire-wrapped) and firmware was developed by one engineer, though later a second engineer laid out a PC board. The firmware was written in assembly language, and implemented all of the required functionality of the product. A third engineer wrote the prototype of the server side; it was written in Perl and ran on a Sun workstation.

Marketing didn't think the character LCD looked pretty enough. The hardware was redesigned to use a Motorola DragonBall (MC68328) processor at 16 MHz, 64KB of flash memory, 16KB of static RAM, and a monochrome graphics LCD. Basically the new hardware was like a Palm Pilot, but with a lot less memory, and in a much bigger box.

For political reasons, there were two independent firmware development efforts. I was on the team led by the engineer that designed the first prototype, but he had been pushed into "advanced research", which was not supposed to actually result in a product.

The main development team was led by an "expert" C++ programmer, and so naturally their firmware was written in C++, with far too many layers of abstraction. This resulted in multiple rounds of increasing the flash and SRAM sizes of successive prototypes. I think it ultimately hit 1MB of flash and 256KB of SRAM.

There was a Dallas Semiconductor "one-wire" device of some sort; I don't recall specifically what it did. As you're undoubtedly aware, the one-wire protocol is self-clocking but depends on pulse widths being within certain bounds for bits to be recognized as zero or one. A bit time generally is under 100us.

The lead software engineer just absolutely could not make it work. He insisted he was doing everything according to the datasheet. He had no experience debugging hardware, so a hardware engineer brought over an oscilloscope to look at what was going on.

The C++ code had an abstraction for the Dallas Semi chip. So far, so good. Then there was an abstraction for the one-wire protocol. Still not unreasonable. Then there was an abstraction for a transaction, then an abstraction to represent the reset pulse to start the transaction. The actual body of the transaction was handled by abstractions to read and write sequences of bytes. Then there were abstractions to read and write individual bits. Finally, there were abstractions to send and receive individual bits.

The C++ objects implementing each of these abstractions were allocated from the heap, and had constructors and destructors.

By the time all of the heap allocations and deallocations, and constructors and destructors were executed, a single bit time on the bus was about 3ms, and the zero and one bit timings actually looked almost alike. The bus low (active) time was always over 1ms. A one-wire device considers a low pulse of 480us or more to be a bus reset. Every attempt to send a bit was doing a bus reset.

That is only the lowest level of the firmware. Everything in that firmware was written that way. It's a miracle that the firmware ever got the LCD display cleared, let alone present any text or graphics to the user.

In the same time period, the "advanced research" team produced firmware written in C (not C++) that ran on the second DragonBall prototype, with only 128KB of flash and 32KB of SRAM. It met all of the product requirements and ran fast enough that the user interface had no perceptible delay.

The manager for the main firmware team was NOT happy, and did everything he could to sabotage the "advanced research" team. The main firmware team could have used some of our code, but the technical lead was still convinced that everything had to be in C++, so rather than import any of our working C code, the C++ code was all rewritten, presumably with nowhere near as many layers of abstraction.

There's a similar story about the server side, which was rewritten in Java with tons of "middleware," and ran on Sun servers, but that's a topic for another day.

This is from Matthew Eshleman:

Years ago, a large and complex SoC with reasonable 2D graphics hardware acceleration, was delivered to the team. We were also told to use the provided four layers of software abstraction on top of that hardware. We noticed that our UI seemed slow. We benchmarked and found that we could set up a simple framebuffer and then use basic software rendering with less CPU usage and faster updates than using the multi-layered code we were asked to use. Some time later a few of those layers were eliminated and we were able to use the rectangle move (BitBlt) operations more efficiently through the abstraction than we could using software. But even then, that was about it. All that being said, I am still a fan of (properly) abstracted interfaces. Performance must be given due consideration when designing such interfaces.
In Honor of April Fool's Day

In honor of April Fool's Day, here's an article I wrote for EDN in 1996.

A scruffy pair of wingtips were planted two inches in front of my face. I moaned, rolled over, feeling bits of dried blood flake out of my hair. The sour taste of last night's whiskey reminded me that somehow I hadn't made it to that AA meeting.

"Wake up, Jake." One wingtip prodded my nose. "You lousy bum. I oughta have your PI license pulled."

Lenahan stretched out an hand, yanking me roughly to my feet. "Ain't ya got some speeders to bust?" I asked, trying hard to maintain my balance. Taking stock it was pretty clear to me that balance was all I could ask for, my dignity obviously long gone. Feeling my pockets the night slowly came back. Wallet - gone. She was pretty but greedy. .45 - also missing. Hazily an image of trading it and the brass knuckles for a bottle of hooch formed in my still spinning brain.

"Angel's gonna be pissed at ya, Jake. You promised her you'd lay off the sauce." My leggy receptionist was a crusader, out to reform me, not quite understanding how impossible her mission was. I encouraged her, needing her support and willingness to miss paychecks far too often. Lenahan is right - I am a bum.

"Your child support payments are six months behind, Jake. I'm taking you in."

We'd been through much together, Lenahan and I. He spared my dignity by not cuffing me for the ride downtown.

Since they re-instituted debtor's prison back in '98 the character of the jails here in Silicon Valley has changed.  Violent criminals are rare, most released to make room for the Valley's hordes of bankrupt electronic executives.  I'd moved to Baltimore to get away from hanging judge Tito, who gleefully accepted his low-profile role of incarcerating EEs, but all too often found myself out here on the shores of the Pacific.

The joint's like an old home for me; I see it from the inside more than I'd like to admit. The regulars all waved happily. Most of the inmates are lifers. Even while doing hard time they all scheme new scams. You could see it in the graffiti - the walls were covered with schematics and financing plans. "Jake, ya got any overhead transparency material?" a young offender whispered. "I can trade ya a carton of smokes."

A few smuggled whiteboards were propped in hidden corners, their owners crouched with fistfuls of multicolored markers, glimpsing furtively over their shoulders every few seconds for a sign of the screws.

Evidence of Spike's last test equipment caper were everywhere. Smuggled scopes cast a blue glow over the dirty ceiling.  "Psssst - ya got a 20L8?" one gnarled veteran asked in a barely-audible murmur. The poor soul must have been 35 at least. "Hell, who uses them anymore?" I replied, "where've you been, old-timer? It's all PLDs and FPGAs now." He shuffled back to the bunk, grumbling "they keep stealing my copies of EDN."

I looked up as a trusty idly pushed a cart down the aisle. "Candy. Playboys. Smokes. Xilinx software" he chanted. Several prisoners greedily grabbed for the latest disks. "Register as John Doe," the pushcart vendor hissed, "you'll get your annual maintenance contract cheaper that way."

Three of us shared a tiny room lined with bars. One cellmate was a small man with thick glasses and tasseled shoes. He seemed out of his depth, a novice unable to cope with the hardened prisoners.  "I was a venture capitalist" he sobbed, " and they got me for copying Office 7 onto my computer at home. Now these brutes shove business plans at me all day long. I can't take it anymore. I'll do anything to get out of here."

The other seemed tougher. His pocket protector showed him to be a man of depth, one who was no stranger to the seamy side of the Valley.  "You Jake?" he asked, the question framed like a command.

"Yeah."

"I need a little job done. This pencil-necked geek" - pointing to the quivering VC - "promised funding for my new product.  We get the prototype working and The Organization will take care of marketing it for us. I got an engineer working on it now, and it's almost ready to go. Whatdaya say we bust outa here and you help me get it working?"

I had to do something before the DTs set in. "Sure" I agreed, and we set to work on the jail's alarm system.

The county-installed system had been improved by countless inmates. Engineers just can't keep their hands off anything, making lousy stuff good and good stuff better. I had enough 10K resistors and clip leads in my pocket, though, that bypassing it wasn't too hard. These folks were not here because of their competence, and the alarm system was now as buggy as the products they designed on the outside.

With the alarm deactivated we slipped down through the laundry room and out through the yard. The guards were gone from their posts; no one bothered us as we marched out the front gate.

Until, that is, a long white limo screeched to a halt, blocking our view of the street.  The door flew open and a meaty hand grabbed me around the throat, drawing the three of us into the car.

"Youse check bounced," Bruno growled, a red flush working its way up from his over-starched shirt collar. "You louse. I oughta off you now."

"Wait" I squeaked. "I'm good for it, Bruno, really I am. My associates here are just about to do a killer application. You can have half of my share."

"70%, and I get two seats on the Board," he demanded.

I remembered why I had moved to Baltimore. The Valley is a tough place, where the numbers do the talking, the IPO is the goal, and a decent price/earnings ratio is everyone's dream.

I needed a drink, bad, but Bruno had his driver set off at once for the lab. Bruno is the Valley's highest paid electronics consultant, and with him in our gang there was no question we'd get the product to market.

We walked down the hallway, technicians scurrying away from Bruno's shotgun eyes and terrified by his reputation. Like an out of control schoolbus he barreled through every obstruction and banged into the lab. "Where's da system?"

"Ah! here! Bruno, ah, Mr. Gessapetti." The terrified engineer pointed to a bench cluttered with equipment and debris.

"Youse slobs. Hows a artist gonna work with all dis crud around?" Bruno's tree-sized arm swept an Uzi off the bench onto the ground, along with a dozen cans of Jolt and an equal number of Twinkie wrappers. On hitting the floor the weapon fired one burst. Though three people were hit they were only co-op students, so no harm was done. After all, weren't they here to get a taste of what industry was all about? The corporate downsizings back in '99 had set a new tone in business indifference. I pried the stock options from their hands and hid them in my coat.

Spike raised his ever-present flask to his lips, tilted his head back, and took a long slow swallow. "It's like this, Bruno," he belched, "no matter what we do, we keep blowing up chips. And I mean blowing them up. Danny the Dynamiter saw the Beta 1 unit smoke and grabbed it. He ran outa here hours ago."

I had heard it on the street. Danny claimed to have gone straight, "doing parties" - whatever that meant - but now he was clearly up to no good.

"It's gotta be a static control problem" he went on. "We replaced the carpet with anti-static tile, use ozone generators, and even put our own anti-static workstations together." Waving his hands in the general direction of a line of the assemblers I noticed a dozen Slavic faces, their ankles all cuffed to the benches. "We realized the handcuffs are metallic, and it sure increases productivity while eliminating static."

So this was where the $7/hour engineers were. Now that DOD no longer tolerated underbidding on contracts, these poor souls must have been downsized into the mob. Their mournful chanting of The Song of the Volga Boatmen punctuated distant shouts of "just ship the damn thing."

I made a mental note to talk to Spike about modern management techniques.

Bruno's rumbled "whatcha doin, makin assumptions? You assume it's static? Da last guy dat assumed stuff in my outfit is now pushin up daisies. Youse need data and youse need a open mind. Use yer brain."

A midget head perched atop his enormous body gave lie to the advocates of phrenology, as I knew Bruno is as smart as he is rich. I could also see the wheels turning as he examined the circuit board.

He lightly dragged the back of his hand over the chips. "My fingers don't feel no heat no more," he muttered, "too much soldering. Da back of da hand is much more sensitive to heat. Dis chip is about ta go." A rancid smell announced a stream of smoke from the part's plastic package.

Pouring over the schematics the red started moving up from his collar again. I backed away, secretly wishing I were back in the slammer, and not the possible victim of his rage.

"Youse driving a cable with fast logic," he roared, "what kinda amateurs you got here? I betcha there's no termination on the receiving end!"

I'd seen this performance before. The scope probe was almost lost in his big hands, as he followed the signal as it propagated between boards. With each test he moved the ground clip, a pathetically short segment of wire I knew he used to keep signal integrity.

"Look here. Dere's 2 volts a undershoot at the far end of dis." Spike's protestations that the path was only 10 inches overall did nothing to soothe the enraged Bruno. He angrily threw the probe on the floor, commanding Spike to put a Thevenin termination on the receiving gate. "Yeah, right, static. Static. Static my..." his voice faded as he slumped in a chair.

Bruno had forgotten more about fast logic than I'll ever know. I sat with him, trying to understand. "Jake, ya gotta understand. Them chip vendors is shipping fast parts," he complained. "Da stuff youse buy today is much faster den da stuff we bought back when we was just learning to hot wire cars. Even wid da same part number. Ya gotta think of every signal as a transmission line."

Though the terminator cured the problem Bruno was despondent. "Da schools dese days don't make good engineers. I is da last of da Renaissance men." Trying to soothe him I dragged him into the next room, where Angel's 30th birthday party was in progress.

Looking for the bar I spied Danny the Dynamiter standing alone in a corner, a maniacal grin lighting his face, a control unit trailing wires in hand.

"Angel!" I yelled, tackling her and skidding under the table.

A second later the Piñata exploded, showering candy everywhere.

Failure of the Week

A sign of the times or a bug? Ray Keefe sent this:

Christopher Diaz sent this. He wrote "I found this fail while reviewing data sent in by a customer bewildered by the high readings on his monitoring.
The actual KW for L2 and L3 are 11.4 and 12.8KW respectively total KW is 38."

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.
Jobs!

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

 
Joke For The Week

These jokes are archived here.

David Cherry sent this excerpt from a datasheet:

 
About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.