Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
 Issue Number 373, May 6, 2019
Copyright 2019 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com
   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Contents
Editor's Notes

Express Logic

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

Jack's latest blog: Can AI replace firmware development?
Quotes and Thoughts

"Make it work, then make it beautiful, then if you really, really have to, make it fast. 90% of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful!" Joe Armstrong, the recently-deceased force behind Erlang
Tools and Tips

SEGGER Embedded Studio The leading cross platform IDE

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Doug Gibbs responded to last issue's article on metamorphic testing:

The section on Metamorphic testing reminded me of something that happened to a product from my past.

The product was functionally tested using tests in Python. I was looking for some unusual bugs that only occurred sometimes. The common knowledge was the bug could not exist because we were testing that section of the code.

I added tests one at a time and quickly saw problems. The test code would always run all tests in the same order. The issue had to do with uninitialized data. Previous tests had initialized things as expected. Running the tests out of order caused problems.

We added random test order to the framework. A few more problems showed up.

The simplest form of metamorphic test is be sure you can run your test in any order.
Freebies and Discounts

Dan Corley won last month's Cmicrotek current probe.

Embedded in Embedded book

Jason Long is donating three copies of his brand new book Embedded in Embedded for this month's giveaway. A short review is later in this Muse.

Enter via this link.
Scope Color Temperature and Debugging Firmware

People have come up with an amazing number of ways of extracting debugging info from embedded devices. Decades ago we used a shortwave radio to tune into a system's EMI; it was possible to tell which loops were executing by the sound. Now we have great tools like Segger's free SystemView, Percepio's Tracealyzer, Micrium's uC/Probe, and more.

In the last issue I wrote about the "color temperature" feature on some oscilloscopes, and wondered how we could use this in debugging firmware. The ideas poured in, and here's a sampling to hopefully stimulate more thoughts.

Pete Friedrichs had an especially interesting idea:

One of my tricks was to take two 8-bit DACs--one connected to the high byte of the address bus and the other connected to the low byte—and then use the DACs to feed the X and Y inputs of a scope. As you ran the code, the scope would draw a Lissajous pattern representing the program counter's migration through address space. Every program has a unique "fingerprint" and with little practice it was sometimes possible not only to detect unwanted behavior, but ascertain where in the code the problem originated.

The fleeting nature of (analog) CRT always worked against you, but a digital scope with capture and "color temperature" capability could make this idea relevant again, and even more useful/powerful than before.

Clyde Shappee wrote:

In an article you wrote you suggested a 3 bit D/A on three GPIO pins and watching the output on the scope, writing a different value as you enter a given module.

With the color, one would get a cheap profiler, I believe.

Dominic Doty's idea:

One idea I had for using the color temperature on the scope would be to visualize the schedule of a couple services. Either using a DAC or R2R on a spare port and switching it to a different value for each service. If you could have the scope trigger at the same point in the schedule you'd get a nice view of it.

Using GPIO to indicate which service is running isn't a new idea, but the color temperature would give a nice visual indication of jitter in the time that each service is dispatched, as well as variability of execution time, and the most common order in which services are executed.

Just a thought, but that's what popped into my head when I saw that capture in the newsletter.

Larry Marks is interested in tuning performance:

For performance tuning. Suppose you have, say, 3-5 modules and want to see where the code is spending its time. And you have a DAC channel or PWM digital output. Each module sets its own value on the output.

Daniel Wisehart uses the scope in X-Y mode:

On your question of "color maps" and using them in debugging firmware, I do this using EMA equivalents. Let's there there are five possible values a monitored register might have and I would like to know often each value appears, with a heavier weight on recent observations. For each of the five values--plus one for "other" or "unexpected"--I allocate a debug register. Then on each pass through my detector, I add some number--say 8 (larger numbers decay more slowly)--to the debug register that corresponds to the monitored value that is detected, and subtract 1 from each of the other debug registers whose values were not detected. By playing around with the number you add and sometimes the number of debug registers you have--by binning the monitored values--you can see a good breakdown in just how "hot" some values are.

So a straight EMA requires floating-point math. Result 'y' at time 't' for observation 'x' is y(t) = 0.8 * y(t-1) + (1-0.8) * x. So for this EMA I am taking the result to be 80% of the old result and 20% of the current observation. An EMA equivalent in integer math is y(t) = y(t-1) + (+8 or -1).

Once I have these debug registers with the strength or intensity (temperature) values for each of the values I am monitoring, I use two DAC's and a scope in X-Y mode. The X-axis and DAC are for the sample number. The Y-axis and DAC are for the intensity.

John Lagerquist and others were thinking about tasking:

I don't have a scope with this functionality but I do tend to put hooks into my RTOS to spit out the currently running task ID as an analog voltage via a DAC so that I can monitor/log which tasks are running and for how long by only using 1 pin. The color temperature may be an interesting indicator of task allocation over time.

Dave Kellogg had several thoughts:

  1. Seems like the color temperature is a Binning technique.
  2. This may be applicable to control loops as a method for visualizing the error.  For example, scope monitors a signal that corresponds to the position command in the PID controller.  The color would be an indication of the following error.  Thus a single trace (position) can be used where otherwise two are needed (position and following error).
  3. The back of my brain thinks that somehow color temperature might be useful when signals have derivative relationships.  E.g., position/velocity/acceleration/jerk.
  4. This technique might be useful for any quantity that has a ± tolerance.

Steve Wheeler contributed:

With regard to the color temperature mode on the SDS2304X scope, the only thing that comes to mind is to display how often a particular code path gets taken, such as how often a particular decision goes one way or the other. Toggle an I/O bit at the start of the code section for a trigger, then toggle another under specific conditions you want to observe. I've done similar things with incrementing debug counters, but that doesn't give you real-time information, only after-the-fact.

Eric Roesinger addressed latency:

Color "temperature" to measure distribution of a particular latency in a real-time system over a period of operation with intermittent failures, to determine whether a particular, violated latency requirement might be a (root, proximal or intermediate) cause.

Muiris Ó Cléirigh wrote:

Regarding your question for a use of Oscilloscope Color Temperature, my suggestion would be to toggle an output at each run through the main loop, and pass this through an RC filter. With a suitably low capacitor the change in voltage level would be equivalent to the average of the PWM signal, and would indirectly show when an embedded system is running slower for some reason. Maybe.

A Lesson for All of Us

There is a lesson from the 737 MAX crashes that all of us should consider. The preliminary report has data from the flight data recorder which is revealing:

737 MAX flight data recorder data

(Click on the image to get a much bigger version)

The big arrow points to the (red) line which is data from the angle of attack sensor. In under a second the AOA goes from roughly zero degrees to 75°. A little math suggests the pilots would experience an acceleration of at least two gs. The measured acceleration (bottom black line) is 1 g, and it seems improbable a large aircraft would go nearly vertical in under a second. The AOA data is either garbage or highly suspect.

Yet the computer apparently accepted it as gospel.

In Muse 370 various people opined on the notion of perfection or resilience in firmware. The bottom line is that when we read data from the real world it's wise to compare the sensed values with a model of what is reasonable and possible, before making decisions. Noise reduction via filtering is often important, but unless a sanity check is applied, filtered garbage is still garbage.
Embedded in Embedded - the Book

Jason Long of Engenuics has created a non-profit Embedded in Embedded program, which, if I understand it correctly, is an effort to bring practical real-world training in the art of embedded design to academia. He has worked with a number of universities to present the course.

He has a new book out that covers this material. Embedded in Embedded is a 550 page tome that introduces embedded wannabies to the field. I'm sure that in conjunction with the course this is an ideal way to learn quite a bit about the subject, but green hands without mentoring may struggle a little bit, as some concepts are used before being introduced (e.g., pipelining, hex notation). This is a minor complaint.

The first chapter is elementary for any engineer, but is a good and important introduction for the novice, covering Ohm's Law, switches, and the like. Though hardware is never neglected, the book's focus is mostly on crafting firmware. At every stage that firmware is about driving real-world peripherals, like PWMs, ADCs, DMA, etc.

It presumes the reader has a development board available from Jason's company. This Cortex M3 board has a raft of I/O. Working with the free version of IAR's toolchain the reader is led step-by-step through many projects. Jason recommends the use of at least one, preferably two, 24" monitors for getting the most from IAR's IDE. I agree, and find 24" barely adequate with these tools.

All of the code is provided with excellent narrative explaining all of the details. The Cortex MCUs typically includes fairly complex I/O, so I like the level of detail provided.

Chapter three teaches some assembly language and takes the reader through developing and running real code. One chapter isn't enough to dive deeply into this as the Cortex's instruction set is big (and pretty sweet, really, with a single instruction capable of doing quite a bit). Though assembly seems to be increasingly deprecated, someone has to write the startup code, and being able to understand a disassembly window is still important.

Of course, most of the focus is on C. Students will need a decent C book (as Jason suggests). I like that he integrates the lessons with the use of Doxygen and Git, providing readers with a more real-world scenario than just cranking code.

As mentioned, the MCU's I/O can be daunting, and the book can be as well. This isn't really a beach read; best, work through the examples in front of a computer with the recommended board.

Experienced developers who work at a high level may not know much of this material. Fiddling peripheral bits is a critical skill, and I hear from many managers that they can't find firmware people adept at low-level programming. Working entirely at a high-level? The book would be a career-enhancer. It could also be a great one-semester university course.

I do have one small quibble: the text is all sans serif, which is harder on the eyes than Times New Roman or a similar font.

Overall, the book is a valuable addition to the canon. It's available here.
This Week's Cool Product

I was recently on Sao Miguel in the Azores, inspecting the "Ballistic Calculator" held in the Forte de S. Bras museum. That facility is interesting though offers few details about the artifacts. The calculator is a mid-20th century "computer" used to figure out how to lob artillery shells. Inputs included altitude of the gun, barometric pressure, range, and more. It's entirely mechanical and is beautifully machined. We didn't always need a few billion transistors to compute things!

Ballistic calculator
Photo courtesy of Scott Rosenthal. Scott created this image from a number of photos as the lighting was awful and a layer of glass protects the device. Click on the image for an enormous version of it.
Jobs!

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.
Joke For The Week

Note: These jokes are archived here.

They say a picture is worth a thousand words. Sometimes two words are worth a picture. From a datasheet:
About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.