You may redistribute this newsletter for non-commercial purposes. For commercial use contact email@example.com. To subscribe or unsubscribe go here or drop Jack an email.
Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.
Jack's latest blog: Can AI replace firmware development?
|Quotes and Thoughts|
"Make it work, then make it beautiful, then if you really, really have to, make it fast. 90% of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful!" Joe Armstrong, the recently-deceased force behind Erlang
|Tools and Tips|
Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.
Doug Gibbs responded to last issue's article on metamorphic testing:
|Freebies and Discounts|
Dan Corley won last month's Cmicrotek current probe.
Jason Long is donating three copies of his brand new book Embedded in Embedded for this month's giveaway. A short review is later in this Muse.
Enter via this link.
|Scope Color Temperature and Debugging Firmware|
People have come up with an amazing number of ways of extracting debugging info from embedded devices. Decades ago we used a shortwave radio to tune into a system's EMI; it was possible to tell which loops were executing by the sound. Now we have great tools like Segger's free SystemView, Percepio's Tracealyzer, Micrium's uC/Probe, and more.
In the last issue I wrote about the "color temperature" feature on some oscilloscopes, and wondered how we could use this in debugging firmware. The ideas poured in, and here's a sampling to hopefully stimulate more thoughts.
Pete Friedrichs had an especially interesting idea:
Clyde Shappee wrote:
Dominic Doty's idea:
Larry Marks is interested in tuning performance:
For performance tuning. Suppose you have, say, 3-5 modules and want to see where the code is spending its time. And you have a DAC channel or PWM digital output. Each module sets its own value on the output.
Daniel Wisehart uses the scope in X-Y mode:
On your question of "color maps" and using them in debugging firmware, I do this using EMA equivalents. Let's there there are five possible values a monitored register might have and I would like to know often each value appears, with a heavier weight on recent observations. For each of the five values--plus one for "other" or "unexpected"--I allocate a debug register. Then on each pass through my detector, I add some number--say 8 (larger numbers decay more slowly)--to the debug register that corresponds to the monitored value that is detected, and subtract 1 from each of the other debug registers whose values were not detected. By playing around with the number you add and sometimes the number of debug registers you have--by binning the monitored values--you can see a good breakdown in just how "hot" some values are.
So a straight EMA requires floating-point math. Result 'y' at time 't' for observation 'x' is y(t) = 0.8 * y(t-1) + (1-0.8) * x. So for this EMA I am taking the result to be 80% of the old result and 20% of the current observation. An EMA equivalent in integer math is y(t) = y(t-1) + (+8 or -1).
Once I have these debug registers with the strength or intensity (temperature) values for each of the values I am monitoring, I use two DAC's and a scope in X-Y mode. The X-axis and DAC are for the sample number. The Y-axis and DAC are for the intensity.
John Lagerquist and others were thinking about tasking:
I don't have a scope with this functionality but I do tend to put hooks into my RTOS to spit out the currently running task ID as an analog voltage via a DAC so that I can monitor/log which tasks are running and for how long by only using 1 pin. The color temperature may be an interesting indicator of task allocation over time.
Dave Kellogg had several thoughts:
Steve Wheeler contributed:
With regard to the color temperature mode on the SDS2304X scope, the only thing that comes to mind is to display how often a particular code path gets taken, such as how often a particular decision goes one way or the other. Toggle an I/O bit at the start of the code section for a trigger, then toggle another under specific conditions you want to observe. I've done similar things with incrementing debug counters, but that doesn't give you real-time information, only after-the-fact.
Eric Roesinger addressed latency:
Color "temperature" to measure distribution of a particular latency in a real-time system over a period of operation with intermittent failures, to determine whether a particular, violated latency requirement might be a (root, proximal or intermediate) cause.
Muiris Ó Cléirigh wrote:
Regarding your question for a use of Oscilloscope Color Temperature, my suggestion would be to toggle an output at each run through the main loop, and pass this through an RC filter. With a suitably low capacitor the change in voltage level would be equivalent to the average of the PWM signal, and would indirectly show when an embedded system is running slower for some reason. Maybe.
There is a lesson from the 737 MAX crashes that all of us should consider. The preliminary report has data from the flight data recorder which is revealing:
(Click on the image to get a much bigger version)
The big arrow points to the (red) line which is data from the angle of attack sensor. In under a second the AOA goes from roughly zero degrees to 75°. A little math suggests the pilots would experience an acceleration of at least two gs. The measured acceleration (bottom black line) is 1 g, and it seems improbable a large aircraft would go nearly vertical in under a second. The AOA data is either garbage or highly suspect.
Yet the computer apparently accepted it as gospel.
In Muse 370 various people opined on the notion of perfection or resilience in firmware. The bottom line is that when we read data from the real world it's wise to compare the sensed values with a model of what is reasonable and possible, before making decisions. Noise reduction via filtering is often important, but unless a sanity check is applied, filtered garbage is still garbage.
Jason Long of Engenuics has created a non-profit Embedded in Embedded program, which, if I understand it correctly, is an effort to bring practical real-world training in the art of embedded design to academia. He has worked with a number of universities to present the course.
He has a new book out that covers this material. Embedded in Embedded is a 550 page tome that introduces embedded wannabies to the field. I'm sure that in conjunction with the course this is an ideal way to learn quite a bit about the subject, but green hands without mentoring may struggle a little bit, as some concepts are used before being introduced (e.g., pipelining, hex notation). This is a minor complaint.
The first chapter is elementary for any engineer, but is a good and important introduction for the novice, covering Ohm's Law, switches, and the like. Though hardware is never neglected, the book's focus is mostly on crafting firmware. At every stage that firmware is about driving real-world peripherals, like PWMs, ADCs, DMA, etc.
It presumes the reader has a development board available from Jason's company. This Cortex M3 board has a raft of I/O. Working with the free version of IAR's toolchain the reader is led step-by-step through many projects. Jason recommends the use of at least one, preferably two, 24" monitors for getting the most from IAR's IDE. I agree, and find 24" barely adequate with these tools.
All of the code is provided with excellent narrative explaining all of the details. The Cortex MCUs typically includes fairly complex I/O, so I like the level of detail provided.
Chapter three teaches some assembly language and takes the reader through developing and running real code. One chapter isn't enough to dive deeply into this as the Cortex's instruction set is big (and pretty sweet, really, with a single instruction capable of doing quite a bit). Though assembly seems to be increasingly deprecated, someone has to write the startup code, and being able to understand a disassembly window is still important.
Of course, most of the focus is on C. Students will need a decent C book (as Jason suggests). I like that he integrates the lessons with the use of Doxygen and Git, providing readers with a more real-world scenario than just cranking code.
As mentioned, the MCU's I/O can be daunting, and the book can be as well. This isn't really a beach read; best, work through the examples in front of a computer with the recommended board.
Experienced developers who work at a high level may not know much of this material. Fiddling peripheral bits is a critical skill, and I hear from many managers that they can't find firmware people adept at low-level programming. Working entirely at a high-level? The book would be a career-enhancer. It could also be a great one-semester university course.
I do have one small quibble: the text is all sans serif, which is harder on the eyes than Times New Roman or a similar font.
Overall, the book is a valuable addition to the canon. It's available here.
I was recently on Sao Miguel in the Azores, inspecting the "Ballistic Calculator" held in the Forte de S. Bras museum. That facility is interesting though offers few details about the artifacts. The calculator is a mid-20th century "computer" used to figure out how to lob artillery shells. Inputs included altitude of the gun, barometric pressure, range, and more. It's entirely mechanical and is beautifully machined. We didn't always need a few billion transistors to compute things!
Photo courtesy of Scott Rosenthal. Scott created this image from a number of photos as the lighting was awful and a layer of glass protects the device. Click on the image for an enormous version of it.
Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.
Note: These jokes are archived here.
They say a picture is worth a thousand words. Sometimes two words are worth a picture. From a datasheet:
The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at firstname.lastname@example.org.
The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at email@example.com for more information.