Follow @jack_ganssle
Go here to sign up for The Embedded Muse.
The Embedded Muse Logo The Embedded Muse
Issue Number 366, January 22, 2019
Copyright 2018 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Contents
Editor's Notes

Express Logic

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and hugely decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Latest blog: Software Process Improvement for Firmware.

Quotes and Thoughts

"Simple solutions seldom are. It takes a very unusual mind to undertake analysis of the obvious." A. N. Whitehead

Tools and Tips

SEGGER Embedded Studio The leading cross platform IDE

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Stephen Irons had a good idea about the watchdog timer discussion:

A most useful piece of system-level diagnostic data is a non-volatile boot counter: it simply increases by one every time the system restarts. It should happen early in the startup process so that every reboot gets counted. Even a 3- or 4-bit count can be enough if storage or communication is expensive (on one system, we pay 5c per 9-byte message, or about 0.1c per bit, so we really do count bits).

Obviously, you can get more sophisticated by having separate counters for different types of restart (cold start, watchdog, brown-out, user restart, firmware upgrade, etc), if you can tell the difference. And if you cannot tell them apart, you really need to persuade the system architect that you need this data, even if it adds a bit to the BOM cost, operating cost, etc.

Want to do a startup? Luis Uribe sent this link to how Bob Metcalfe (the inventor of Ethernet) won his "overnight" success.

Phaedsys's always interesting newsletter had a link to a site with a number of papers giving advice about building security into IoT devices.

Freebies and Discounts

This month's giveaway is a 10 MHz function generator - or, rather, a bag of parts that can be assembled to be a function generator. I bought this from eBay a couple of years ago and just haven't had time to put it together. It could be a great project to do with a young 'un. There is no plastic box, and I can only presume all of the components are included.

Free function generator

Enter via this link.

AI Means "Ain't Intelligent"

I'm not sure where we are on the hype cycle for AI, but AI and IoT are increasingly paired. IEEE Spectrum and so many other publications offer a never-ending stream of articles about how AI is reshaping the world. And it has scored amazing successes. I have no doubt that we will be seeing an avalanche of applications for AI, and it will find its way into small embedded devices. One mind-blowing example of success is the AlphaZero chess-playing robot: this article is stunning.

Gartner thinks that by 2022 AI will be part of 80% of IoT devices. A Google search on "IoT AI" gives 178 million hits. I have yet to see any surveys on the use of AI in the embedded space today, and I suspect only a tiny fraction of the systems we build currently use it. To go from around zero to 80% of all IoT devices in a mere three years seems a bit of a stretch.

Many people conflate AI with "intelligence," which it is not. Wikipedia states: Intelligence has been defined in many ways, including: the capacity for logic, understanding, self-awareness, learning, emotional knowledge, reasoning, planning, creativity, and problem solving.

Current AI systems, for all of their amazing capabilities, are basically pattern classifiers. They do mirror a lot of the human brain, which itself is partially an incredible pattern classifier that can extract meaning from the visual field and auditory inputs. But the "intelligence" we posses is what happens after that classification. I doubt AI, as it exists today, will gain the capacity for understanding, self-awareness, etc. Future incarnations may change that.

Sans AI, we build systems by understanding the problem and constructing a solution whose operation is understood. AI reverses this paradigm: no one really knows how an AI system works. An ocean of coefficients selected not by design, but by training, emits answers. No one really knows why input A generated result X.

Just like the brain.

While the prospects of AI are fascinating and exciting, we need to consider the implications of the unknowability of how a particular AI system works.

What inspired this is a thoughtful email from frequent correspondent Charles Manning:

What this does bring to mind is the importance of verification and  the huge problems ahead with embedding "AI" code.

Almost all AI code is based on pattern matching processes (Bayesian classification, Neural Nets etc). With these methods, a programmer writes the Bayesian/NN engine, then the software "learns" from being presented with millions of data patterns. The result is, essentially an impregnable binary blob which nobody understands and cannot be verified.

There is no way to verify that what the AI has "learned", let alone whether it is perfect. Sure we can test it, but we can't test for everything because the real world is messy and we can't test for everything.

A decade or three back, an experimental system for the Army "learned" to identify tanks from a photo library. Almost perfect in a lab - better than people. However it failed dismally in the field. It turns out the photos with tanks in them were all lighter (or darker or whatever)  than the other photos and the classifier had really just learned to classify light vs dark. Oops!

Your AI based self driving car might have learned to change lanes safely, but it also might have learned to swerve violently to the left if it sees two blue triangles. We just don't know and we can never tell.

Considering how Toyota got skinned for $1.2bn over unintended acceleration (where no causal  fault was found).

If we, as an industry, can't get a software accelerator cable right and satisfy the lawyers, then what hope do we have with complex software embedding AI?

AI code looks like a feast for lawyers:

Shiny-pant Lawyer: So, Mr Engineer, what does your code do when it sees a red light and there are two motorcycles in front of the car?

Mr Engineer: I don't know.

Shiny-pants Lawyer: But you wrote the code. Can't you tell us what it does?

Mr Engineer: Sorry, I don't know what it does. Nobody does. It is impossible to know what it does.

...

Mr Engineer: Can I please take my tie off, it's killing me?

Kent Beck, the inventor of eXtreme Programming, said "The larger the scale, the more you must rely on emergence," by which he meant "write enough code and a design will emerge from the code." That always makes me shudder. But is AI the manifestation of that philosophy?

Minimal Debugging Strategies

Replying to my thoughts about using two GPIOs and I2C as a debugging tool, Dan Daly wrote:

A really enjoyable part (to me) of developing microcontroller firmware has always been creative ways to instrument them, especially on I/O-bound systems.

I cut my teeth on the HP1650 series of logic analyzers. Early in my career, we used this instrument, in "state-analysis" mode, to debug microcontroller firmware. With elaborate probing, we connected the entire address and data bus of the system to the analyzer, so that the analyzer could clock in every instruction or data fetch, and every write to memory. With the right triggers in the analyzer, one could capture bus activity at any point in execution of the firmware.

Those days are long gone; most micros available today do not expose their address and data bus, and no one (that I know) uses a logic analyzer that way anymore.

That would be the end of the story, until years later, at another job, I had to troubleshoot some microcontroller firmware with only one or two bits of GPIO I could configure as output for instrumentation. I could have used any logic analyzer, or even an oscilloscope, in a pinch, to capture whatever instrumentation I could shove out on those two lines, or, I could dust off the state-analysis mode of an HP1650B we just happened to have laying around the office: I would use one I/O as a strobe, and whatever other lines I could scrounge for data. With a tiny bit of additional firmware, I had a tiny, quick subroutine that could clock out whatever data I wished to see on the analyzer. Any other output lines I could repurpose made my instrumentation that many bits wider.

It gets better. The HP1650 and 16500 analyzers (I'm sure others did this as well) had a variation of the state-analysis mode that accommodated the multiplexed address/data buses common at the time those analyzers were popular. Devote an additional line as a secondary strobe, and the remaining lines do double-duty - with four spare GPIO, I can clock out four bits of instrumentation (two for "address", and two for "data", making up one clocked state. That may not seem like much, but the ability to instrument the progression through sixteen discrete states, with timing information (a time stamp for each state), in real-time (without having to shift data out through a serial port), was enough to solve many of the debugging problems we had at the time. If I were lucky enough to have six lines to use, I could clock out an entire byte of data per state.

Even if I could only get one output pin, I could hang a probe on every output of the microcontroller and strobe the state of the system out to the analyzer as often as I needed.

In their time, these instruments were popular, powerful, and expensive.

That they still can be useful is a testament to their designers.

As an epilogue, I'll give a caveat, especially regarding the venerable

HP16500 series. As time goes by, it's getting increasingly difficult to find power supply units which still operate well. As they sit on shelves in warehouses and workshops, going years with no power applied to them, electrolytic capacitors dry out. The 16500 power supply unit generates several voltages and has undervoltage/overvoltage/undercurrent/overcurrent protection built in which borders on legendary - fitting as no one would have wanted their $20K logic analyzer to fry from a power-supply glitch. The downside today is that a component does not have to stray very far out of spec for the PSU to throw up its hands and refuse to power your 16500. PSUs are still available, but you have to know where to look; my trusty 16500B is on its third one; I keep it plugged in and on standby 24/7 now.

On a side note: HP 16500s are still available on eBay. They are big beastly machines that could be configured to suck in more channels than most of us would need in a lifetime. Today, small USB logic analyzers have taken the world by storm. I see tons of teams using Saleae's. My favorite USB LA is the one from Intronix, which offers an amazing value proposition. ZeroPlus has a couple of great offerings as well.

It can be tough to get information out of a tiny embedded system. Daniel McBrearty suggested the use of a very small printf() library which is freely available here.

Several others wrote that they sent Morse Code out via a speaker to get a sense of what was going on. This is a pretty low-bandwidth solution, but it is a nice trick. Though some ham radio operators still use Morse, it is no longer required to get any amateur radio license. Once lighthouses transmitted their ID via low-frequency radio in Morse; sailors used the Morse to identify the lighthouse, and then took bearings on the signal to navigate. But even that is gone, now that we're expected to put all of our navigational hopes in GPS. So I wonder if there's any incentive anymore to learn Morse Code, and suspect it will go the way of the sextant, slide rule, and abacus.

Knowing Hardware

Sarah Wang had some comments about my thinking that firmware folks should engage in at least some of the hardware aspects of this field:

Fortunately, a recent debugging effort reset my perspective. My coworker was describing a bug that went something like this: he was trying to communicating with a 1-Wire slave device using the uC's 1-Wire master module, but he couldn't see any activity on the logic analyzer or oscilloscope. When he toggled the uC pin as a plain GPIO however, he was able to initiate communication and receive a response from the slave device.

As he was describing this to me, I thought of the following:

  1. Digital outputs typically come in two flavors: push-pull and open-drain.
    1. Push-pull outputs either "push" the output to HIGH or "pull" the output to LOW.
    2. Open-drain outputs either are high impedance ("open" switch) or sink current ("drain") to bring the output LOW.
    3. In order to achieve a logical HIGH output with an open drain configuration, an pull-up resistor is required to connect the output to the desired HIGH voltage level.
  2. Protocols in which the master and slave device share a data line (such as 1-Wire) require open-drain outputs so that no contention occurs on the line.
  3. uCs are smart nowadays -- enabling alternate functions on a GPIO, e.g. 1-Wire, usually sets the appropriate output configuration. When no alternative functions are enabled, typically the default setting is push-pull.

So I got this feeling that because nothing happening in 1-Wire mode but something was happening in normal GPIO mode, perhaps a pull-up resistor was missing. I went to check the schematic and indeed, there was none! For good measure I checked the datasheet to see if the uC handled the output configuration if 1-Wire mode was enabled. It did. He also came by to inform me that the library function he was using to initialize the pin set the output mode to push-pull. Bingo!

Being a young firmware developer, I get giddy over little accomplishments like this. I also recently just finished David J. Agans' "Debugging" book (via your recommendation), which features several war stories for which the lesson is "know the hardware too!!!" It was really fun to see it all come together. It felt good to be helpful. Experiences like these motivate me polish my hardware knowledge so I can be a more valuable member of my team. Thank you for helping me realize that.

John Taylor agreed:

Working at a cell phone company 15 years ago I got a call to come in and help diagnose a problem where a new phone failed a test that was gating our ability to ship the new product to the customer. The entire team for the project had prior to the call worked the problem for a week and having found no solution, panic had set in.

The test consisted of taking a number of phones (I believe the sample size was 20), turning them on with fresh batteries and then letting them just sit there on the network until the batteries died. They passed the test if zero samples locked up or reset themselves before the batteries were drained but about half of the devices failed this test.

It took about 6 hours to figure out that the problem was caused by one line of code being moved: The command to the DRAM to go into self refresh mode was moved to be after the code that powered down the CPU for the hardware sleep cycle (Which was about 110% of the DRAM worst case refresh period) so that if it was too long since the DRAM was refreshed and that section of memory was critical, the memory would become corrupted and the phone would lose its mind. Note: It would have taken less time to diagnose but being a hardware guy, I went through the hardware first!

Now here is the point of the story: After wrapping up the analysis and posting the report, I got a visit from a VP who asked "How did we get this far into the project and no one knew about this problem!?!". My answer was "This problem was likely always there and someone had seen it but just thought that it was a glitch, reset or exchanged their phone and never told anyone about it". The VP replied "You know, I had a couple of phones do that..." I could not resist but tell him that he was the person I had just described!

Debounce Code

There's a never-ending interest in debouncing contacts; my article on the subject gets downloaded many thousands of times per month. Andrew Kollosche sent in his debounce solution:

I came across an article, "Contact-debouncing algorithm emulates Schmitt trigger" published in EDN magazine, by Elio Mazzocca, Technical Consultant, Adelaide, South Australia (ref 1). In what could be described as a software Resistor-Capacitor (RC) Digital Filter algorithm, written in assembler for the ATmega8 microcontroller. Mazzocca states "This debouncing algorithm is effective because it 'remembers' past inputs transitions and assigns a 'weight' to each transition depending on how long ago it occurred".

I found another web reference to a similar algorithm, this time written in 'C' code, called "Debouncing Switches" from The Department of Electrical and Computer Engineering, Oregon State University (ref 2).

The tutorial declares, "This Digital filter mimics an analog RC filter, with a first-order recursive low pass filter. It has good EMI filtering, quick response. With a nearly continuous output like an analog circuit".

The digital filter features appeared to fit my purpose so I put it to work on my switch problem. In my case removing the Schmitt Trigger Flag feature as I needed auto-repeat function (listing 1). While the basic algorithm worked much better than my previous de-bounce attempts it was still occasionally sporadic. I tried variations to the algorithm applying negative feedback and hysteresis, all to no avail.

Debounce code version 1

Next I tried changes to the time-constant. Mazzocca stated in his article the algorithm could also use other time-constants (tc) 7/8+1/8 or 15/16+1/16 up to 63/64+1/64. So I started experimenting with some code and soon found the lower threshold point rose from 15 to 63 as the time-constant decreased. This was a problem at 63/64+1/64 where the 1/64 value was 3 and the lowest threshold count value was 63. The function exited before it could do any work.

To solve this I added the following lines of code:

 else                   //if input = FALSE then: RCold = RCold – 1.
   {                    //makes all the difference.
   if (RCold)           //must be positive!
   RCold = RCold - 1;   //subtract constant 1/128th.
   } 

Even changing time-constants didn't fully solve the de-bounce problem. Still getting the very occasional contact bounce, enough to be annoying.

I felt the algorithm needed a better way of being triggered into action and only when necessary i.e. change of switch state. I remembered an article titled "Debouncing Switches and Encoders" by John Youngquist published in The Embedded Muse, Issue Number 292, (ref 3). Youngquist declared his function, "requires only 3 instructions and one status storage byte. It can be triggered by positive or negative transitions. It is a simple logical edge detector".

The code was simple enough, first remember the last switch state (open or closed, i.e. 0 or 1). Next get the current switch state and use the logical XOR operator with last switch state to see if there has been any change of state? If not exit with the last known state of the switch. Otherwise process new switch state using the Resistor-Capacitor (RC) Digital Filter algorithm. Result of the RC filter is then saved as the last switch state (listing 2).

Debounce code version 2

Its fast, on a PIC16F88 @ 2MHz with no change to input switch, the function runs in 50us with delay of 400us for port configuration changes. When edge detected the RC filter takes 9ms to filter input using time-constant of 1/32nd. Not bad eh!!

I've nearly worn out the switches trying to get false triggers, so far this new variation of the filter function appears to be working quite well.

Final code:

/*
 * Switch De-bounce using Edge Detection & Resistor-Capacitor Digital Filter.
 *
 * This Digital filter mimics an analogue RC filter with first-order
 * recursive low pass filter. It has good EMI filtering and quick response,
 * with a nearly continuous output like an analogue circuit.
 *
 * Based on Elio Mazzocca, Contact-debouncing algorithm emulates Schmitt trigger.
 * Based on Oregon State University, Debounce switches algorithm.
 * Based on John Youngquist, Debouncing Switches and Encoders.
 * Version by Andrew M. Kollosche, 2015-2018.
 *
 * Expects switch contact to be on PORTB bit 0 to ground or logic zero.
 * Filter time constant: 1/32nd.
 *
 * Time-Constant  TC, UTH, LTH, Approx execute times on a PIC16 @ 2MHz.
 * No change      -         -      50us. Ideal for Superloop operation.
 * Quarter        63,  242,   15,  1ms.
 * Eight          31,  238,   12,  2ms.
 * Sixteenth      15,  230,   10,  3ms.
 * Thirty-second  07,  215,   07,  9ms.
 * Sixty-forth    03,  184,   03,  15ms.
 * Note: Thresholds count for each RC time-constant, 
 * is calculated on 3 time-constants or for Upper threshold is 96% and 
 * 3% for Lower Threshold.
 *
 * Enter - void.
 * Exit  - 0 – no contact. 50us.
 * 1 – contact.    9ms.
 */
 byte ED_RC_DFL(void)
 {
 static byte RCold = 0;              	//RC integrator hold.
 static byte Pold = 0;               	//port bit pattern hold.
 byte pin;                           	//port pins variable.
 //start of edge detector!
  pin = PORTB & 1;                       //get port bits & strip off bit 0!
  pin = pin ^ 1;                         //flip bits for active true input.
  pin = Pold ^ pin;                      //look for pin edge change.
  if(pin)                                //is there an edge detection?
    {                                    //YEAH! Now make sure its a real contact!
    while(1)                             //start infinite loop or until ...
      {                                  //... switch contacts are de-bounced.
      //first compute RCnew = RCold * fraction. Using shift and subtraction.
      //  temp = (RCold >> 2);           //use shift RCold this gives Quarter.
      //  temp = (RCold >> 3);           //use shift RCold this gives Eight.
      //  temp = (RCold >> 4);           //use shift RCold this gives Sixteenth.
      pin = (RCold >> 5);                //use shift RCold this gives Thirty-second.
      //  temp = (RCold >> 6);           //use shift RCold this gives Sixty-fourth.
      RCold = RCold - pin;               //then subtraction fraction for result.
      //get port switch bits & mask off required pins.
      pin = PORTB & 1;                   //get port bits & strip off bit 0!
      pin = pin ^ 1;                     //flip bits for TRUE input.
      if (pin)                           //got any pins?
        {                                //Yep! So add constant.
        //      RCold = RCold + 63;      //plus Quarter constant.
        //      RCold = RCold + 31;      //plus Eight constant.
        //      RCold = RCold + 15;      //plus Sixteenth constant.
        RCold = RCold + 7;               //plus Thirty-second constant.
        //      RCold = RCold + 3;       //plus Sixty-forth constant.
        }
      else                               //Zero pins input.
        {                                //subtract 1 makes all the difference.
        if (RCold)                       //must be positive.
        RCold = RCold - 1;               //subtract constant 1/128th.
        }
      Pold = pin;                        //save filtered port pins for reuse.
      //check upper & lower threshold.
      if(RCold > 215)                    //got a valid result?
        {                                //Yes, reached upper threshold.
        return(pin);                     //exit switch closed!
        }
      if(RCold < 7)                      //wait for a valid result?
        {                                //Ok, reached lower threshold.
        return(pin);                     //exit with segment bits.
        }
      }                                  //until threshold reached!
  }                                      //edge change.
   return(Pold);                         //exit with last filtered port pins.
}

References:

1) Elio Mazzocca, Technical Consultant,
https://www.edn.com/design/analog/4324067/Contact-debouncing-algorithm-emulates-Schmitt-trigger

2) Oregon State University,
http://web.engr.oregonstate.edu/~traylor/ece473/lectures/debounce.pdf

3) The Embedded Muse, Issue Number 292,
https://www.ganssle.com/tem/tem292.html

This Week's Cool Product

Microchip's new MCP6V51 op amp is advertised as a zero-drift (it's actually very low drift, ±15 uV and ±36 nV/°C), 140 dB open-loop gain, 134 dB PSRR and 135 dB CMRR device. It will take power supplies from 4.5 to 45 volts. There are built-in EMI filters. With a gain-bandwidth product of 2 MHz it's not particularly speedy, but is targeted at high-precision DC and low-frequency measurements. A buck each in 10,000 lots, $1.36 in singles.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Jobs!

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

Note: These jokes are archived here.

Not a joke; perhaps this section should be "Funny of the Week." Chris Shore, responding to the comments in Muse 364 about watchdog timers, wrote:

Very early in my career (so late 80's I guess) I was involved with developing a radio switching system. We bought in a multi-channel digital switch from a UK supplier and incorporated it. It had a green LED on the front of the box labelled "Watchdog" which flashed at a steady, slow and reassuring rate all the time the machine was operating. One day, we managed to make it crash and noticed that the watchdog was still flashing. Curious, we took the lid off and discovered that the LED was wired to the output of a 555 timer connected straight across the power rails. No wonder it always flashed!

Advertise With Us

Advertise in The Embedded Muse! Over 28,000 embedded developers get this twice-monthly publication. .

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.