Go here to sign up for The Embedded Muse.
TEM Logo The Embedded Muse
Issue Number 330, June 5, 2017
Copyright 2017 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com.

Editor's Notes

Did you know it IS possible to create accurate schedules? Or that most projects consume 50% of the development time in debug and test, and that it's not hard to slash that number drastically? Or that we know how to manage the quantitative relationship between complexity and bugs? Learn this and far more at my Better Firmware Faster class, presented at your facility. See https://www.ganssle.com/onsite.htm for more details.

Quotes and Thoughts

"Any application that contains the string 'TEMP' will be difficult to maintain because that string indicates temporary work that probably was done carelessly." - Jerry Weinberg and Gary Okimoto.

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

I'm a big fan of the Ada and SPARK languages. The latter's mantra is Correctness by Construction - that is, the code is right from the get-go; it isn't beaten into submission via debugging. There's a new (free) guide to implementing SPARK available here. There's also a SPARK manifesto here.

Should you use const or #define? Phil Koopman's always interesting blog has excellent thoughts about this here.

Another reader has nice things to say about Saleae. Dan Smith writes:

I've owned Saleae logic analyzers since 2011, starting with their original (and at that time, only) logic analyzer, aptly named "Logic", which was an 8-channel unit and very good host-side software to go with it.  Never had a problem with the unit or the software.

Fast forward to a Saturday morning in 2017, where I was debugging a strange bus timing problem under a tight project deadline.  I'd woken up early because I couldn't "turn my mind off" from the night before.  I was using a newer, more expensive model with features I needed (analog capture, must higher bandwidth, etc.)  For some reason, the unit wouldn't enumerate when I plugged it in that morning.  I contacted support on a Saturday morning; to my surprise, I had a response later that day from Mark, one of the founders and also the primary architect and developer of the host-side software. After discussing the problem and my situation, they sent me a replacement unit right away, even before the old unit was returned and inspected.  I'd also received excellent support a few weeks earlier getting the older Logic unit working with a strange combination of MacBook Pro, outdated version of OSX and an uncooperative USB port.

My point is simply that even though the Saleae products -- hardware and software -- are excellent, it's their customer service that has earned my loyalty.  Too often, great service and support go unmentioned; in my case, it's what saved me.  And yes, I debugged the problem that weekend and met the deadline!

Freebies and Discounts

Enter the contest via this link.

Keysight is offering a free USB drive that includes "webcasts, application notes, videos and blogs on a wide range of subjects from design simulation to digital and RF/uW measurements." I have one on order. Use this link.

Embedded Ransomware

Olaf Pfeiffer wrote an interesting article that he let me republish here:

Could Ransomware Go Embedded?

For criminal hackers, ransomware has become increasingly popular. Ransomware locks a PC or encrypts its data and ask for a ransom to be paid to the hackers to unlock the PC or decrypt the data.

To which extent are embedded systems vulnerable to similar attacks? How realistic is it that firmware update mechanisms are used by hackers to install foreign code? Although loading malicious code to deeply embedded systems might seem far-fetched, some of the Snowden documents have shown that this already happened to the firmware in disk drives. Also, the well-documented Jeep Cherokee attack in 2015 that allowed a remote operator to almost entirely remote control the vehicle shook the industry. A wake-up call?

The Challenges

For hackers, the challenging part is that even though there has been a development to use more off-the-shelf hardware reference designs and software, most Embedded Systems platforms are still different from each other. Different microcontrollers require different code, so that ransomware has to be tailor-made for a specific microcontroller. The bootloader mechanisms in place are also different which means hackers need to find exploits for every one they are trying to attack.

A hacker's task would be to write an exploit that manages to replace the entire original code and includes an own, password-protected, bootloader. With payment of the ransom, the hacker would share details on how to use his bootloader. There would of course always be the risk that this feature was not tested well enough by the hacker and a restore was not possible at all. It can be assumed that far more effort would have gone into generating the exploit and replacement code than the unlocking and restoring procedure.

Note that many microcontrollers have a built-in on-chip bootloader that cannot be erased or disabled, so if such a bootloader is usable in a device, a device with ransomware could be re-programmed on-site by the manufacturer or a technician. However, that might still be impractical or expensive if, for example, a very large number of devices were affected and/or the devices were at very remote locations.

A Theoretical Example

To pick a specific application example, let's have a look at an elevator / lift system: It consists of multiple microcontroller systems that are interconnected for example by CAN or CANopen and let us further assume they also feature a CAN/CANopen based bootloader mechanism.

A hacker installing ransomware replacing the existing bootloader with their own would need to get access to the system (either physical by installing a sniffer or remotely through a hacked PC that is connected to the system), know which microcontrollers are used, know how the CAN/CANopen bootloader mechanism works (with some CANopen profiles, some details about it are standardized).

This information might be stored on multiple PCs: with the manufacturers, distributors, technicians or operators of the system. If one or multiple of those get hacked, an attacker might have all this information readily available. Note that the risk of a rogue or disgruntled employee with inside knowledge is often underestimated. The information above will typically be accessible by many people.

With this information, a hacker would be able to generate and load his own ransomware loader replacing the original code in all devices, which would disable the system. Now buttons, displays and controls would all stop working and every affected device / microcontroller would require a restore of its original firmware. If the affected devices still have an on-chip bootloader and if it can be activated, then a technician could manually update all affected devices. For large elevator systems with 20 or more floors and multiple shafts this task alone could take days.

The sophistication level required for the attack described above is quite high. Not only does it require "traditional" hacker knowledge but also in-depth knowledge of embedded systems. At this time it might be unattractive to most hackers as there are possibly still many "easier" targets out there. However, with enough resources thrown at the task, a determined hacker group could achieve the tasks listed above.

What are possible counter measures?

The most basic pre-requisite for an attack as described here is the knowledge about the specific microcontroller and bootloader mechanism used. This information can be obtained by either monitoring/tracing the CAN/CANopen communication during the firmware update process or by access to a computer that has this information stored. Protecting these in the first place has the highest priority.

The designer has to make sure that the firmware update process is not easy to reengineer just by monitoring the CAN/CANopen communication of a firmware update procedure. Things that we can often learn just by monitoring a firmware reprogramming cycle:

How is the bootloader activated? Often the activation happens through a specific read/write sequence.
Counter measure: Only allow authorized partners to activate the bootloader, best by using encryption such as CANcrypt or at least a challenge/response mechanism that is not repetitive.

What file format is used? ".hex" or binary versions of it can easily be recognized.
Counter measure: Use encryption or authentication methods to prohibit that "any" code can be loaded by your own bootloader.

What CRC is used? Often a standard-CRC stored at end of the file or loadable memory.
Counter measure: If file format doesn't use encryption, at least encrypt the CRC or better use a cryptographic hash function instead of a plain CRC.

These counter measures are fall-back safeguards to protect the system if a higher security level has failed before. A hacker should not get bootloader access to a deeply embedded system in the first place. Ensure that all remote-access options to the bootloader level are well-secured.

One wonders a few things:

Does it make sense to grind markings off the CPU chip? But with so many devices using Cortex-M parts, a smart attacker could make some assumptions about the processor type.

If there's a debug port, should that be closed off?

Given that so many devices now sport nice GUIs and connectivity, Linux is a logical choice of operating systems. But it is big and vulnerable, so how does one manage Linux patches and upgrades? I can't help but wonder if it makes sense to use an RTOS coupled with GUI/networking packages from the RTOS vendor. These typically have a smaller attack surface than a big OS.

A TI Starter Kit

Mat Bennion had some experience with the TI Starterware:

We Need Simpler Ways to Access I/O on MCU"

I've been using TI's Starterware and Processor Development Kit for the AM335x (Sitara) processors for the past few months.  It's a mixed bag – literally "mixed", with little coherence between the various parts.  A few examples:

  • The sample code for the Ethernet peripheral (a fairly complex device with two ports and hardware switching capability) comes with a fully-featured TCP/IP stack that doesn't use memory copies – thus requiring a complicated memory management system that spans right from the top-level sockets down to the interrupt service routine.  In the likely event of not wanting to use that stack, you need to dismantle 30,000 lines of code to get to the Ethernet driver.  A simpler example would have been much more useful.
  • I really wanted to find a simple bootloader example, but failed.  There's plenty of support for building Linux boot images, but little or nothing bare bones.  The one example I did find, I couldn't program, because the FlashWriter tool works with some dev boards, but not others.  There's also missing documentation, e.g. I've found a description of what options can be selected by the jumpers on the dev board, but not which jumper position corresponds to which option!
  • The PDK documentation has been produced via Doxygen, so is really just a collection of function header comments – there's no overview or structure.
  • The structure of the source files is not explained, and it's complex, with a typical sample application pulling files from multiple parts of the installation tree.
  • A colleague used one of the samples to get text displayed on the dev board's LCD panel in minutes – so that feels like a success. But given how much is going on behind the scenes (e.g. font rendering), it's unlikely to be very re-usable; I imagine you'd be better off running Linux if you were planning to do graphics, to take advantage of well-maintained, standardised drivers.

In essence, it's 10 or 20 sample applications that may or may not be relevant to you and are likely to include far more than your want, so most of your effort is spent in stripping them down, rather than understanding the key points about the peripheral.  They feel like a half-way house: I imagine that users will either want to run Linux out-of-the-box, in which case these applications are irrelevant as "someone" will already have written a full suite of drivers, or they'll be running bare metal, in which case these applications are irrelevant as they're too complicated.

I'd have preferred much simpler examples, each one self-contained and well-documented in a coherent folder structure.  I didn't see any signs of a GUI for configuring the peripherals and I can't really envisage how this would work given the diversity of usages.  However, my hardware friends did use the pin mux tool to generate C code, so that seems worthwhile.

Store Config Data in Binary or XML?

A reader has a question for the Muse community. Does it make sense to store configuration data in binary or XML? Specifically:

I've been trying to find some information about people that discuss the benefits and weaknesses of using XML in embedded systems.  When you're dealing with processors that are around the 1GHz range, the 'performance' argument goes out the window – especially if you only read those configuration files at power-on. 

Have you seen any discussions weighing up the different benefits of each? 

The pro's of XML as I see them:

  • Easier to manage changes to the structure since each item is tagged, rather than having to check what version of binary structure you have with a separate handler for each
  • 'Human readable' – which doesn't matter so much once it's on your embedded device
  • Easier to update by hand – only an issue during development
  • Easier handling of Unicode text strings (I think)

The pro's of binary structures:

  • More efficient use of memory (especially for objects that can grow)
  • Quicker transfers across low-bandwidth interfaces (without needed to convert/compress first)
  • Harder for a third party to determine data structures and fiddle (assuming they can get access)
  • Easier to store large chunks of binary data…
  • Easier to protect with CRC/Hash etc

I've just put this together in a few minutes so there are likely plenty of things I haven't thought about…  I guess I was curious to know your thoughts on this subject.   Interestingly, there doesn't seem to be a lot of reading material available.

As we create more and more PC to embedded connectivity, what is going to be the way forward for passing that configuration data from your front end to your embedded systems… 

I replied:

I guess my feeling is binary is the way to go for performance reasons (speed of transfers, file size, etc.). But if those are not an issue I think XML makes more sense. One reason so many of us are frustrated with Windows is that everything is binary, whereas in Unix many files are text and thus human-readable and understandable. XML is even better since one can just define more tags to extend functionality, whereas in binary one may have some major recoding to do. And with XML generally the ordering of fields isn't important since the tags define the context. If these are configuration files then with binary you often need to write a "compiler" that translates the parameters entered by a user (perhaps via a GUI front end) into binary.

Your thoughts?


Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Joke For The Week

Note: These jokes are archived at www.ganssle.com/jokes.htm.

Ben Sweet sent some mock O'Reilly book covers over; I imagine they are copyrighted so Googled around and found these hilarious sites:



Who can resist titles like Resume-Driven Development or Excuses for Not Writing Unit Tests?

Advertise With Us

Advertise in The Embedded Muse! Over 27,000 embedded developers get this twice-monthly publication. .

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.