Click here to go to the on-line version The Embedded Muse 485

The Embedded Muse Logo The Embedded Muse
Issue Number 485, March 4, 2024
Copyright 2024 The Ganssle Group

Editor: Jack Ganssle, jack@ganssle.com

   Jack Ganssle, Editor of The Embedded Muse

You may redistribute this newsletter for non-commercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

Contents
Editor's Notes

SEGGER embOS Ultra Cycle-resolution accurate RTOS

Tip for sending me email: My email filters are super aggressive and I no longer look at the spam mailbox. If you include the phrase "embedded" in the subject line your email will wend its weighty way to me.

Quotes and Thoughts

So-called "natural language" is wonderful for the purposes it was created for, such as to be rude in, to tell jokes in, to cheat or to make love in (and Theorists of Literary Criticism can even be content-free in it), but it is hopelessly inadequate when we have to deal unambiguously with situations of great intricacy, situations which unavoidably arise in such activities as legislation, arbitration, mathematics or programming. - Edsger Dijkstra

Tools and Tips

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

  I'll be speaking at the upcoming Embedded Online Conference. Muse readers can use promo code GANSSLE24 to get a $100 discount if used before March 31st.

Fixing Software

Alert reader Chuck Chester sent a link to a report released by the White House last month entitled "Back To The Building Blocks: A Path Toward Secure And Measurable Software". It has two mandates to make code secure: use memory-safe languages, and use metrics to quantitatively assess software quality.

While I'm sure this is well-intentioned, the anonymous authors have produced a report that's bland and not particularly useful for those of us actually building code. A few comments:

C and C++ are, alone, called out as problematic. Rust is mentioned as a possible solution, though the report notes that it has a limited track record. No mention is made of Ada or Spark, two options whose qualities have been long proven. Strangely, one vulnerability it notes was in a Java package, a language that is considered memory safe.

The report states: "Since many cybersecurity issues start with a line of code". Well, yeah, but an awful lot stem from poor design. When a default password is included, sure, that's embodied in the code, but comes from someone making lazy design decisions.

Then there's this: "up to 70 percent of security vulnerabilities in memory unsafe languages patched and assigned a CVE designation are due to memory safety issues." The study cited about this is from Microsoft, and is about security issues in their own code. One could reasonably conclude that Microsoft has a tendency towards these sorts of errors. And one could also wonder why, given this data, they haven't changed to a memory-safe language.

I was amused by "in the case of Apollo 13 the near disaster was inadvertently caused by the laws of physics". How can the laws of physics be "inadvertent"? Software was not involved in that malfunction, and it wasn't due to the laws of physics: someone tested the tank incorrectly, and other human errors compounded the problem.

Hardware can come to the rescue: "The chip, in particular, is an important hardware building block to consider. " What chip? An AND gate? An FPGA? Are all chips MPUs or MCUs?

Formal methods are discussed at some length. But the examples given are all post-design, post-coding, and are applied only in the test phase. Static analysis is cited, but I'd hardly call that a formal method, at least as understood by the software community.

Regarding metrics: "The dynamic and evolving nature of both software development and cyber threats means that what is measured today may not be relevant tomorrow, making the
development of forward-looking, predictive measures an arduous task. These challenges are substantial, but the research community should revisit this hard problem." OK, sure, but this has been an active area of study for 50 years. It goes on at length how these as-for-now undefined metrics will change things, but is very aspirational and lacking in any specifics. How do we move forward today, other than using a memory-safe language? The report is mostly silent on this.

I'm completely sympathetic to the need for better security, and am sure the use of a memory-safe language, and some (unknown) metrics will improve things. But software is hard, software is a business endeavor. Who will pay for this? Should the quadzillion lines of extent C be recoded? How do we go about maintaining old code (COBOL taught us that old code never goes away)? These are important questions, that are sadly left unaddressed in this report.

Dysfunctional Teams

In the last issue a reader asked for advice in dealing with a dysfunctional team. That generated a lot of replies!

Steve Johnson suggested an article:

... by Phil Koopman at Carnegie-Mellon:  https://betterembsw.blogspot.com/2021/05/job-and-career-advice.html

An anonymous reader wrote:

I have some thoughts on teams. I have worked in the defense/aerospace industry for 40 years. I am an embedded software & hardware engineer. The facility I work at is an important part of military R&D.

My experience with teams is varied. My day-to-day works involves specialized hardware and software to support the warfighter. My current team is two people. Together we create a detailed specification for the hardware and software. I do the software development and my partner does hardware and FPGA development. I think we are very good at our jobs.

We don't go to many meetings, but talk and exchange email regularly to fine-tune requirements, discuss problems, etc. As a result, we spend almost 100% of our time on the current project.

My experience with large teams is not so good. Large teams spend entirely too much time in meetings. What's worse, most of the meetings have no agenda, so much of the time is wasted on off-topic things. Often there is no detailed specification for the project.

Software is written with no comments and no informational header. As a result, it's difficult or impossible to know who wrote the software, what toolchain they used, and the required external libraries, etc.

Another big problem is a lack of prototyping. As a result, hardware is usually purchased without knowing if it's appropriate or even capable of solving the problem. Issues usually end up being hacked in the software and the desired performance is degraded. Sometimes, the hardware just doesn't work in the expected way and has to be replaced with something else.

In summary, small teams made up of skilled people are usually much more effective than "all hands on deck" teams. They often produce a better product in less time. Good management minimizes the number of meetings and practices management by walking around.

Em wrote:

What I've noticed in 3 companies and 6 teams is that the people always made the biggest difference in how happy I was. The 2 best teams I was on were full of highly engaged and focused professionals, the worst teams I've been on were full of unengaged folks happy with the status quo and doing the barest minimum. One of those awesome teams worked on remarkably boring things from a tech point of view. So, I'd say look around, find the people and worry less about the tech assuming money works out (odds are the money will work out anyway).

As to the prestige thing, I wouldn't personally value it. The best engineers I've worked with came from odd backgrounds and non-prestigious companies whereas the prestigious companies produced mediocre or worse engineers as a rule of thumb. Again, follow the people, they make a bigger difference than the tech or prestige.

Job security is a weird one, I'd probably say you're never as secure as you'd like to think. Myself and other folks I know had secure jobs we got laid off from or bullied out of. At 26, you're presumably in a better position to take risks too. This is another place where people trump other considerations too, folks that value your input and drive will work hard to keep you around. People win again.

So, I guess that's where I sit. If you don't like your team and you've got options, ditch the team. Odds are you'll be happier, make the same money or more and get new colleagues that help you grow.

Clayton Auld had these suggestions:

I am an employee of a telecom provider in Alaska. We are a predominant provider across the state and have been for the past 40 years. I am roughly 7 years into my career, with 3 being at my current company. Dysfunction, miscommunication, and company politics in this organization are a given. Different groups can't communicate well together and we go through a company reorganization an average of once per year since 2018. I have learned that in this company the leadership mantra of "If you run out of ideas, reorg!" is strong.

I have been a part of groups that lack good communication, and have moved to a group that excels in that manner. The common denominator in all these situations is the manager. If the manager is a poor communicator, all members of the team will have a difficult time. However, I have a default mentality of collaboration and will do everything I can to work with anybody, no matter how difficult they are. This really does put you above the rest of the team, and good managers or leaders will see this. It is often the difference between a lackluster or poor performing employee compared to a top performer. Collaboration, communication, and teamwork are a must-have to be successful in engineering.

My advice is to learn everything you can about developing these skills. Think of it as an investment in your future training and employability, either at this company or any other. The ability to speak to managers, lead from within a team, and be a good communicator will put you above everyone of your peers. Then when the time comes where you have the opportunity to work for a manager or leader who is organized and an excellent communicator, they will quickly see your worth. My current manager saw this in myself and pushed for a 25% pay increase after 6-months on his team. I'd been paid below market value from the date I was hired, but this manager spent some political capital to match the pay structure he felt I was worth. These days a 25% pay bump is unheard-of, but by sticking with it, showing my worth, and developing my skills I was spotted by a great manager and it was worth it to stick with it and deal with the frustrating people.

TL;DR - Think of communication and collaboration skills as a training opportunity for yourself. You can learn to be a leader from within the team. Dysfunction has a cost, and I'm sure there are other members of the team who feel the same frustration you do. It takes time to build relationships, but take some time on your own or during some work hours to read about leadership. There are MANY books out there on this subject, but one of my favorite authors is Simon Sinek. Leaders are not always managers, and not all managers are leaders. By developing the leadership and collaborative mindset/skillset you can be a key part of the team and create a functional environment from within, but be patient. It takes time. And when a good manager comes along, you will be seen as a top performer who can help a team of individual contributors become a well-oiled software dev team.

And there's this from David Liu:

Having worked at six companies of very different products, cultures and team composition, I have a few suggestions to the young engineer who asked the question regarding dysfunctional teams.

It is hard to over-emphasize the importance of a good tech lead – the expertise in design and implementation details, the focus on delivery, and most importantly, the willingness to share wisdom with less experienced teammates and to involve them in discussions at all levels. My fellow engineer was reluctant to leave the Prestigious company because of "location, projects, and career advancement opportunities". In my observation, poor execution can turn any interesting project into a nightmare, and a dysfunctional team can diminish any prospect of career growth. What's left now is the location. In the mid-20s, presumably without the burden of a house, the best location is where a good career can be found. My suggestion is to join a good team for a couple of years, build up skills in software engineering and a vision in product development, then come back to the current field where he feels passionate about, and be a tech lead and a mentor for the young engineers who will be in a similar position as he is in now. That, I believe, will be a very fulfilling experience.

Long-time friend Jim Kneale wrote:

The company that I was with that got bought out by a major defense contractor was run into the ground by bean-counters chasing commercial real estate, but they taught me one thing:  Everybody at a publicly traded company is employed to increase shareholder value - from the CEO to the custodian.  The higher up in management, the more that drives decisions.  

Look at your activities and priorities through that lens.  Whining does nothing but point out problems and it probably isn't actionable.  On the other hand, documenting the interfaces to your work products so that others can better utilize them or re-use them will bring the team's performance up.  Better yet, if your work gets re-used, the value of the investment in your pay just doubled.  If some important information is fuzzy and you are guessing, capture your guess in a shared document with your name on it, have it reviewed for a consensus (right or wrong), and fix it as necessary so you now have a Source of Truth.  

Unless management is an idiot, they will recognize a trend and your stature among the quarreling newbies will rise.  One of the most satisfying things is to tell someone "RTFM, and come back if you have any questions"

Marc Fain says Bail!:

Regarding Dysfunctional teams. If the software group is as you say, then I would think your assessment that the "managers are good" is wrong. They should be able to recognize and rectify the situation. My advice, GET OUT. Not necessarily out of the company. I've heard it said, don't look for a good job, look for a good company. Once inside you'll gain insight, and there will be opportunities to move within the organization. Seeing you are already in, and assuming the company is large enough to offer other opportunities, look around. If that does not yield fruit then look outward.

Paul Marshall has struggled with a similar problem:

Read with interest the anonymous piece on dysfunctional teams, it has some resonance for me. I've worked for the same SME (electrical switchgear and drives OEM) for 38 years since graduating. For a few years in the late 90s to mid-2000s, there were two of us working 'together' designing and developing 8/16bit embedded controller boards (8051, 80C167). From the off, I have always tried to be as thorough as possible (proof of concept, specifications, design, coding, commenting, testing, archiving) but not the fastest. 

My colleague, however, seemed obsessed with doing everything as quickly as possible, often without regard for the end result. His approach was 'get it out, get it gone'.  He raced ahead with hardware and software design, sometimes cutting corners to finish jobs, but when they hit test and the test engineers found bugs, the response was 'it's designed, it's done'. Test engineers discovered text hanging off the ends of lines on the LCD, unexpected behaviour when entering parameters and other anomalies, but he usually wasn't interested in fixing the bugs, or would only do it under duress. He also repeatedly refused to work overtime to fix problems, even when the rest of us were in at weekends and evenings supporting production, doing site visits, or fire fighting.

Meanwhile, I plodded along diligently with my own projects, unable to do much to improve things. A weak manager who was averse to conflict didn't help. Ultimately, we got a new engineering director who was both technically savvy and managerially strong. He favoured my own approach rather than that of my colleague; my maverick colleague became a marked man, and eventually left after being repeatedly called to account. But we still occasionally bump up against an issue or compromise which can be traced back to his code or hardware, over 15 years after he left the business. I have since worked with other embedded engineers who were more team-oriented, which was much more satisfying. 

Nick Soroka has a twp-step program:

Where are the managers during the team meetings where time is wasted?  Your managers may be nice people, but they are not managing the project.  How about the stakeholder (the person who wants the project completed)?  Are they attending these meetings?  Do they see the lack of direction and focus?  If they are not attending the meetings, I encourage you to invite them.  Do it by saying something like 'we could use some direction in these development meetings'.  This is step 1.  It's low hanging fruit that might get things back on track.  The other thing you can do is take the lead by organizing/architecting things on an EASY TO READ flowchart.  Caps for emphasis.  Be open to making changes but if you outline things well and use that as a roadmap, it can cut down on time wasting meetings.  

Step 2.  If the managers are attending the meetings (regularly), I encourage you to look for new opportunities elsewhere.  You are not senior personnel or management so it really shouldn't be on your back to fix a team.  If you have to work long hours to learn new things to get a project completed, that is occasionally acceptable.  Working long hours to pick up the slack means you earn less and less per hour while the company earns more.  Avoid that.  There are many opportunities out there with good prospects and there are many beautiful places to live, wherever you are.  Any large city will have at least a dozen employers.  

Gabriel focuses on the team:

Instead of advice, I will describe what I did in my case (I think you should decide by yourself on yours)/

My case was identical to yours. I worked 7 years in an embedded team. Job was good and challenging, also paid extremely well compared to other jobs in the area. I tagged along the best in the team and learned a lot. I also spotted those who were there just doing the bare minimum, barely pushing things forward.
2 years before I left, the leader and all the good software engineers left the company for greener grass. In those 2 years, I was the 'senior' developer in the team.

At the time, the thought of being the 'senior' just felt wrong. Not only that but, seeing other devs doing the bare minimum while you give all you got, just to be rewarded in the same way, it's really demotivating.

That and since management could not care less for hiring good/experienced software engineers, I decided to look for other jobs. Staying there would not have made me grow like I did. 4 years since I left, no regrets.

I'm very thankful for that job, but staying in it without good teammates to share/build things was just a no go for me as a junior engineer.

Steve Ericksen has a book suggestion:

I have a comment on the subject of "Dysfunctional Teams":

Chances are there are others on the team that feel the same frustrations but do not know how to express it or have decided not to push against the current.

My tip for the young engineer is to step up and lead. Do not wait for someone else to come along. You can influence your team from any role. 

A book/author that would have served me well in my early career is Extreme Ownership by Jocko Willink.

Jon Willoughby was straight to the point:

Responding to “But the location, projects, and career advancement opportunities are hard to beat.”
Dude, you're golden, stay the course!

Daniel McBrearty is spot on when he writes:

The piece about the young engineer in a dysfunctional team struck a chord, and I guess most of us have been there at some point. I would offer the following advice to the young man or woman. 

Especially when you are in the early phase of your career, don't be too scared to search for something better. These situations happen way more often than they should. Working culture is a bit like the atmosphere in a bar - the attitude of the people who run the place permeates everywhere. Software engineering is still a relatively young art/science (compared to other branches) and poor working practices are still all too common. As you become more experienced, this gets a bit easier to handle, as you have more chance of gaining enough influence to effect some positive change, but if you are in the first decade of your career, that is unlikely. Moreover, it is pretty important, in your early years, to try to get yourself amongst people who really know what they are doing and who care about doing a good job. The mentorship you can receive in a good team is truly priceless.

So my advice is : look around. Actively seek a team that looks like it has a positive culture. Go to interviews with the objective of sussing this out. It is not always found at places with the most prestigious reputation - you can also find it in small to medium organisations. But anyway, this is what you need. Team culture has the biggest influence on quality. Don't get stuck and depressed, move on. The fact that you recognise these problems and care about them suggests that your attitude and ability will be recognised and welcomed elsewhere - as it should be.

As the saying goes : "Don’t wrestle with pigs. You both get filthy and the pig likes it."

More on Branchless Programming

Jonathan Morton wanted to add to the discussion on branchless programming:

> The real purpose of what I wrote was a musing (fitting for the Embedded Muse) as to whether branchless processing could be seen as a way to reduce the complexity due to removing branches. I think that even if the compiler was to produce branches under the hood then that would be the case, according to the definition of complexity. Certainly the obfuscation of branchless programming is unlikely to make  the code more reliable.

I would argue that explicitly branchless programming, by the usual methods of "select between two results based on a condition" or "conditional move", usually DOESN'T reduce cyclomatic complexity.  The fact that it ALSO usually impairs code readability makes this a double-whammy, since poor readability leads to more bugs being introduced.

The reason is straightforward: you still have two mutually exclusive paths of computation, and the only way to exercise both of them is to run at least two distinct tests.  Never mind that all the instructions get executed in both paths; the result of one path is rendered invisible when the final result comes from the other path.

There are cases, however, where writing code in a branchless form IMPROVES readability and PREVENTS certain types of bugs. In particular, C's ternary operator directly expresses this construct in a way that ensures that, no matter what the condition is, SOME value is assigned.  Good compilers will be able to convert this into a register-select, a conditional move, a pair of conditional stores, or plain old branches depending on what works best on the target CPU.

For example, this function might come up in machine learning:

Most compilers and targets I tried with the above in Compiler Explorer resulted in a conditional branch over the expf() call, or equivalently a conditional return.This makes sense, since the transcendental function is more expensive to compute (even with hardware FPU support) than a conditional branch.

With a simpler function:

float  ReLU(float x)
              {
              return  (x < 0) ? 0 : x;
              }

Now compilers start to employ conditional moves and so on, since both branches of the computation are available to such an instruction with less latency than a conditional branch.  This output from GCC on ARM is illustrative:

 ELU(float):
              vcmpe.f32 s0, #0
              vmrs        APSR_nzcv, FPSCR
              bxpl         lr;  conditional return
              push       {r4, lr}
              bl         expf
              vmov.f32 s15, #1.0e+0
              vsub.f32 s15, s0, s15    ; slight inefficiency here
              vmov.f32 s0, s15
              pop        {r4, pc}  ;  unconditional return
 ReLU(float):
              vldr.32 s15, .L15
              vcmp.f32 s0, s15
              vmrs  APSR_nzcv, FPSCR
              vmovmi.f32 s0, s15 ;  conditional move
              bx  lr
 .L15:
              .word  0

Testing would not reliably uncover this bug unless the (x<0) branch of ELU() was exercised. Testing (x>0) or (x==0), the latter being the boundary condition, would yield a correct result and thus hide the bug.

Failure of the Week

From Dejan Durdenic:

Eric Nelson got this compelling ad from Fry's Electronics:

Have you submitted a Failure of the Week? I'm getting a ton of these and yours was added to the queue.

Jobs!

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

 

Joke For The Week

These jokes are archived here.

Hardware, n.: The parts of a computer system that can be kicked.

About The Embedded Muse

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster.

Click here to unsubscribe from the Embedded Muse, or drop Jack an email.