Go here to sign up for The Embedded Muse.

Embedded Muse 206 Copyright 2011 TGG April 6, 2011

You may redistribute this newsletter for noncommercial purposes. For commercial use contact jack@ganssle.com. To subscribe or unsubscribe go to https://www.ganssle.com/tem-subunsub.html or drop Jack an email at jack@ganssle.com.

EDITOR: Jack Ganssle, jack@ganssle.com

Editor's Notes

Are you happy with your bug rates? If not, what are you doing about it? Are you asked to do more with less? Deliver faster, with more features? What action are you taking to achieve those goals?

In fact it IS possible to accurately schedule a project, meet the deadline, and drastically reduce bugs. Learn how at my Better Firmware Faster class, presented at your facility. See https://www.ganssle.com/onsite.htm .

I stumbled across the self-billed "World's largest online CPU museum."
http://www.cpushack.com/ . And then there's CPU Galaxy (http://www.cpu-galaxy.at/). It seems some folks collect processors. If you're into the history of this field the sites are worth a gander.

Quotes and Thoughts

Do not believe any programmer, manager, or salesperson who claims that code can be self-documenting or automatically documented. It ain't so. Good documentation includes background and decision information that cannot be derived from the code.
- Jef Raskin

Tools and Tips

The tool tips keep pouring in! Keep `em coming.

Steve deRosier is yet another one who likes the Logic from http://www.saleae.com: "I too love the Logic. It's simple and the price is right. But perhaps the best thing about it is the software is cross-platform. I purchased it shortly after it was released, on the _promise_ that they'd soon have a Mac and Linux client. It took longer than I liked, since I had to run it in a VMWare virtual machine on my Mac, but eventually they released a cross-platform software package. Thank you! Finally an embedded tools vendor that understands there's other operating systems out there than Windows."

Backups

Just in time for World Backup Day a TV show lost much of their data which was held on their ISP's server. An ex-employee of the ISP logged in and reportedly went on a deletion spree.

There are a couple of lessons here: first, change passwords when people leave! Second, don't trust any single backup. Don't trust a DVD, hard disk, or the cloud. Use defense in depth: multiple solutions. Don't just mirror your drive; be sure the backup strategy keeps multiple copies of deleted and changed files. Audit the backup process from time to time to ensure that it is working, and that you can effectively restore files.

A lot of people are resorting to backups in the cloud, but be wary. Will the service survive or will it succumb to the usual irrational exuberance of Internet-based businesses? Some of the services have rather checkered histories and policies, so be sure to investigate deeply before committing to one. I have seen complaints that some services throttle bandwidth so much that it can take a year or more to fully upload a disk's worth of data. and there's little to indicate that the backup is incomplete. In other cases users complain that when trying to restore data the cloud backup is missing files.

Debouncing

Mark Armbrust found a really, really lousy switch that frustrated the ideas in my special report on debouncing:

>From https://www.ganssle.com/debouncing-pt2.htm
> Figure 1 shows the classic debounce circuit. Two cross-
> coupled NAND gates form a very simple Set-Reset (SR) latch.
> The design requires a double-throw switch."

I did this for years using 74LS00 for the NANDs. Last year I found a switch so crappy that this technique failed me.

I had a student who was showing an interest in hardware so I spent 3 days with him over Thanksgiving break with my old protoboards, a box of TTL and my cheap analog scope. (I'll do just about anything to encourage potential HW engineers!)

On the third day he ended up building an 8-bit adding machine (adder/accumulator). One button to clear the register, one to load it with the sum of its output and an 8-bit DIP switch, 8 LEDs for output. He was using LS00 debounced pushbuttons from the previous day's project.

The mystery was that he was sometimes getting multiple adds from pushing the add button. I put the scope on the clock where it went into the register and couldn't see anything definite, but there was significant ringing and I thought it might be worse than it appeared since we we using a cheap 60MHz scope. Adding termination reduced the problem but didn't eliminate it.

Later at home I got out the DSO an tracked it down. The add switch was really funky; sometimes it spent a long time in the analog region, causing the LS00s to have bad output rise times. Changing to Schmitt trigger NAND gates (74LS13 or 74LS132) solved the problem.

I think that there must be some oxide from one of the switch fingers that came loose and is bouncing around in the switch. Sometimes it gets between the fingers and makes a point contact diode resulting in a 1 voltish level before there is permanent closure.

More on Local Knowledge

My thoughts on Local Knowledge last issue spawned quite a bit of debate.

From a reader who wishes to be anonymous: "I worked for a Japanese automotive OEM for 7 years. Their method of maintaining "local knowledge" for items that have cost the company lots of money (due to design failures) is to keep those items in a large database. At the start of each new design cycle, all engineers working on the project must review every item that relates to their area (i.e. engine, electrical, transmission, body etc). Then, after every issuance of drawings and prototype builds, this database needs to be updated to show that each engineer has checked his/her drawings for each item that he is responsible for. It's extremely tedious and time consuming but gets the job done. Unfortunately, I have not come across any North American company that performs this type of check."

Another who wants to remain anonymous wrote: "A year after leaving my employer with a severance package, they called me back in to solve a problem their remaining engineers could not. I had institutional knowledge that the institution no longer had. So I went back as a contract worker and solved the problem in weeks. They kept me on board for several months and I was amazed at how often I was called upon because of my Local Knowledge. Someone would stop me in the hallway, pick my brain for 10 minutes, and it would save him or her two or three days of work. I, a contract worker, was asked to be on focused troubleshooting teams.

"Before I left the company, I watched them try to outsource parts of their development. A local company could not do it because they did not have the Local Knowledge. Outsourcing to India had limited success but was hampered with their frequent turmoil causing us to keep retraining new people to be our replacements. Opening a site in China and moving a few local engineers there to train them was wasted effort when they closed those doors after two years.

"Even locally, I observed how much of the procedures and methodologies are unwritten when a new engineer came on board. Knowledge of the magic incantation for different boot modes kept changing so documentation was out of date.

"Those are a few of the short comings of Local Knowledge. So what, from my engineering but non-management point of view, could solve this problem? Companies (by that I mean corporate, the big bosses, etc.) need to have a long-term view of things. If they are going to spend money to expand Local Knowledge, commit to sticking with it for at least 3 product generations or about 5 years. Provide enough engineering bandwidth so engineers have time to learn about other engineer's jobs, enabling them to step in if that engineer leaves. Document long-term insights (such as where to put the gas tank in a car) but not worry about changing procedures (such as the boot mode incantations de jour.)"

Steve deRosier wrote: "Usually the engineers that are being laid off, as well as the survivors, know exactly how important the Local Knowledge is! Usually the short-sighted management either doesn't know or care!"

Charles Manning submitted his thoughts: "There are at least two or three sides to this:

"While you've talked about older people, it is really about anyone who has been around long enough to be one of those repositories of knowledge.

"They often have wisdom about an industry segment or particular product. They are often the repositories of tribal knowledge. In that they are valuable.

"However a system that depends on the presence of these people is broken. Eventually that person will be lost to the organisation and it vital that their knowledge is infused into the organisation before that happens. Of course this does not just apply to older folks. It also applies to any single points of failure in the human department. If your business is dependent on the health of one individual then you have a problem.

"The corollary to your example of customer retention is this: How will your customers feel when you say that you can't support that product and they will have to wait until Old Jim comes back from hospital/world cruise? Or worse if he dies.

"You also need to be careful that the old bloke does not feel threatened and entrench his position. I've seen too many organisations which depend on an "engineering hero" who is the only person understands some particular code or technology - often some arcane precarious code written in 6805 assembler or such, but when you flow chart it the code is doing something quite simple in a very cryptic way.

"The trick is to get the benefits without the downsides. There are some ways to do that:

"* Mentoring. Have the old guy mentor and work with fresh blood. Let the tribal knowledge infuse. Get him involved in the next generation products and design reviews.

"* Capture knowledge: Make that corporate culture. Get the old guy writing docs explaining how stuff works. Get him to recode the assembler in C and run it on a simulator. Older people can easily learn new languages and skills too.

"* Phase out those old products. If the market segment is worth it, then redesign (involving the old bloke). If not, then cut the product."

Geoff Patch said: "You article in EM 205 about keeping older engineers around pressed one of my many buttons.

"I've been here at CEA just shy of 20 years. The 4 guys who report directly to me have been here for 23, 17, 10 and 9 years respectively.

"Long term retention of staff has always been one of our key competitive advantages, and the company has never laid off a staff member due to lack of work. Even through difficult times, we've kept staff on that weren't making money, because the guys who own the company see it as making good commercial sense in the long term, as well as just being the decent thing to do. They're good blokes to work for, and I like working in a place where loyalty goes down, as well as up.

"Anyway, we're still here and doing well after 27 years of continuous growth, so I reckon that they have the right idea, and other companies would do well to learn the lesson.

"We regularly work with other companies where the staff have virtually no knowledge of their products and can barely support them because the original developers were laid off years ago. In contrast, last year we made some substantial modifications to a radar system that we installed in 1995. We were able to do the work quickly, inexpensively and to the customers great satisfaction simply because many of the people who built the original system are still here. Without our local knowledge, the project may well have been infeasible.

"And I thought I was pretty damn good back in 1995, but looking at that dusty old code from the perspective of 2010 made me shake my head in wonder at how primitive it all was. But I guess that's a story for another day."

And Tom Mazowiesky wrote: "Regarding local knowledge, perhaps the software that IBM developed to play Jeopardy might have some application here. I think your observations regarding local knowledge are spot on."

Jobs!

Let me know if you're hiring firmware or embedded designers. No recruiters please, and I reserve the right to edit ads to fit the format and intents of this newsletter. Please keep it to 100 words.

Tamura Corporation of America, in San Diego County, has a two to six month position for an embedded s/w engineer to help write and approve the software spec, design and write the s/w part of a small system, and support the final hardware/software integration and the DVT. Must have recent experience with PIC16 assembly and be familiar with basic control-loop theory for power converters up to 30W. Prefer BSEE, BSCE, or BSCS with hardware experience.

Please send resume to the HR Manager: starkso@tamuracorp.com

Au-Zone has two positions open in Calgary, Canada. First is for a Senior Embedded Firmware Architect:

Responsibilities:
- Detailed embedded F/W design and development
- Hands on development of BSP (board support packages), libraries and user applications
- Device Driver development, debugging and board bring-up in collaboration with H/W designers
- Technical and team leadership

Requirements
- B.Sc/M.Sc EE, CE or CS
- 8 -> 12+ years embedded software development experience
- 4+ years of embedded Linux
- 3+ years device driver experience with common cores, i.e. ARM, PPC, x86, OMAP, i.MX
- Familiarity with embedded OS such as VxWorks, LynxOS, Linux, etc

Desirable
- Do-178B experience
- Experience optimizing Linux kernels
- Experience with DSP/FPGA algorithm development
- Experience with SVN, Git

Personal Characteristics
- Positive, proactive problem solving attitude
- Strong communicator

They are also looking for an Intermediate Embedded Firmware Developer:

Requirements
- B.Sc/M.Sc EE, CE or CS
- Experience using gcc and gdb in a cross compile environment
- comfortable with scopes, logic analyzers, etc.

Desirable
- Familiarity with embedded OSs: Embedded CE/XP, pSOS, ThreadX, etc..
- Experience with optimizing Linux kernels, file systems and drivers for real time operation
- Knowledge of the setup and use of debuggers, profilers, coverage tools and trace tools

Personal Characteristics
- Comfortable working as part of a development team and able to follow technical direction
- Positive, proactive problem solving attitude

Contact garthburlingham@shaw.ca

Joke for the Week
-----------------
COSTELLO CALLS TO BUY A COMPUTER FROM ABBOTT

ABBOTT: Super Duper computer store. Can I help you?

COSTELLO: Thanks, I'm setting up an office in my den and I'm thinking about buying a computer.

ABBOTT: Mac?

COSTELLO: No, the name's Lou.

ABBOTT: Your computer?

COSTELLO: I don't own a computer. I want to buy one.

ABBOTT: Mac?

COSTELLO: I told you, my name's Lou.

ABBOTT: What about Windows?

COSTELLO: Why? Will it get stuffy in here?

ABBOTT: Do you want a computer with Windows?

COSTELLO: I don't know. What will I see when I look at the windows?

ABBOTT: Wallpaper.

COSTELLO: Never mind the windows. I need a computer and software.

ABBOTT: Software for Windows?

COSTELLO: No. On the computer! I need something I can use to write proposals, track expenses and run my business. What do you have?

ABBOTT: Office.

COSTELLO: Yeah, for my office. Can you recommend anything?

ABBOTT: I just did.

COSTELLO: You just did what?

ABBOTT: Recommend something.

COSTELLO: You recommended something?

ABBOTT: Yes.

COSTELLO: For my office?

ABBOTT: Yes.

COSTELLO: OK, what did you recommend for my office?

ABBOTT: Office.

COSTELLO: Yes, for my office!

ABBOTT: I recommend Office with Windows.

COSTELLO: I already have an office with windows! OK, let's just say I'm sitting at my computer and I want to type a proposal.
What do I need?

ABBOTT: Word.

COSTELLO: What word?

ABBOTT: Word in Office.

COSTELLO: The only word in office is office.

ABBOTT: The Word in Office for Windows.

COSTELLO: Which word in office for windows?

ABBOTT: The Word you get when you click the blue 'W'.

COSTELLO: I'm going to click your blue 'w' if you don't start with some straight answers. What about financial bookkeeping? You have anything I can track my money with?

ABBOTT: Money.

COSTELLO: That's right. What do you have?

ABBOTT: Money.

COSTELLO: I need money to track my money?

ABBOTT: It comes bundled with your computer.

COSTELLO: What's bundled with my computer?

ABBOTT: Money.

COSTELLO: Money comes with my computer?

ABBOTT: Yes. No extra charge.

COSTELLO: I get a bundle of money with my computer? How much?

ABBOTT: One copy.

COSTELLO: Isn't it illegal to copy money?

ABBOTT: Microsoft gave us a license to copy Money.

COSTELLO: They can give you a license to copy money?

ABBOTT: Why not? THEY OWN IT!

(A few days later)

ABBOTT: Super Duper computer store. Can I help you?

COSTELLO: How do I turn my computer off?

ABBOTT: Click on 'START'...............

About The Embedded Muse

The Embedded Muse is a newsletter sent via email by Jack Ganssle. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. can take now to improve firmware quality and decrease development time.