|Jack Ganssle's Blog
This is Jack's outlet for thoughts about designing and programming embedded systems. It's a complement to my bi-weekly newsletter The Embedded Muse. Contact me at email@example.com. I'm an old-timer engineer who still finds the field endlessly fascinating (bio).
|For novel ideas about building embedded systems (both hardware and firmware), join the 40,000+ engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype and no vendor PR. Click here to subscribe.
Problems in Ramping Up Ventilator Production
April 28, 2020
Having brought many products to market I know that production involves a maze of problems that are easily overlooked. Tom Archer wrote the following, and I think it's pretty insightful.
Adding Manufacturing Capacity: Herding Snowflakes, By Thomas Archer
In the current Covid-19 crisis a critical shortage is medical ventilators. Manufacturers are increasing
production within their operations and by transferring technology to other companies. One example
in the national news is Ventec Life Systems Inc., an established ventilator supplier, working with
Increases in ventilator production are possible, perhaps even 100% if raw materials are already in the
chain. Hearts, minds, money and bodies will be in the right places, but multi-digit increases in
domestic production in 30 to 90 days isn't going to happen. Irregularities in the supply chain
The Ventec GM effort is exceptionally difficult because it includes a radical, forty times, increase in
production that will challenge the existing, disparate, fractured supply chain and requires major
The unstated assumption is always that the product design is valid and stable. That assumption will
be tested as issues arise from changes in manufacturing, and end use environments. Ventilators that
worked well in air conditioned hospitals with stable power, trained operators and maintainers will be
stressed by temporary hospitals with generators, where the power and environment is "dirty."
A prescription for exponential trouble is changing the design while transferring manufacturing. New
manufacturers take the transfer event as an opportunity to drive cost and production time out of the
product by changing parts, by changing suppliers. Sometimes proven suppliers simply don't have
the parts and changes are the only option. New parts may meet the old spec, but have subtle,
sometimes disastrous, unpredictable differences.
There are more regulatory requirements and certifications than could be listed, starting with 21 CFR
820, but in the crisis at hand they will be met, waived or expedited to the point they will be driven off
the critical path. With the waivers come risks. There are reasons those requirements are in place and
make sense, the bureaucratic process that evaluates conformance may not.
Certifications may be waived, some of the underlying best practices including manufacturing
readiness reviews, a pilot run, first article inspections and validation of "run at rate" should not. All
are intensive and important, though time consuming. A single recall wipes out all the savings from a
For a product that is required to function 24/7 a 320 hour (10 days) continuous run of 20 machines
under load helps validate conformance. Tough to compress that time. It could be executed in
parallel with early shipments but what happens when they miss the required performance or fail
completely; a common occurrence with new builds?
With the increased production, everything related to manufacturing changes, everything. Designing
the new production system is a combination of art and science. There are nuances of difficulty that
an experienced bench assembler deals with easily, but are problematic with a more aggressive, time
constrained process and newly trained people. There is a time consuming learning curve.
In the new facility, new work instructions, a new quality system, traceability to the component level,
a new structure to the bill of materials, training, different material transport; just-in-time supply to
assembly, are all needed. There will be serious stumbles.
The "gotcha's," with anything involving electronics and software, are rooted in supply chain issues.
The variability in electronic components is unfathomable to the uninitiated. Supposedly identical
electronic components and purchased parts, ordered to the same specification, are like snowflakes; no
two are exactly alike. Most will be close enough to be functional under the usual circumstances.
Some will not and not all circumstances are "usual." Some simply will not meet specifications. There
is no easy remedy; the vendor may be in another country and for all practical purposes out of reach,
perhaps even unknown through many hands. Traditional manufacturing disciplines related to change
control have been abandoned; certifications are meaningless. Vendors miss delivery dates, sometimes
by weeks, sometimes forever.
Well documented problems with counterfeit, non-conforming electronic parts are a constant threat.
High reliability in complex electronics is achieved by eliminating variables. Components are single
sourced, sometimes specified to a single lot or batch, assembled entirely with automation, then tested
and "stressed," at length. When the volumes are large, economies come from scale. Where the
volumes are low, the reliability comes at cost.
To assure reliability, cycling the final product, under a simulated end-use load in an environmental
chamber, for hours, is specified. It is a major task to set up test chambers. In consumer electronics
failure is a warranty charge, in medical and aerospace failure can be deadly.
Any change requires extensive, expensive, time consuming requalification for the medical product
producer. There is always resistance both internally and externally to the burden of requalification;
engineering and manufacturing changes are "obviously safe;" "nothing could go wrong." (See FDA
March, 2020 Guidance Document)
Easily repeatable problems are solvable. Intermittent problems, those most common in the complex
interaction between electronics and embedded software, can be a nightmare that can take anxious,
frustrating weeks to resolve with skills and equipment a manufacture may not have.
Electronics are subject to cumulative damage in out-of-specification conditions such as heat and
power excursions or invisible noise. Every device plugged into a wall socket receives power, but
also can impose damaging voltage upstream. Modern devices shouldn't do that, but through failure
or non-conformance they do.
Manufacturing transitions are transparent but consumed by unknowns and unknowables. Issues are
revealed sequentially, nothing can force them to occur in parallel. Paraphrasing Fred Brooks of
Brook's Law fame, throwing people at the project will delay it. The force of will, of need, cannot
overcome the uncertainty of product complexity and unreliability in the underlying supply chain.
Thomas Archer is a retired engineer with a career in engineering and manufacturing. He has no
direct knowledge of, or involvement with Ventec, General Motors or their relationship. He can be
reached at firstname.lastname@example.org April 4, 2020.
Feel free to email me with comments.
Back to Jack's blog index page.
If you'd like to post a comment without logging in, click in the "Name" box under "Or sign up with Disqus" and click on "I'd rather post as a guest."
Recent blog postings:
- Marvelous Magnetic Machines - A cool book about making motors
- Over-Reliance on GPS - It's a great system but is a single point of failure
- Spies in Our Email - Email abuse from our trusted friends
- A Canticle for Leibowitz - One of my favorite books.
- A 72123 beats per minute heart rate - Is it possible?
- Networking Did Not Start With The IoT! - Despite what the marketing folks claim
- In-Circuit Emulators - Does anyone remember ICEs?
- My GP-8E Computer - About my first (working!) computer
- Humility - On The Death of Expertise and what this means for engineering
- On Checklists - Relying on memory is a fool's errand. Effective people use checklists.
- Why Does Software Cost So Much? - An exploration of this nagging question.
- Is the Future All Linux and Raspberry Pi? - Will we stop slinging bits and diddling registers?
- Will Coronavirus Spell the End of Open Offices - How can we continue to work in these sorts of conditions?
- Problems in Ramping Up Ventilator Production - It's not as easy as some think.
- Lessons from a Failure - what we can learn when a car wash goes wrong.
- Life in the Time of Coronavirus - how are you faring?
- Superintelligence - A review of Nick Bostrom's book on AI.
- A Lack of Forethought - Y2K redux
- How Projects Get Out of Control - Think requirements churn is only for software?
- 2019's Most Important Lesson. The 737 Max disasters should teach us one lesson.
- On Retiring - It's not quite that time, but slowing down makes sense. For me.
- On Discipline - The one thing I think many teams need...
- Data Seems to Have No Value - At least, that's the way people treat it.
- Apollo 11 and Navigation - In 1969 the astronauts used a sextant. Some of us still do.
- Definitions Part 2 - More fun definitions of embedded systems terms.
- Definitions - A list of (funny) definitions of embedded systems terms.
- On Meta-Politics - Where has thoughtful discourse gone?
- Millennials and Tools - It seems that many millennials are unable to fix anything.
- Crappy Tech Journalism - The trade press is suffering from so much cost-cutting that it does a poor job of educating engineers.
- Tech and Us - I worry that our technology is more than our human nature can manage.
- On Cataracts - Cataract surgery isn't as awful as it sounds.
- Can AI Replace Firmware - A thought: instead of writing code, is the future training AIs?
- Customer non-Support - How to tick off your customers in one easy lesson.
- Learn to Code in 3 Weeks! - Firmware is not simply about coding.
- We Shoot For The Moon - a new and interesting book about the Apollo moon program.
- On Expert Witness Work - Expert work is fascinating but can be quite the hassle.
- Married To The Team - Working in a team is a lot like marriage.
- Will We Ever Get Quantum Computers - Despite the hype, some feel quantum computing may never be practical.
- Apollo 11, The Movie - A review of a great new movie.
- Goto Considered Necessary - Edsger Dijkstra recants on his seminal paper
- GPS Will Fail - In April GPS will have its own Y2K problem. Unbelievable.
- LIDAR in Cars - Really? - Maybe there are better ideas.
- Why Did You Become an Engineer? - This is the best career ever.
- Software Process Improvement for Firmware - What goes on in an SPI audit?
- 50 Years of Ham Radio - 2019 marks 50 years of ham radio for me.
- Medical Device Lawsuits - They're on the rise, and firmware is part of the problem.
- A retrospective on 2018 - My marketing data for 2018, including web traffic and TEM information.
- Remembering Circuit Theory - Electronics is fun, and reviewing a textbook is pretty interesting.
- R vs D - Too many of us conflate research and development
- Engineer or Scientist? - Which are you? John Q. Public has a hard time telling the difference.
- A New, Low-Tech, Use for Computers - I never would have imagined this use for computers.
- NASA's Lost Software Engineering Lessons - Lessons learned, lessons lost.
- The Cost of Firmware - A Scary Story! - A hallowean story to terrify.
- A Review of First Man, the Movie - The book was great. The movie? Nope.
- A Review of The Overstory - One of the most remarkable novels I've read in a long time.
- What I Learned About Successful Consulting - Lessons learned about successful consulting.
- Low Power Mischief - Ultra-low power systems are trickier to design than most realize.
- Thoughts on Firmware Seminars - Better Firmware Faster resonates with a lot of people.
- On Evil - The Internet has brought the worst out in many.
- My Toothbrush has Modes - What! A lousy toothbrush has a UI?
- Review of SUNBURST and LUMINARY: An Apollo Memoir - A good book about the LM's code.
- Fun With Transmission Lines - Generating a step with no electronics.
- On N-Version Programming - Can we improve reliability through redundancy? Maybe not.
- On USB v. Bench Scopes - USB scopes are nice, but I'll stick with bench models.