Fire Code for Software

As a collector of software disasters, I have to ask, why is there no fire code for software?

Published in Embedded Systems Programming

For hints, tricks and ideas about better ways to build embedded systems, subscribe to The Embedded Muse, a free biweekly e-newsletter. No hype, just down to earth embedded talk. 23,000 other engineers subscribe. It takes just a few seconds (all we need is your email address, which is shared with absolutely no one) to subscribe to the Embedded Muse.

 Sweeping fires are so unusual that the once dreaded word conflagration sounds quaint to our modern ears. Yet in 19^th century America a city-burning blaze consumed much of a downtown area nearly every year.

 Fire has been mankind's friend and foe since long before Homo sapiens or even Neanderthals existed. Researchers suspect proto-humans domesticated it some 790,000 years ago. No doubt in the early days small tragedies - burns and such - accompanied this new tool. As civilization dawned, and then the industrial revolution drove workers off the farm, closely-packed houses and buildings erupted into conflagration with heartrending frequency.

 In 1835 a fire in lower Manhattan destroyed warehouses and banks, the losses bankrupting essentially every fire insurance company in the city. The same area burned again in 1845.

Half of Charleston, SC burned in 1838.

 During the 1840s fire destroyed parts of Albany, Nantucket, Pittsburg, and St. Louis. The next decade saw Chollicothe, OH, St. Louis (again), Philadelphia and San Francisco consumed by flames. Most of my hometown of Baltimore burned in 1904. San Francisco was hit again during the 1906 earthquake; that fire incinerated 4 square miles and is considered one of the world's worst fires ever.

 Mrs. O' Leary's cow may or may not have started the Great Chicago Fire that took 300 lives in 1871 and left some 90,000 homeless. The largely wooden city had received only 2.5 inches of rain all summer so turned into a raging inferno advancing nearly as fast as people could flee. But Chicago wasn't the only Midwestern dry spot; on the very same day an underreported fire in Peshtigo, WI killed over 1000.

 A year later Boston burned, destroying 8% of all the capital in the state of Massachusetts. 1889 saw the same part of Boston again ablaze.

 Theaters succumbed to the flames with great regularity. Painted scrims, ropes, costumes, and bits of wood all littered the typical stage while a tobacco-smoking audience packed the buildings. In Europe and America 500 fires left theaters in ruins between 1750 and 1877. Some burned more than once: New York's oft-smoldering Bowery Theatre was rebuilt 5 times.

 The historical record sheds little light on city-dwellers' astonishing acceptance for repeated blazes. By the 1860s fireproof buildings were well understood though rarely constructed. Owners refused to pay the slight cost differential. At the time only an architect could build a fireproof building because such a structure used somewhat intricate ironwork which required carefully measured drawings. Few developers then consulted architects, preferring instead to just toss an edifice up using a back-of-the-envelope design.

 Crude sprinklers came into being in the first years of the 19^th century yet it wasn't till 1885 that New York law required their use in theaters. But even those regulations were weak, reading "as the inspector shall direct." Inspectors' wallets fattened as corruptness flourished. People continued to perish in horrific blazes.

 The 1890 invention of the modern sprinkler reduced the cost of a fire to just 7% of that incurred in a building without the devices. As many as 150 theaters had them by 1905.

 Yet as recently as 1980 87 people died and 679 were injured in the MGM Grand Hotel fire in Las Vegas. Though fire marshals had insisted that sprinklers be installed in the casino and hotel, local law didn't require them. The building's owners refused to fork over the $192,000 needed. They eventually paid out $223 million in legal settlements.

 The Las Vegas law was changed the following year.

 Fire codes evolved in a sporadic fashion. Before the Civil War only tenements in New York were required to have any level of fireproofing. But the New York Times made a ruckus over an 1860 tenement fire that eventually helped change the law to mandate fire escapes for some - not many - buildings.

 A fire at New York's Conway's Theatre in 1876 killed nearly 300 people and led to a more comprehensive building code in 1885. 13 years after the Great Fire, Chicago finally adopted the first of many new codes.

 This legislation by catastrophe wasn't proactive enough to insure the public safety. Consider the 1903 Iroquois theater fire in, of course, Chicago. Shortly before it opened, Captain Patrick Jennings of the local fire department made a routine inspection and found grave code violations. There were no sprinklers, no exit signs, no fire alarms. Standpipes weren't connected. Yet officials on the take allowed the theater to open.

 A month after the first performance 600 people were killed in a fast moving fire. All of the doors were designed to open inwards. Hundreds died in the crush at the exits.

 Actor Eddie Foy saved uncounted lives as he calmed the crowd from the stage. Ironically, he and Mrs. O' Leary had been neighbors; as a teenager he barely escaped the 1871 fire.

 Afterwards a commission found fault with all parties, including the fire department: "They seemed to be under the impression that they were required only to fight flames and appeared surprised that their department was expected by the public to take every precaution to prevent fire from starting."

 All criminal cases against the owners, builders and (vastly corrupt) officials failed on technicalities. The judge missed a chance for a sagacious ruling when he found that fire codes, with no basis in the State's constitution, were illegal. 30 civil suits were settled for $750 each. All others were dismissed for arcane reasons of legal minutia.  

 Carl Prinzler had tickets for the performance but the press of business kept him away. He was so upset at the needless loss of life that he worked with Henry DuPont to invent the panic bar lock now almost universally used on doors in public spaces.

 Fast forward 83 years. Dateline San Juan, 1986. 97 died in a blaze at the coincidently-named DuPont Plaza hotel; 55 of those were found in a mass at an inward opening door. In 1981 48 people were lost in a Dublin disco fire because the Prinzler/DuPont panic bars were chained shut. In 1942 at Boston's Coconut Grove nightclub 492 were killed in yet another fire, 100 of those were found piled up in front of inward opening doors. Others died constrained by chained panic bars.

 Many jurisdictions did learn important lessons from the Iroquois disaster but took too long to implement changes. Schools, for instance, modified buildings to speed escape and started holding fire drills. Yet 5 years after Iroquois a fire in Cleveland took the lives of 171 children and two teachers. The exit doors? They opened inwards.

 Changes to fire codes came slowly and enforcement lagged. But the power of the press and public outrage should never be underestimated. The 1911 fire at New York's Triangle Shirtwaist Company was a seminal event in the history of codes. Flames swept through the company's facility on the 8^th, 9^th and 10^th floors. Ladders weren't tall enough and the fire department couldn't fight it from the ground. 141 workers were killed; bodies plummeting to the ground eerily presaged 9-11.

 But at this point in American history reform groups had taken up the cause of worker protections. Lawmakers saw the issue as good politics. Demonstrations, editorials and activism in this worker-friendly political environment led to many fire code changes.

 Though you'd think insurance companies would work for safer buildings they had little interest in reducing fires or mortality. CEOs simply increased premiums to cover escalating losses. In the late 1800s mill owners struggling to contain costs established the Associated Factory Mutual Fire Insurance Companies, an amalgamated non-profit owned by the policyholders. It offered far lower rates for mills made to a standard, much safer, design.

The AFM created the National Board of Fire Underwriters to investigate fires and recommend better construction practices and designs. 1905 saw the first release of their Building Code. 6,700 copies of the first edition were distributed. Never static, it evolved as more was learned. Amendments made to the code after the Triangle fire, for instance, improved mechanisms to help people egress a burning building.

 MIT-trained electrician William Merrill convinced other insurance companies to form a lab to analyze the causes of electrical fires. Incorporated in 1901 as the Underwriters' Laboratories, UL still sets safety standards and certifies products.

The Pattern

Our response to fires, collapsing buildings and the threats from other perils of industrialized life all seem to follow a similar pattern. At first there's an uneasy truce with the hazard. Inventors then create technologies to mitigate the problem, such as fire extinguishers and sprinklers. Sporadic but ineffective regulation starts to appear. Trade groups scientifically study the threat and learn reasonable responses. The press weighs in, as pundits castigate corrupt officials or investigative reporters seek a journalistic scoop. Finally governments legislate standards. Always far from perfect, they do grow to accommodate better understanding of the problem.

 Though computer programs aren't as yet as dangerous as fire, flaws can destroy businesses, elections and even kill. Car brakes are increasingly electronic, steering is headed that way. Software errors in radiotherapy devices continue to maim and take lives. Bad code has been implicated in a number of deadly aircraft incidents. NIST claims the cost of bugs runs some $60b/year in the USA alone.

 Why is there no fire code for software?

 Today the feds mandate standards for some firmware. But take a gander at the Federal Election Commission or FDA rules. The regulations are loose and woefully inadequate. Firmware is at a point in time metaphorically equivalent to the fire industry in 1860. We have sporadic but ineffective regulation. The press occasionally warms to a software crisis but by and large there's little furor over the state of the art.

 Rest assured there will be a fire code for software. As more life- and mission-critical applications appear, as firmware dominates every aspect of our lives, when a bug causes some horrible disaster, the public will no longer tolerate errors and crashes. Our representatives will see the issue as good politics.

 Just as certain software technologies lead to better code (for instance, C code is generally at least an order of magnitude buggier than that written in Ada), the technology of fireproofing was well understood long before ordinances required their use. The will to employ these techniques lagged, as they do for software today.

 There's a lot of snake oil pedaled for miracle software cures. Common sense isn't one of them. I visited a CMM level 5 company recently (the highest level of certification, one that costs megabucks and many years to achieve) and found most of the engineers had never heard of peer reviews. These are required at level 3 and above. Clearly the leaders of this group were perverting what is a fairly reasonable, though very heavyweight, approach to software engineering. Such behavior stinks of criminal negligence. It's like bribing the fire marshal.

 I quoted the Iroquois fire's report earlier. Here's that sentence again, with a few parallels to our business in parenthesis: "They (the software community) seemed to be under the impression that they were required only to fight flames (bugs) and appeared surprised that their department was expected by the public to take every precaution (inspections, careful design, encapsulation, and so much more) to prevent fire (errors) from starting."

Douglas Adams said "Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." After 790,000 years of firefighting we have finally learned that fire is, well, kind of dangerous and we'd better construct buildings appropriately.

 I collect software disasters, and have files bulging with examples that all show similar patterns. Inadequate testing, uninspected code, shortcutting the design phase, lousy exception handlers and insane schedules are probably responsible for 80% of the crashes. We all know these things, yet seem unable to benefit from this knowledge. I hope it doesn't take us 790,000 years to institute better procedures and processes for building great firmware.

 Do you want fire codes for software? The techie in me screams "never!" But perhaps that's the wrong question. Instead ask "do I want conflagrations? Software disasters, people killed or maimed by my code, systems inoperable, customers angry?" No software engineering methodology will solve all of our woes. But continuing to adhere to ad hoc, chaotic processes guarantees we'll continue to ship buggy code late.

 When researching this article a firefighter left me with this chilling thought: "I actually find bad software even more dangerous than fire, as people are already afraid of fire, but trust all software."