Fog Creek Software
Discussion Board




Liability, Accidents and Security

In the latest crypto-gram ( http://www.schneier.com/crypto-gram-0309.html) Bruce Schneier dives into the difference between safety and security in design.

"Some years ago computer-security researcher Ross Anderson described the difference as Murphy vs. Satan. Defending against accidents, he said, means designing and engineering in a world ruled by Murphy's Law.  Things go wrong because, well, because things go wrong.  When you're designing for safety, you're designing for a world where random faults occur.  You're designing a bridge that will not collapse if there's an earthquake, bed sheets that won't burst into flames if there's a fire, computer systems that will still work -- or at least fail gracefully -- in a power blackout.  Sometimes you're designing for large-scale events -- tornadoes, earthquakes, and other natural disasters -- and sometimes you're designing for individual events: someone slipping on the bathroom floor, a child sticking a fork into something (accidental from the parent's point of view, even though the child may have done it on purpose), a tree falling on a building's roof.

Security is different.  In addition to worrying about accidents, you  also have to think about nonrandom events.  Defending against attacks means engineering in a world ruled by Satan's Law.  Things go wrong because there is a malicious and intelligent adversary trying to force things to go wrong, at the very worst time, with the very worst results.  The differences between attacks and accidents are intent, intelligence, and control."

Our computing infrastructure is a complex, interconnected system. All of these systems we operate were originally designed and operated with an "accident" mindset. Over the years this has changed into a security framework. This is met with stiff resistance by the users, who do not wish to give conveniences, power and privacy even in the face of what for all intends and purposes is a digital war raging across the system.

Our civilian infrastructure is a complex, interconnected system. It too was designed, and is operated with an accident mindset. Here also, any changes proposed to transform towards a security framework is met with stiff resistance, since also here we are not prepared to give up convenience, power and privacy even in light of a changing sociological climate.

There is an understandable whish in every one of us to keep the luxuries affordable in an accident world and demand they are upheld in the security world. However strong this desire, there is a reality that says some sacrifices are inevitable. We get to choose whether those sacrifices will be in terms of less convenience, power or privacy, or in terms of casualties. This will always be a balancing act, where only fools call out for polarized positions.

As some risks are inevitable, who should be liable when things go wrong? Should a tire manufacturer be held liable for accidents due to faults in the tire design? Should the liability end when the tire was shot out by a malicious individual with the intent of crashing the car? What if the tire was meant for operation in a war zone? Should the tire manufacturer also warn against driving in certain inner city boroughs? Should the tire manufacturer prevent the driver from steering the car into unsafe areas? Is a manual override on such a system allowable? What if five years form now there are terrorists at every corner trying to shout out the tires of random cars? Should tire manufacturers be forced to adapt? Would they still be liable for old, pre-ubiquitous terrorism tires not withstanding gunshots?

In our computer systems we face the same questions. We went from an accident world into full cyber war. How much can manufacturers be held liable? Where does liability end? What about the systems from a previous era? What should we be prepared to give up?

Just me (Sir to you)
Tuesday, September 16, 2003

*poop*

Johnny Bravo
Tuesday, September 16, 2003

Well, at least this got one reaction.
I'll just assume everybody groked the full complexity of the subject and wisely abstained from idle commentary ;-).

Just me (Sir to you)
Wednesday, September 17, 2003

You forgot the usual triggers: "outsourcing to India", Microsoft monopoly", "Open Source Movement", "IDE".

Johnny Bravo
Wednesday, September 17, 2003

*  Recent Topics

*  Fog Creek Home