How Safe Are Our Roads?

What does ten million lines of code look like? This is such a large number that it’s hard to conceptualise. If printed on paper, it would occupy nearly 100 boxes each containing 2000 sheets of paper. If it was punched on paper tape, the paper tape would be over 600 miles long. Another way of conceptualising this much code is to consider cursorily examining it, spending perhaps 3 seconds per line. That would take over 5 person years, and using typical industry rates of code production, it would take several thousand person years of effort to write this much code.

So what’s special about this figure of 10,000,000? This is reportedly the number of lines of embedded code on a Chevy Volt Automobile, more than the 8,000,000 lines on a Boeing 787. When it comes to planes, we worry a lot about safety and security. Code is written to very exacting standards and has to pass through a rigorous certification process (DO-178B). It’s not perfect, but it’s remarkably reliable — no lives have been lost on commercial aircraft due to software coding bugs.

Turning to cars, we first observe that by several orders of magnitude more people die in car accidents than in plane accidents, so one would expect to find at least the same level of care. If indeed you do expect that, you will be disappointed to learn that by comparison, the standards are very light and huge amounts of automobile software is produced with almost no formal certification.

So should we worry? I think so! The software on cars is complex and getting more so all the time. We really don’t know how bad the situation is, because a shroud of secrecy covers the details of automotive software. We already know of safety-recalls that address software defects. Some people, even those in the software industry, routinely assume that large programs will be full of bugs. I even heard an eminent lawyer from Yale arguing for relaxed product safety standards for software on the grounds that it was unreasonable to expect software to be reliable. People who have watched their PC crash and burn numerous times get used to this lack of reliability and every week we can read of ‘glitches’ (the very use of this word implies a kind of inevitability) that have horrible consequences. If you really believe that all large software systems are inherently unreliable then I advise you not to fly, since whenever you fly you are at the mercy of very complex software and a bug in this flight control software could easily cause the plane to crash.

But as mentioned earlier, in practice we can do amazingly well if we use the right techniques. You might wonder why these techniques are not more widely adopted. That’s a good question, which is difficult to answer. Partly it’s a matter of lack of education, we teach our students how to write simple web applications, but we don’t teach them the techniques for writing reliable code (see “Computer Science Education: Where Are the Software Engineers of Tomorrow?” Dewar and Schonberg, Crosstalk, Jan 2008).

Partly it is the misguided impression that it’s too expensive to write reliable code. To the latter, we have to ask the question, how much do you think it will cost if a preventable software bug in embedded automobile code can be shown to have caused fatal accidents? It’s not going to be easy to convince the software industry generally to adopt techniques for producing more reliable software, but most certainly, cars present the same kind of safety concerns as planes, so that’s a good place to start. And in the automotive domain, in contrast with the aircraft industry for example, it’s partly the market pressures that encourage rapid change/new functionality as a sales advantage. Especially with the increasing dependence on software for critical functions, there is the risk that reliability and perhaps safety may be compromised. This is an important difference; the ‘safety first’ culture that has been a defining attribute of the avionics community will need to be embraced by the automotive industry, with all its ramifications.

These days we don’t only have to worry about safety, but also security. With millions of lines of code aboard a car, are there weaknesses that would let bad guys do bad things? There are plenty of access points at this stage. All cars have umbilical connections for diagnostics and also support Bluetooth and WiFi, as well as allowing media to be loaded into the infotainment system. The answer is that, yes, there are weaknesses and they are very scary. Reports can be found online of some amazing successful attacks, including a CD that appears to play music when loaded into the CD player, but is silently infecting the software of the car with viruses that can, for example, shut off the car’s lights, lock its doors, kill the engine and release or slam on the brakes. Hard to believe — almost the stuff of science fiction — but these are real experiments with real cars.

Recently, DARPA launched an initiative called ‘High-Assurance Cyber Military Systems’ (HACMS), that is intended to address the issue of how to make systems such as the ones found on commercial automobiles safe. At a recent talk on this initiative (off-camera, because the speaker did not want alarming video material going online), more amazing and very worrisome examples were given, after which a lot of people felt a whole less secure in climbing into their vehicles. One example was another case of a CD which when loaded played music while also disabling the steering wheel. Perhaps you thought your steering wheel was somehow mechanically attached to the wheels? If so, you thought wrong! Increasingly the computer software is interposed between the controls and the mechanical systems. You are driving essentially a computer simulation, and the computer software is supposed to convert the results of this simulated control to corresponding actions of the real mechanical systems.

So what’s to be done? It’s possible to write reliable software using well established technology and we are improving our capabilities in that area. When I get onto a modern aircraft, I feel safe and secure, despite the huge amounts of software aboard, because I know the rigorous process that has gone into the development and verification of this software. Yes, it’s not 100% guaranteed, but in practice the avionics industry’s processes and safety culture have effectively minimised the risk of software bugs causing catastrophic failure. When it comes to security, we are increasingly learning how to use formal mathematical techniques to prove that software meets security requirements.

The bad news is that so far, we are nowhere near requiring the same level of scrutiny for cars as we do for planes, and even trains get much more care when it comes to software. A recent project built a bus that follows strips in the road. It has a lot of software on board and when it was first delivered it transpired that the strips in the road make it officially a train rather than a bus. Suddenly the software was subjected to rigorous certification requirements that had not been anticipated. If the bus had been controlled by cameras aboard, it would not have been subjected to the same standards. That makes no sense of course, but that’s the way things are now.

The good news is that automobile manufacturers are showing increased concern with the problem and are busy investigating how to improve the software safety and reliability of cars. Toyota recently revealed it now uses SPARK; a language and toolset that provides mathematical techniques to prove that the software does what it should and is free of security vulnerabilities. These developments can’t happen too soon. Cars already have huge amounts of critical software, but the future, with its vision of fully autonomous vehicles, will raise the complexity, safety and security stakes to previously unheard of levels.