Meter-Reading Between The Lines

The DECC (Department of Energy and Climate Change) has just invited bids to run the UK’s biggest ever network project. It will connect some 53 million smart meters in order to monitor and control the delivery of electricity, water, gas — and potentially other services — to around 30 million homes and businesses across the nation. The target completion date for this £11.5bn project is 2019.

The idea of a smart grid for utilities has been around for a while; it allows replacing teams of meter readers and field engineers with a network that feeds back readings from field equipment to a central controller and also reports on faults, or even allows some control of the supply itself. In the USA, for example, there is the suggestion that the power supplier should be able to selectively switch off major energy-consuming systems such as air conditioning in order to reduce load and protect the regional power supply against failure. The move to ‘green energy’ makes it all the more important to respond quickly to changing consumer demand and faults: a smart grid does that and allows greater efficiency too.

But this new project has some special features, apart from its sheer scale. For a start it is addressing all the public utility meters in one project, and it also seems to be proposing an all-wireless solution. Although there is an obvious argument for using electricity power lines as the backhaul medium from an electricity meter, making the entire smart meter population 4G-enabled means that any meter can be located where most convenient, with no need for additional physical connections by a host of local third party suppliers.

In terms of efficiency and optimisation, it’s a wonderful idea. But US Defence Secretary, Leon Panetta, sounded a note of warning when he said that today’s attackers ‘are targeting the computer control systems that operate chemical, electricity and water plants, and those that guide transportation throughout the country’. Speaking at the annual awards dinner of Business Executives for National Security in October 2012, he concluded that: “A cyber attack perpetrated by nation states or violent extremist groups could be as destructive as the terrorist attack of 9/11.”

So it will be necessary not just to secure this network, but to be able to test its security to the limits of every new state-of-the-art cyber attack strategy as it arises. For we are not just talking about naughty kid hackers now, but the possibility of one nation intent on crippling another’s power supply as an act of war. Especially in times of political tension, there is another equal danger: that some sort of system overload could result in a chance crash of the network, and be mistaken for an actual act of aggression and lead to war.

The real problem, then, is not just to test the network’s security but also its functionality and performance under every (un)likely type of loading. This level of testing has become almost routine for major telecoms players and the specialist network test companies that serve them, but public utility companies may not have the same experience of large scale testing of critical computer networks. This article outlines some of the issues they will need to address.

Defining The Network

In theory the network linking smart meters could be totally independent of any public service, running on its own dedicated network across the nation, but this would add enormous extra expense. In this case, the network will be running across 4G mobile phone networks, likely sharing the public telecommunications infrastructure and connecting to the Internet.

It might seem risky using the public Internet rather than a quarantined private network, but that is a likely option because it allows far greater flexibility for the future. A control system using the Internet allows greater flexibility for customers to switch between providers in a competitive market of the sort currently deemed desirable. So the smart grid has to provide direct and immediate access for the new provider of choice, and a public network makes that easy.

Enterprise IT teams have already been addressing a similar convergence problem for over a decade. There, previously independent systems such as fire and burglar alarms, smoke detection, and industrial control systems — systems that used to be either manually operated or linked by their own separate control networks — are now increasingly all connected across the same enterprise IT network.

The IT industry has accepted this as a viable and efficient solution and has developed appropriate measures to safely tunnel the critical signals through relatively open networks. They call this growing challenge ‘network permeability’: whereas the earliest networks consisted of isolated computers linked by fixed cables, today’s networks have to cope with mobile staff plugging in their laptops anywhere on the network, also with wireless access from smartphones and with data transfer via USB memory sticks.

So, instead of sealing the network from public services like the Internet, the main focus has been on ways to run secure services over such public networks. A host of solutions are available, including firewalls, intruder detection systems and deep packet inspection devices to examine all the traffic on a network and look for incoming or outgoing anomalies.

Such forms of protection will be vital for a nation’s critical infrastructure where so much is at stake. Don’t be fooled by the argument that public utilities rely on highly customised systems, with no two alike, so that hacking them would be impossible without insider knowledge. Hackers have long known how to get such knowledge and according to Leon Panetta: “We know of specific instances where intruders have successfully gained access to these control systems. We also know they are seeking to create advanced tools to attack these systems and cause panic, destruction, and even the loss of life.”

However, as everyone with experience in IT networks knows, every addition to a network, however necessary, increases its complexity. And that makes it harder to predict.

The Testing Imperative

So the real challenge for any national smart grid is this: how can we secure a highly complex system? Unless the engineers managing the grid have long experience with complex networks — the sort of experience gained over decades by telecommunications and IT network engineers — it is easy underestimate this challenge. Security vendors will assure the utilities setting up smart grids that their products will make the network utterly secure and provide good arguments to justify these claims.

It all sounds very good and secure on paper, but consider the following ‘thought experiment’: you have won a prize to be the first civilian to fly to the moon it’s exciting, but also scary. So the NASA team spends a whole day showing you the blueprint of the spacecraft and explaining all the fail safe and security provisions built in. That is encouraging, but still pretty scary. Now imagine that you are also told that the same space ship has flown hundreds of moon missions and never once failed. That is far more reassuring, because we humans do have an innate sense of complexity and know that, however smart the design, there is really only one way to prove a complex system, and that is by repeated testing.

And that is how it is with today’s complex IT networks. You design in all the safety and security features that are needed, but you then submit it to rigorous testing under realistic operating conditions as well as extreme loads and attack situations in order to make sure it is secure and also to allow fine-tuning of the network for optimal performance.

This sort of testing will be critically important for the sort of smart metering solutions being planned for our national utility infrastructure. The good news is that there are companies that have long experience in testing IT and telecoms networks, and sophisticated tools are available to facilitate testing of highly complex networks under ‘real world’ conditions.

Step By Step testing Of Critical Networks

The first lesson from years of network testing is that you need two types of test; both functional testing and performance testing.

##IMAGE_3_R##The need for security testing functionality against attacks and system faults has already been discussed, what is less obvious is that a complex network can develop surprising problems under varying loads. A telecoms network, for example, might be able to handle hundreds of gigabits of data per second during file transfers and yet fail at a much lower bandwidth when handling a mix of different types of traffic — such as video and voice over IP. So it is necessary not just to test the network’s greatest data capacity but also to test how it performs under a whole range of realistic traffic scenarios and combinations of traffic.

Functional testing should include three main stages. First the sort of assessment that might be provided by the security system vendors: an experienced eye looking over the existing or planned network for obvious weak points or vulnerabilities, and making sure it is strengthened at those points.

The second stage is to simulate actual attacks under real-world operating conditions. Today’s sophisticated test tools can not only simulate all combinations of normal operating conditions, but also combine these with state-of-the-art malware attacks. Most relevant to a national smart meter grid would be the so-called Denial of Service attacks that could cut off users from the service and cause widespread panic. So today’s most advanced test tools are integrated with a cloud database that is kept up to date with every new attack or virus as they occur, rather than waiting days or weeks for patches to be distributed.

The third stage is to explore further for unknown vulnerabilities, and today’s smart test solutions have the flexibility to allow very detailed testing around the boundaries of normal operation. For example: what happens when a system requires a long pass-code to be input and an operator mistakes a capital ‘O’ for a zero? Does it simply report an error, or does the wrong type of character crash the whole system? The best test solutions allow for “fuzz testing” – testing such variations from normal behaviour to anticipate problems that might accidentally arise during human operation.

Timing Issues

There is a special consideration in the case of a smart metering grid, and that is the question of time synchronisation. The Internet is a ‘good enough’ solution for everyday time keeping — it keeps PC clocks accurate to the second — but it is not precise or sufficiently deterministic to support very critical communication signals. Certain operations on the smart meter grid might be highly time critical to milliseconds rather than seconds; maybe cutting power before a spark can occur in an explosive atmosphere, time synchronisation could be an important test parameter in such a grid.

And then we come to performance testing. Certain signals could be broadcast, or fed back, on a massive scale across a smart meter network that is every bit as dense as the existing Internet it is joining. Might these not crash the entire communications structure?

Today’s networks have to carry many types of traffic — data, video, voice, control signals and more — and a range of different protocols for each. It is not enough to know the maximum bandwidth capacity; how the network behaves under a whole spectrum of different operating conditions is crucial. The right test solution in the hands of an experienced network test engineer will be able to test the network to all its limits, providing clear reports to show where problems could occur.

The ideal would, of course, be to have a network designed and built to handle every possible operating condition and survive any type of attack or local fault that could arise, but this is hardly realistic. The real value of a comprehensive network test report is often that it spells out the system’s limits rather than proving that it is perfect. During a crisis when a certain type of traffic is surging, the grid operators know where the danger point lies, and can take precautionary steps or issue warnings before that point is reached.

In the USA the Department of Homeland Security is working with service providers to lay down certain common standards for testing and security of critical public networks such as the proposed smart meter grid. According to Leon Panetta, however: “Information sharing alone is not sufficient. Working with the business community, we need to develop baseline standards for our most critical private-sector infrastructure — including power plants, water treatment facilities and gas pipelines.”

Meanwhile, in the early stages of planning such a radical new public network, it makes a lot of sense to ask the advice of a specialist network test company; one that has already gained decades of experience in the business of testing the performance and security of the world’s most critical financial, government, medical and corporate networks.