Originally posted on the XeroCrypt Blog
Packet inspection is something we’ll read about a lot, especially with the Communications Data Bill going through at the moment, and other stuff. It’s directly related to the how of surveillance, traffic management and sometimes censorship. The technology for intercepting Internet traffic and scanning content is commercially available, but who is using it, and how is it being used? As it happens, Deep Packet Inspection (DPI) is deployed widely enough that there’s a good chance everything going over the Internet unencrypted is being read as it crosses the public Internet.
An Overview of Packet Inspection
First it’s important to recognise there’s a difference between packet inspection and Deep Packet Inspection (DPI).
Invented around the mid-1990s, packet inspection was originally for use in a stateful firewall/IDS setup, which is useful where applications might change the ports they’re communicating on, or where someone might attempt to tunnel malicious traffic through a standard port – things a basic port blocking firewall isn’t good at.
This looks at protocol headers and port numbers to determine what services are communicating and how, which in practice means looking for specific bits that represent port numbers, synchronisation, flow control, etc. in a bitstream that makes up the TCP packet, so it’s fairly easy to build that function into a programmable IC. In fact, it shouldn’t be too difficult for an electronics enthusiast with the right equipment and a little programming skill to build something that does this. Even some entry-level routers with just 2-4 MB memory include basic packet inspection capabilities as a Quality of Service (QoS) feature for managing traffic on a local network, although this can also be done by Ethernet port or MAC address.
Some years later the technology was developed enough to scan the TCP payload in real-time with negligable latency, and so we have Deep Packet Inspection that looks at the content of whatever’s being communicated. Although this provides some additional security and traffic management benefits, this is where we quickly run into problems with surveillance and censorship, as Nate Anderson explains in his detailed article:
‘Looking this closely into packets can raise privacy concerns: can DPI equipment peek inside all of these packets and assemble them into a legible record of your e-mails, web browsing, VoIP calls, and passwords? Well, yes, it can. In fact, that’s exactly what companies like Narus use the technology to do, and they make a living out of selling such gear to the Saudi Arabian government, among many others.’
The capabilities of a DPI setup depend largely on the implementation, in particular the location where it’s installed, how much traffic goes through it, and whether the equipment’s used for stripping and warehousing certain packets in a database. For example, an intelligence agency or ‘behavioural advertising’ firm might strip out traffic containing keywords and store it for later analysis.
DPI vendors seem reluctant to reveal much about how their products perform in the real-world, such as the data rates they can handle and their effects on latency. To actually do this in real-time without substantiallly affecting latency, it could only realistically be used for comparing payloads against signatures, or determine the general behaviour of traffic through statistical analysis, but again this could still be used in deciding which packets to archive for later analysis. It’s likely those functions must somehow be built into the hardware itself.
DPI’s Use in Targeted Advertising
DPI has already been used against tens of thousands of Internet users in the UK several years ago, and it was actually done by a ‘behavioural advertising’ company called Phorm. Unlike conventional methods of targeted advertising, Phorm gathered information about peoples’ browsing activities by actually scanning the content of whatever web pages they were visiting, using some DPI equipment installed at the service providers’ exchanges and a cookie that linked the collected data to peoples’ browsers. In other words, Phorm was intercepting their private communications and probably breaking a few laws in doing so.
Several ISPs, notably BT, helped Phorm to secretly trial this without informing their customers, and it was discovered when someone noticed their connection was being redirected. There was enough outrage and campaigning against this over the next few years that the ISPs eventually cut their ties with Phorm. In some sense this might have been good thing, as it made the public more aware of the existence of DPI and lessons were learned, hopefully among those who towed the ‘nothing to hide, nothing to fear’ line.
The Legal Stuff
So, was Phorm acting illegally? Could it have been prosecuted? Two laws relevant here are the Data Protection Act and the Regulation of Investigatory Powers Act 2000. Generally the Data Protection Act prohibits the collection of personal data without the data subject’s consent, unless it’s absolutely necessary and in the interests of the public or those of the data subject. Although there are many exceptions to this, none of them seem relevant to what Phorm was doing. Companies like it are in the business of selling that data to other third-parties, so there’s another possible violation.
RIPA is less ambiguous, stating that it’s an offence to intercept telecommunications ‘intentionally’ and without ‘consent’ from the sender or recipient, unless a court order of some sort was involved. My interpretation is that Phorm would have needed the prior consent of the ISPs’ customers, or to gain the consent from the owners of every server those customers were in communication with. That consent might have been given unknowingly at some point, perhaps by signing something their ISPs dropped through the letter box, or by entering a competition without reading the small print. An opt-out policy certainly wouldn’t have made it legal.
Another (perhaps the most common) application of packet inspection and DPI is traffic management, and I’m still undecided on the issues of that in relation to privacy and net neutrality. It’s a short fix for a problem that’s going to get much worse – there’s a real limit to the amount of data the infrastructure can carry, and it would be unfair if people couldn’t access critical services like email, Twitter and MySpace because most the bandwidth was taken up by certain ‘consumers’ expecting an unlimited bandwidth service with stupidly high definition media. Anyone who’s been reading the Networking+ magazine will also know there’s also a colossal amount of other corporate WAN-related crap going across those wires that’s defined as ‘critical’, and the peak usage times (somewhere between 5pm and 8pm) are probably the result of an overlap between that and non-corporate traffic.
This is the basic reasoning behind traffic management by ISPs with more customers than they can support, why ‘unlimited’ services are a myth, and why net neutrality, however desirable, might be unacheivable without some feats of engineering.
Encryption is Important
Today it’s no secret that commercial DPI equipment is being commonly used for a variety of reasons, whether it’s for detecting copyright infringment, managing Internet traffic, ‘behavioural advertising’ or surveillance. Where DPI has become a security risk, countermeasures were quickly used. I could argue that anything going over the Internet unencrypted is fair game when it comes to interception, because it’s out there on the public Internet for anyone with the resources to capture. People have kind of accepted that, and the Internet is adapting to this problem – 15 years ago hardly anyone was using encryption, and today the major sites are accessible over HTTPS/SSL. I can imagine encrypted connections and IPSEC will become standard for any Internet exchange in the next few years, which in turn will make effective DPI-based traffic management much harder to achieve. We’ll certainly have a far more secure Internet, but the effects of making legitimate DPI ineffective remains to be seen.