Ofcom report confirms risk of IP address over-blocking

Ofcom’s 2010 report on “Site Blocking” to reduce online copyright infringement concluded that using IP addresses to block infringing sites “carries a significant risk of over-blocking given that it is common practice for multiple discrete sites to share a single IP address” (page 5). They have now published a report commissioned from CMSG that shows the level of IP address sharing in various top level domains and confirms that the risk of over-blocking is indeed high.

Although humans navigate the Internet using names, such as www.example.ac.uk, computers communicate with each other using numeric addresses, traditionally written in the form 127.0.0.1. Conversions between the two are done by a system known as the Domain Name Service (DNS). There’s no requirement in DNS that every Internet name has to have a unique numeric IP address, however the original web protocol, HTTP, did make that assumption. If multiple names translated to the same IP address, then they would all see the same web content. Around 1996 a new header was introduced that allowed many different sites (e.g. example.com, example.co.uk, example.ac.uk) to share the same IP address, but the web server program could now see which name had been used and serve different content to different names. Since IP addresses (at least the still common version 4 ones) are a relatively scarce resource this option became very popular; for companies whose business is to host thousands of websites for their customers, it is essential.

The new report confirms that for the .org, .com and, .net top-level domains - 113 million websites in total - 97% of sites now share an IP address with at least one other site (in other words only 3% of websites do not share their IP address). Excluding IP addresses that are thought to be “domain parks” holding currently unused names reduces this slightly, but the most optimistic estimate is still that 92% of websites share their address with an average of 7.5 sites per address. Comparing with a survey from 2003 shows a decrease in unshared sites from 13% to 3%. Since the number of sites has grown from 20 million to 121 million in that time, I think that means that even the number of unshared sites, not just the percentage, has fallen.

A different method was used to obtain the list of websites in the .uk domain but the percentages are very similar: 97% of .uk websites share an IP address with at least one other, discarding addresses thought to be domain parks reduces this to 94% and there is an average of 8.5 sites per address.

The survey admits that it is likely to underestimate the amount of sharing. First, only websites of the form www.example.com were considered so running sales.example.com on the same IP address would not count as sharing. And since only four top level domains were considered, sharing an IP address with a website in any other top level domain would also have been missed.

I don’t expect this confirmation will come as a surprise to anyone: Ofcom and judges in both European and UK courts have already been alert to the risk of over-blocking. Indeed in the UK cases where blocking orders were made there was an explicit check that the blocked addresses were only used by a single site. However it’s good to now have the figures and trend as a reminder of the problem.