Comments on: Pakistan brings down YouTube

By: Cyde Weys

Cyde Weys — Mon, 25 Feb 2008 01:33:28 +0000

Here's some more history about these kinds of routing mistakes. They've happened a lot more often than you would like to think. This most recent one in Pakistan appears to have been a mistake like the others, but it wasn't a pure mistake; someone was setting up censorship and messed that up.

By: William (green)

William (green) — Sun, 24 Feb 2008 23:08:04 +0000

You think it’ll take a number of these, or maybe one that drops the Google main page or something?

By: Cyde Weys

Cyde Weys — Sun, 24 Feb 2008 22:54:31 +0000

I remember studying BGP in Networking class at UMD and being blown away by the complete lack of security. Back in 1997, during the AS7007, the whole Internet was effectively brought down for a little while by a misconfigured router (although it could’ve easily been done by someone malicious). Various security layers have since been added on top of BGP, including various filters on who is allowed to define routes. But obviously not enough has been done if a huge site like YouTube can be brought down by a random router in Pakistan.

Unfortunately, I don’t think this incident will prove to be big enough to really precipitate any further major changes like the AS7007 incident did. Although this did hit Google, and Google is huge and influential, so who knows?

By: drinian

drinian — Sun, 24 Feb 2008 22:33:39 +0000

Wow, I didn’t realize that there was so much trust involved in that part of the routing system. I hate to imagine what could happen to a small, personal site…

By: Greg Maxwell

Greg Maxwell — Sun, 24 Feb 2008 21:58:49 +0000

William, There is absolute certainty that Pakistan telecom is the guilty party here. They were advertising youtube’s IP space. That isn’t a matter up for debate. I pasted enough instructions that you could have validated it for yourself while it was happening, … and soon enough the historical routing data will show up on the various archive sites.

It is unlikely that they caused the global outage intentionally: Anyone in a position to perform this act intentionally would have known that it wouldn’t last long.

By: William (green)

William (green) — Sun, 24 Feb 2008 21:40:53 +0000

I wouldn’t necessarily say that Pakistan is the one that cut them, but who’s to say that it’s not somebody else? It seems conceivable that this could have been wrought by the hands of whoever that was. I have no supporting evidence, but I don’t it should be entirely dismissed.

By: Greg Maxwell

Greg Maxwell — Sun, 24 Feb 2008 21:19:04 +0000

It just came back up globally a few minutes ago. It’s been working for perhaps an hour or so on little islands of the internet due to my above mentioned efforts by youtube to work around the issue. Now either PCCW is filtering out the bad route, or Pakistan telecom has stopped announcing it.

If it was PCCW that started filtering it, their sudden attention to the matter might have been brought on by the several networks that dropped peering sessions to them… or that could just be coincidental. :)

The outage was 2hr 15 mins approximately.

By: Cyde Weys

Cyde Weys — Sun, 24 Feb 2008 21:07:21 +0000

I would say it’s very unlikely. Unless you think Pakistan was the one behind the cutting of those cables.

By: William (green)

William (green) — Sun, 24 Feb 2008 21:05:44 +0000

Another thing I’m curious about is if this has any connection to the cable cuttings from a while back. Different place, but a similar vein, you know?

By: William (green)

William (green) — Sun, 24 Feb 2008 20:49:25 +0000

I wonder how long that outage lasted. I can’t find any useful articles on it, just random blog posts.
What’s to stop anybody from doing this at any given time, to any other site, I wonder?

By: Greg Maxwell

Greg Maxwell — Sun, 24 Feb 2008 20:46:34 +0000

It may never be knowable if this was an intentional act or an accidental one… Normally we would presume accidental, but since the Pakistan government and by extension Pakistan telecom are already acting maliciously by intentionally blocking youtube within their own country I don’t know that we can invoke Hanlon’s razor quite so quickly.

A little technical background: The internet is built on trust. Well, it’s buffered trust, but trust none the less. If you configure your routers to claim to have a route to a particular destination other routers will believe it unless they are configured otherwise. If you are a small network with competent upstreams they will be configured to filter, and they will verify your claims of address ownership before sending traffic to you. Apparently if you are Pakistan telecom (ASN 17557) and your upstream is PCCW .

Youtube is trying to get their address space back by announcing more specific routes, /25 routes to beat the /24 that Pakistan telecom is announcing. However, the same kinds of filters which should have prevented the initial hijacking are very frequently configured to drop any route more specific than /24, so Youtube’s steal-back effort is having little effect.

Some people have been spreading the claim that this outage is DNS related. This is not true. Pakistan telecom is hijacking youtube’s address space, they have managed to configured their routers to claim that youtube is on their network, then they drop the packets.

This is simple to demonstrate:

First we look up youtube’s IP addresses:
[gmaxwell@bessel ~]$ host youtube.com
youtube.com has address 208.65.153.251
youtube.com has address 208.65.153.253
youtube.com has address 208.65.153.238

Then we log into a route server and see who is announcing them:
[gmaxwell@bessel ~]$ telnet route-views.oregon-ix.net
Trying 128.223.51.103…
Connected to route-views.oregon-ix.net.
Escape character is ‘^]’.

**********************************************************************

Oregon Exchange BGP Route Viewer
route-views.oregon-ix.net / route-views.routeviews.org

route views data is archived on http://archive.routeviews.org
…

**********************************************************************

route-views.oregon-ix.net>show ip bgp 208.65.153.251
BGP routing table entry for 208.65.153.0/24, version 1628976
Paths: (39 available, best #39, table Default-IP-Routing-Table)
Not advertised to any peer
5459 3491 17557
195.66.232.239 from 195.66.232.239 (195.66.232.239)
Origin IGP, localpref 100, valid, external
Community: 5459:3 5459:60
16150 3491 17557
217.75.96.60 from 217.75.96.60 (217.75.96.60)
Origin IGP, metric 0, localpref 100, valid, external
Community: 16150:63392 16150:65232 16150:65320
2905 701 3491 17557
…

gmaxwell@bessel ~]$ Then we look up the ASN which is announcing the route, ASN 17557
whois -h WHOIS.APNIC.NET AS17557
aut-num: AS17557
as-name: PKTELECOM-AS-AP
descr: Pakistan Telecom
descr: ITI Region PTCL
country: PK

Youtube is now trying to fight back:

route-views.oregon-ix.net>show ip bgp 208.65.153.128/25
route-views.oregon-ix.net>show ip bgp 208.65.153.128/25
BGP routing table entry for 208.65.153.128/25, version 1636911
Paths: (5 available, best #5, table Default-IP-Routing-Table)
Not advertised to any peer
7500 2516 3549 36561
202.249.2.86 from 202.249.2.86 (203.178.133.115)
Origin IGP, localpref 100, valid, external
7660 2516 3549 36561
203.181.248.168 from 203.181.248.168 (203.181.248.168)
Origin IGP, localpref 100, valid, external
Community: 2516:1030
…

However, /25 routes are supposed to be rejected by ISP’s filters, so YouTube’s attempted fight back is far less effective than Pakistan telecom’s initial hijacking.

The correct solution here is for Pakistan telecom’s upstream to filter their announcement. It seems that Pakistan telecom has several other upstreams through which the youtube route is not being redistributed.

This sort of misconfiguration happens from time to time… about a year ago Cogent managed to hijack the IP address space used to run Wikipedia. But these kinds of events tend to be short lived. In cases of malicious hijacking the guilty network usually gets their service turned off by their upstream(s).

By: In Toledo

In Toledo — Sun, 24 Feb 2008 20:28:46 +0000

“Googlebomb”? I’d say – bombs away!!! Lets teach the Mufti’s a lesson Internet style!