Pakistan brings down YouTube

Through network magic I know not much about, Pakistan has caused YouTube to be inaccessible from the majority of the world. It’s not just that they blocked access to YouTube from within their own country; they did it in a way that isn’t filtered by their upstream ISP, so it affects pretty much everyone else too. What happens now?

Well, this damage is going to be routed around pretty quickly, as Pakistan having the ability to knock off websites is an error that will shortly be corrected. I predict the fallout will be immense though. Censor sites and the world looks down upon you, but do it in a way that (temporarily) removes the rest of the world’s access, and you’re in another circle of hell.

Maybe Pakistan is about to find out what the true meaning of “Googlebomb” is.

Update 1: So after a little more edification, I think I have a better handle on what’s going on. First, read up on the AS7007 incident, because what’s going on now is essentially the same thing. The Border Gateway Protocol that the Internet uses to establish routes prioritizes specific routes over more general routes. A network in Pakistan set up a /24 route, which is about as specific as you can get (/25 and beyond are commonly filtered out), declaring that YouTube was located within their network. Since this was the most specific route, it propagated out across all the routers, and now most of the Internet thinks YouTube is located within that network in Pakistan. Of course, it’s not, and they’re simply dropping all of those packets as part of their censorship. There are two possibilities: a network admin in Pakistan messed up and accidentally implemented their censorship in a way that affected the whole world, or this was done maliciously. If the latter is the case, well, the Pakistanis may soon be discovering that they need the Internet more than the Internet needs them.

Update 2: As of around 16:00 EST, YouTube is back up and working. Either PCCW filtered the bad route or the Pakistanis stopped sending it. And do check out Greg’s comments below; he’s the networking expert.

Update 3 (Feb 25): Here’s the best technical synopsis of what happened to YouTube yet.

Update 4: This animated visualization provides the clearest view of the hijacking yet. Watch all of the routes divert to Pakistan Telecom within a matter of minutes, and then two hours later, revert just as quickly back to YouTube.

Update 5: Hey look, MSNBC has picked up the story! I wouldn’t have guessed that this would make mainstream media. Or that they would get the technical details right. But it looks like they talked to the knowledgeable folks at Renesys, who I linked to in Update 3.


12 Responses to “Pakistan brings down YouTube”

  1. In Toledo Says:

    “Googlebomb”? I’d say – bombs away!!! Lets teach the Mufti’s a lesson Internet style!

  2. Greg Maxwell Says:

    It may never be knowable if this was an intentional act or an accidental one… Normally we would presume accidental, but since the Pakistan government and by extension Pakistan telecom are already acting maliciously by intentionally blocking youtube within their own country I don’t know that we can invoke Hanlon’s razor quite so quickly.

    A little technical background: The internet is built on trust. Well, it’s buffered trust, but trust none the less. If you configure your routers to claim to have a route to a particular destination other routers will believe it unless they are configured otherwise. If you are a small network with competent upstreams they will be configured to filter, and they will verify your claims of address ownership before sending traffic to you. Apparently if you are Pakistan telecom (ASN 17557) and your upstream is PCCW .

    Youtube is trying to get their address space back by announcing more specific routes, /25 routes to beat the /24 that Pakistan telecom is announcing. However, the same kinds of filters which should have prevented the initial hijacking are very frequently configured to drop any route more specific than /24, so Youtube’s steal-back effort is having little effect.

    Some people have been spreading the claim that this outage is DNS related. This is not true. Pakistan telecom is hijacking youtube’s address space, they have managed to configured their routers to claim that youtube is on their network, then they drop the packets.

    This is simple to demonstrate:

    First we look up youtube’s IP addresses:
    [gmaxwell@bessel ~]$ host youtube.com
    youtube.com has address 208.65.153.251
    youtube.com has address 208.65.153.253
    youtube.com has address 208.65.153.238

    Then we log into a route server and see who is announcing them:
    [gmaxwell@bessel ~]$ telnet route-views.oregon-ix.net
    Trying 128.223.51.103…
    Connected to route-views.oregon-ix.net.
    Escape character is ‘^]’.

    **********************************************************************

    Oregon Exchange BGP Route Viewer
    route-views.oregon-ix.net / route-views.routeviews.org

    route views data is archived on http://archive.routeviews.org

    **********************************************************************

    route-views.oregon-ix.net>show ip bgp 208.65.153.251
    BGP routing table entry for 208.65.153.0/24, version 1628976
    Paths: (39 available, best #39, table Default-IP-Routing-Table)
    Not advertised to any peer
    5459 3491 17557
    195.66.232.239 from 195.66.232.239 (195.66.232.239)
    Origin IGP, localpref 100, valid, external
    Community: 5459:3 5459:60
    16150 3491 17557
    217.75.96.60 from 217.75.96.60 (217.75.96.60)
    Origin IGP, metric 0, localpref 100, valid, external
    Community: 16150:63392 16150:65232 16150:65320
    2905 701 3491 17557

    gmaxwell@bessel ~]$ Then we look up the ASN which is announcing the route, ASN 17557
    whois -h WHOIS.APNIC.NET AS17557
    aut-num: AS17557
    as-name: PKTELECOM-AS-AP
    descr: Pakistan Telecom
    descr: ITI Region PTCL
    country: PK

    Youtube is now trying to fight back:

    route-views.oregon-ix.net>show ip bgp 208.65.153.128/25
    route-views.oregon-ix.net>show ip bgp 208.65.153.128/25
    BGP routing table entry for 208.65.153.128/25, version 1636911
    Paths: (5 available, best #5, table Default-IP-Routing-Table)
    Not advertised to any peer
    7500 2516 3549 36561
    202.249.2.86 from 202.249.2.86 (203.178.133.115)
    Origin IGP, localpref 100, valid, external
    7660 2516 3549 36561
    203.181.248.168 from 203.181.248.168 (203.181.248.168)
    Origin IGP, localpref 100, valid, external
    Community: 2516:1030

    However, /25 routes are supposed to be rejected by ISP’s filters, so YouTube’s attempted fight back is far less effective than Pakistan telecom’s initial hijacking.

    The correct solution here is for Pakistan telecom’s upstream to filter their announcement. It seems that Pakistan telecom has several other upstreams through which the youtube route is not being redistributed.

    This sort of misconfiguration happens from time to time… about a year ago Cogent managed to hijack the IP address space used to run Wikipedia. But these kinds of events tend to be short lived. In cases of malicious hijacking the guilty network usually gets their service turned off by their upstream(s).

  3. William (green) Says:

    I wonder how long that outage lasted. I can’t find any useful articles on it, just random blog posts.
    What’s to stop anybody from doing this at any given time, to any other site, I wonder?

  4. William (green) Says:

    Another thing I’m curious about is if this has any connection to the cable cuttings from a while back. Different place, but a similar vein, you know?

  5. Cyde Weys Says:

    I would say it’s very unlikely. Unless you think Pakistan was the one behind the cutting of those cables.

  6. Greg Maxwell Says:

    It just came back up globally a few minutes ago. It’s been working for perhaps an hour or so on little islands of the internet due to my above mentioned efforts by youtube to work around the issue. Now either PCCW is filtering out the bad route, or Pakistan telecom has stopped announcing it.

    If it was PCCW that started filtering it, their sudden attention to the matter might have been brought on by the several networks that dropped peering sessions to them… or that could just be coincidental. :)

    The outage was 2hr 15 mins approximately.

  7. William (green) Says:

    I wouldn’t necessarily say that Pakistan is the one that cut them, but who’s to say that it’s not somebody else? It seems conceivable that this could have been wrought by the hands of whoever that was. I have no supporting evidence, but I don’t it should be entirely dismissed.

  8. Greg Maxwell Says:

    William, There is absolute certainty that Pakistan telecom is the guilty party here. They were advertising youtube’s IP space. That isn’t a matter up for debate. I pasted enough instructions that you could have validated it for yourself while it was happening, … and soon enough the historical routing data will show up on the various archive sites.

    It is unlikely that they caused the global outage intentionally: Anyone in a position to perform this act intentionally would have known that it wouldn’t last long.

  9. drinian Says:

    Wow, I didn’t realize that there was so much trust involved in that part of the routing system. I hate to imagine what could happen to a small, personal site…

  10. Cyde Weys Says:

    I remember studying BGP in Networking class at UMD and being blown away by the complete lack of security. Back in 1997, during the AS7007, the whole Internet was effectively brought down for a little while by a misconfigured router (although it could’ve easily been done by someone malicious). Various security layers have since been added on top of BGP, including various filters on who is allowed to define routes. But obviously not enough has been done if a huge site like YouTube can be brought down by a random router in Pakistan.

    Unfortunately, I don’t think this incident will prove to be big enough to really precipitate any further major changes like the AS7007 incident did. Although this did hit Google, and Google is huge and influential, so who knows?

  11. William (green) Says:

    You think it’ll take a number of these, or maybe one that drops the Google main page or something?

  12. Cyde Weys Says:

    Here’s some more history about these kinds of routing mistakes. They’ve happened a lot more often than you would like to think. This most recent one in Pakistan appears to have been a mistake like the others, but it wasn’t a pure mistake; someone was setting up censorship and messed that up.