Supported by the GlobalNOC at Indiana University

Documentation



BGP Communities

This document describes the BGP Communities used by the Indiana Gigapop. This includes communities that the Gigapop tags simply to add meta-data to BGP prefixes and communities that Gigapop members can set to influence routing behavior at the Gigapop.

BGP Communities the Indiana Gigapop sets:

19782:1001 Tagged to all Indiana Gigapop Member prefixes that receive R&E (ie connectivity to Internet2 and other Gigapop members) connectivity.
19782:1002 Tagged to all Indiana Gigapop Member ipv4 prefixes that receive Commodity Internet connectivity.
19782:1003 Tagged to all Indiana Gigapop Member ipv4 prefixes that receive Settlement-Free Commodity Internet connectivity.
19782:1006 Tagged to all Indiana Gigapop Member ipv6 prefixes that receive R&E (ie connectivity to Internet2 and other Gigapop members) connectivity.
19782:2001 Tagged to all prefixes received from Commodity Internet providers (currently Sprint and Timewarner)
19782:2002 Tagged to all prefixes received from Settlement-Free Peer Networks (currently IQuest, Wintek, and Internet2's Peering Service)
19782:2003 Tagged to all prefixes received from R&E network (currently Internet2 and MREN)
19782:2004 Tagged to all ipv6 prefixes received from R&E network (currently Internet2, BTAA and MREN)
19782:6001 Tagged to all ip(v4 & v6) prefixes received from LHCOne peers (currently Internet2, ESnet)
19782:6002 Tagged to all ip(v4 & v6) prefixes received from LHCOne participating members

BGP Communities that Gigapop Members can set:

19782:666

DDoS Mitigation (Scrub). Prefixes announced with this BGP community will be suppressed from commodity transit and peer networks and announced exclusivly to Internet2 (Zenedge) DDoS Mitigation (Scrubbing) Service. Allowed prefix lengths up to /24 for IPv4 and /48 for IPv6. Longer prefixes will be rejected and not mitigated (scrubbed).

19782:911

Blackhole/Discard, GigaPOP will discard traffic toward this prefix and trigger upstream providers to discard traffic towards this prefix.  Only members who “own” the prefix may set the 19782:911 community.

Upstream Provider Participation

  • Internet2 (BGP Community 11537:911 or 65536:666)
  • Internet2 TR-CPS (BGP Community 65536:666)
  • Hurricane Electric (BGP Community 6939:666)
  • WiscNet Regional Internet Peering Service (BGP Community 2381:911)
  • TeliaSonera (BGP Community 1299:999)
  • Cogent Communications (triggered from 19782:911)

Local-preference modification (member default is 400, R&E Peer default is 300, Settlement-Free Peers 275, Commodity Peer default is 250 and Commodity Transit default is 200):

19782:150 Causes the Gigapop to add 50 to the default BGP Local-Preference applied to Member prefixes.
19782:50 Causes the Gigapop to subtract 50 from the default BGP Local-Preference applied to Member prefixes
19782:600 Causes the Gigapop to set BGP Local-Prefernece to 600
19782:500 Causes the Gigapop to set BGP Local-Prefernece to 500
19782:200 Causes the Gigapop to set BGP Local-Prefernece to 200

Export modification: default behavior is to send all learned GigaPOP prefixes to all BGP peers. The following communities are available per-prefix to allow members and peers to modify the default behavior (XXXXX=remote ASN. i.e. 65100:87 to block prefix to IU):

65000:XXXX Do not send this prefix to R&E Peer network.
65000:65000 Do not send this prefix to any R&E Peer network.
65001-65003:XXXX GigaPOP should prepend one-three ASs to R&E Peer network.
65100:XXXX Do not send this prefix to GigaPOP member.
65100:65100 Do not send this prefix to any GigaPOP member.
65101-65103:XXXX GigaPOP should prepend one-three ASs to GigaPOP member.
65400:XXXX Do not send this prefix to GigaPOP transit.
65400:65400 Do not send this prefix to any GigaPOP transit.
65401-65403:XXXX GigaPOP should prepend one-three ASs to GigaPOP transit.

Note: The Gigapop may need to change the BGP Communities it tags to prefixes in the future. If this is necessary, we will provide sufficient notification so that any routing policy members may have based on these communities can be updated.



Indiana GigaPOP NOC Support - DDoS Mitigation (Scrubbing) Service FAQ

GigaPOP / I-Light DDoS Mitigation (Scrubbing) Service FAQ

 


Q: What BGP Community triggers mitigation (scrubbing)?

A: Please refer to the Indiana GigaPOP BGP Communities documentaiton.


Q: What is the cost to Indiana GigaPOP and I-Light Members?

A: The cost for the scrubbing service will be shared between Indiana GigaPOP members, including I-Light.   Once we know how many of the GigaPOP members are going to participate we will spread the costs proportionally.  We’ll be InTouch with the individual GigaPOP members to discuss these costs in more detail in the near future.


Q: Can members Opt-In and Opt-Out?

A: We think so, initially the scrubbing service will be a manual action by either the Indiana GigaPOP and I-Light Network Engineers or members themselves via BGP Communities.  A future automated process would need to take into account prefixes which would be excluded.  Considerations may need to be made for frequency of Opt-In/Opt-Out changes as well as time required to process changes.


Q: How do members Opt-In and Opt-Out?

A: Members may Opt-In or Opt-Out by notifying the appropriate NOC; Indiana GigaPOP or I-Light.  **Please note the anticipated turnaround time to be three (3) days to change a members status.


Q: Is BGP required to participate in the scrubbing service?

            A: No.  BGP allows the Indiana GigaPOP or I-Light member to initiate the scrubbing service independently of Indiana GigaPOP or I-Light Engineers.  Without BGP, a manual process will be implemented by Indiana GigaPOP or I-Light Engineers.


Q: Will Indiana GigaPOP and I-Light Engineers be available to assist members with BGP administration and verification?

A:  Yes!  Indiana GigaPOP and I-Light Engineers can help in three ways.

  1. Documentation of examples.  Please refer to DDoS Mitigation (Scrubbing) Service Scenario and Configuration Examples.
  2. Proactive discussion on BGP Configuration
  3. Reactive assistance for BGP Configuration Verification


Q: How will members be notified when a DDoS attack is occurring against the network or the member?  How will members be notified when DDoS scrubbing is active?

            A:  The NOC for Indiana GigaPOP or I-Light will send targeted notifications to member institutions when an attack is detected, when mitigation (scrubbing) is implemented and when the attack has subsided.


Q: How will the DDoS Scrubbing service affect Multicast?

            A: The DDoS Scrubbing service only functions on commodity traffic from transit providers.  Currently no active transit providers support Multicast.  Multicast traffic should not be affected internal to the Indiana GigaPOP or I-Light, although public senders or receives can still be attacked and external traffic to those hosts may be scrubbed.


Q: Will scrubbed traffic induce latency?

            A: Yes.  Measured increased RTT (round trip time) was between 22 and 55ms, dependent on the external endpoint and path to Indiana GigaPOP or I-Light member.  To provide reference, latency from Indiana to the West coast is approximately 48ms round trip, to the East coast is approximately 22ms round trip.


Q: Will Indiana GigaPoP and I-Light make attack analytics against members available to the entire member community?

      A:  At this time, our thought is yes.  We feel sharing attack analytics will be valuable to the overall community on several fronts.  Sharing the attack information in some form or another will allow the member community to gain exposure to different attack types and will aid in helping the member community learn how to protect their network and users.  We recognize that the attack information is sensitive and we understand the importance of this information not getting into the wrong hands to be used for nefarious purposes.  With that being said, our work continues on how to distribute the information securely.  We’ll be sure to provide more information on this front at things begin to solidify.


Q: Will Indiana GigaPoP and I-Light mitigate or scrub attacks lasting less than 15 minutes?

      A:  No.  Initial attack analytics indicate most of the attacks span about 1 minute and sometimes repeat 2-3 times within a 15-minute window.  In these cases, the attack has subsided before anyone could actually react.  Reports on short attacks will be available to members.


Q: How will members be notified with respect to DDoS attacks?

    A:  Our plan is to notify members if an attack is occurring or has occurred lasting longer than 15 minutes (see previous question).  The notification to the members will be sent by the GlobalNOC.  The standard policy and language around notification is still being ironed out by the Engineering group and the Service Desk, but our thought is that we’ll notify on the following:

  1. When an attack occurs and subsides within 15 minutes.
  2. When an attack occurs lasting longer than 15 minutes.
  3. When mitigation/scrubbing was initiated.
  4. When the attack has subsided.

* For attacks lasting less than 15 minutes, members will receive a notification that an attack occurred and subsided, no mitigation/scrubbing occurred.


Q: What happens if an attack is so severe that it takes down connectivity for other GigaPoP/I-Light members?

      A:  If the attack creates a situation where the commodity links become saturated, and if we get multiple complaints from other members notifying us that are being effected, then it’s our thought that we should disconnect the targeted member from the network until the issue is resolved.



Indiana GigaPOP NOC Support - DDoS Mitigation (Scrubbing) Service - Scenario and Configuraiton Examples

Please refer to these scenarios and configuraiton examples for guidance in triggering the DDoS Mitigation (Scrubbing) Service by adding a BGP community to your prefix announcement.  These examples are applicable to membes who administer their own BGP configuraiton.  Indiana GigaPOP Engineers are available to assist with configuration, either proactively or reactively, providing recommendations as needed to our members.

Note: The examples below utilize RFC1918 (IPv4 Private IP Addressing) to illustrate the configuraton required to trigger DDoS Mitigation (Scrubbiing).  Please adjust the prefix addressing appropriately for your insttitution.

 

Scenario #1

Member typically announces varous prefix lengths, from a /16 to a /23 to the Indiana GigaPOP.  In the scenario with a DDoS detection against a specific host, the /16 would remain announced to the Indiana GigaPOP Commodity Transit and Peer providers with a more specific /24 announced through Internet2/Zendge for DDoS Mitigation (Scrubbing).  More specific prefixes will force traffic to the specific host within the prefix through the DDoS Mitigation (Scrubbing) Service.

A possible BGP policy configuration for a member's peer defintion to the Indiana Gigapop.

Cisco

!
ip prefix-list MEMBER seq 5 permit 192.168.0.0/16
!
route-map OUT permit 10
 match ip address prefix-list MEMBER
!
route-map OUT deny 100
!

Juniper

policy-options {
    prefix-list MEMBER {
        192.168.0.0/16;
    }
    policy-statement OUT {
        term ANNOUNCE {
            from {
                prefix-list MEMBER;
            }
            then accept;
        }
        term REJECT {
            then reject;
        }
    }
}

Assuming a DDoS attack is occurring against 192.168.73.33. The member would announce 192.168.73.0/24 with the added DDOS BGP community.

Cisco

!
ip prefix-list MEMBER seq 5 permit 192.168.0.0/16
ip prefix-list MEMBER-DDOS seq 5 permit 192.168.73.0/24
!
route-map OUT permit 5
 match ip address prefix-list MEMBER-DDOS
 set community 19782:666 additive
!
route-map OUT permit 10
 match ip address prefix-list MEMBER
!
route-map OUT deny 100
!

Juniper

policy-options {
    community DDOS {
        members 19782:666;
    }
    prefix-list MEMBER-DDOS {
        192.168.73.0/24;
    }
    policy-statement OUT {
        term ANNOUNCE-DDOS {
            from {
                prefix-list MEMBER-DDOS;
            }
            then {
            community add DDOS;
            accept;
            }
        }
        term ANNOUNCE {
            from {
                prefix-list MEMBER;
            }
            then accept;
        }
        term REJECT {
            then reject;
        }
    }
}

 

Scenario #2

Member typically announces a /24 to the Indiana GigaPOP.  In the scenario with a DDoS detection against a specific host, the /24 would would suppress the announce to Commodity Transit and Commodity Peer Networks, instead announcing the /24 through Internet2/Zendge for DDoS Mitigation (Scrubbing). 

A possible BGP policy configuration for a member's peer defintion to the Indiana Gigapop.

Cisco

!
ip prefix-list MEMBER seq 5 permit 192.168.89.0/24
!
route-map OUT permit 10
 match ip address prefix-list MEMBER
!
route-map OUT deny 100
!

Juniper

policy-options {
    prefix-list MEMBER {
        192.168.89.0/24;
    }
    policy-statement OUT {
        term ANNOUNCE {
            from {
                prefix-list MEMBER;
            }
            then accept;
        }
        term REJECT {
            then reject;
        }
    }
}

Assuming a DDoS attack is occurring against 192.168.89.33. The member would add the DDoS BGP Community to their announcement of 192.168.73.0/24.

Cisco

!
ip prefix-list MEMBER seq 5 permit 192.168.0.0/16
!
route-map OUT permit 10
 match ip address prefix-list MEMBER
 set community 19782:666 additive
!
route-map OUT deny 100
!

Juniper

policy-options {
    community DDOS {
        members 19782:666;
    }
    policy-statement OUT {
        term ANNOUNCE {
            from {
                prefix-list MEMBER;
            }
            then {
            community add DDOS;
            accept;
            ]
        }
        term REJECT {
            then reject;
        }
    }
}


Indiana GigaPOP Cache Node Information

Indiana GigaPOP Caching Information

Akamai

Akamai Peering - 193.108.92.62
Akamai V6 Peering - 2600:1400:5:15::c16c:5c3e
Akamai IPv6 Cache Node - 2001:18E8:FFFF:C::/64
Akamai Server Farm - 149.165.180.0/26
Akamai BGP point-to-point peering to ICTC T640 - 149.165.255.132/30
Akamai Cache Server Farm - 149.165.240.64/26

Google

Google V6 Caching Service - 2001:18E8:FFFF:48::/64
Google Caching Service - 149.165.180.64/26
Google Caching Service - 149.165.183.72/30

Netflix

Netflix Cache 20G Node - 2001:18E8:FFFF:9::/64
Netflix Commodity Cache Appliance Gateway RTR.ICTC - 149.165.180.129/32
Netflix Commodity Cache Appliance #1 - 149.165.180.132/32
Netflix Commodity Cache Appliances - 149.165.180.128/26
Netflix Commodity Cache Appliance #3 - 149.165.180.133/32
Netflix Commodity Cache Appliance BGP Peering - 149.165.183.92/30
Netflix Open Connect Cache IPv4 ICTC #3 - 2001:18E8:FFFF:D::/64
Netflix Commodity Cache Appliance v6 BGP Peering - 2001:18e8:ffff:6::/64
Netflix OpenConnect Cache 20G Node #2 - 2001:18E8:FFFF:A::/64
Netflix Cache #3 - 2001:18E8:FFFF:5::4/128
Netflix Commodity Cache Appliance v6 - 2001:18e8:ffff:5::/64
Netflix Commodity Cache Appliance #2 - 149.165.180.134/32
Netflix Cache #1 - 2001:18E8:FFFF:5::2/128
Netflix Cache #2 - 2001:18E8:FFFF:5::3/128
RTR.ICTC Netflix Cache Gateway - 2001:18E8:FFFF:5::1/128

Apple

Apple CDN via 350 Cermak Chicago - 17.1.148.92 on rtr.ictc.indiana.gigapop.net with ASN 714

Apple CDN IPv6 via 350 Cermak Chicago - 2620:149:bb:9520::1 on rtr.ictc.indiana.gigapop.net with ASN 714

Apple CDN received v4 routes - 17.0.0.0/8

Apple CDN received v6 routes:
2620:149:11::/48
2620:149:40::/44
2620:149:bb::/48
2620:149:bb:8000::/50
2620:149:f8::/46
2620:149:fc::/46
2620:149:140::/44
2620:149:a14::/48
2620:149:a15::/48
2620:149:ac0::/48
2620:149:ac2::/48
2620:149:ae0::/48
2620:149:ae7::/48

Apple CDN subnets:
17.0.64.0/18
17.0.128.0/18
17.1.64.0/18
17.1.128.0/18
17.34.0.0/16
17.35.240.0/20
17.38.0.0/15
17.40.0.0/15
17.42.0.0/16
17.47.0.0/20
17.47.1.0/24
17.96.0.0/16
17.97.0.0/16
17.98.0.0/16
17.99.0.0/16
17.110.0.0/15
17.110.112.0/20
17.120.0.0/16
17.121.0.0/18
17.121.128.0/17
17.124.0.0/16
17.125.0.0/16
17.128.100.0/22
17.128.106.0/23
17.128.108.0/22
17.128.112.0/22
17.128.124.0/23
17.128.126.0/24
17.129.0.0/16
17.131.0.0/16
17.132.4.0/22
17.132.4.0/24
17.132.5.0/24
17.132.16.0/24
17.132.18.0/24
17.132.19.0/24
17.132.20.0/23
17.132.22.0/23
17.132.24.0/24
17.132.25.0/24
17.132.26.0/24
17.132.27.0/24
17.132.28.0/22
17.132.28.0/24
17.132.29.0/24
17.133.0.0/16
17.134.0.0/16
17.135.0.0/16
17.136.0.0/16
17.137.0.0/16
17.138.0.0/15
17.140.0.0/19
17.140.32.0/21
17.140.40.0/24
17.140.48.0/20
17.140.64.0/19



Your request has been completed.