Supported by the GlobalNOC at Indiana University

Documentation



BGP Communities

This document describes the BGP Communities used by the Indiana Gigapop. This includes communities that the Gigapop tags simply to add meta-data to BGP prefixes and communities that Gigapop members can set to influence routing behavior at the Gigapop.

BGP Communities the Indiana Gigapop sets:

19782:1001 Tagged to all Indiana Gigapop Member prefixes that receive R&E (ie connectivity to Internet2 and other Gigapop members) connectivity.
19782:1002 Tagged to all Indiana Gigapop Member ipv4 prefixes that receive Commodity Internet connectivity.
19782:1003 Tagged to all Indiana Gigapop Member ipv4 prefixes that receive Settlement-Free Commodity Internet connectivity.
19782:1006 Tagged to all Indiana Gigapop Member ipv6 prefixes that receive R&E (ie connectivity to Internet2 and other Gigapop members) connectivity.
19782:2001 Tagged to all prefixes received from Commodity Internet providers (currently Sprint and Timewarner)
19782:2002 Tagged to all prefixes received from Settlement-Free Peer Networks (currently IQuest, Wintek, and Internet2's Peering Service)
19782:2003 Tagged to all prefixes received from R&E network (currently Internet2 and MREN)
19782:2004 Tagged to all ipv6 prefixes received from R&E network (currently Internet2, BTAA and MREN)

BGP Communities that Gigapop Members can set:

ACTIVE 19782:666 ACTIVE

ACTIVE
DDoS Mitigation (Scrub). Prefixes announced with this BGP community will be suppressed from commodity transit and peer networks and announced exclusivly to Internet2 (Zenedge) DDoS Mitigation (Scrubbing) Service. Allowed prefix lengths up to /24 for IPv4 and /48 for IPv6. Longer prefixes will be rejected and not mitigated (scrubbed).
ACTIVE

19782:911

Blackhole/Discard, GigaPOP will discard traffic toward this prefix and trigger upstream providers to discard traffic towards this prefix.  Only members who “own” the prefix may set the 19782:911 community.

Upstream Provider Participation

  • Internet2 (BGP Community 11537:911 or 65536:666)
  • Internet2 TR-CPS (BGP Community 65536:666)
  • Hurricane Electric (BGP Community 6939:666)
  • WiscNet Regional Internet Peering Service (BGP Community 2381:911)
  • TeliaSonera (BGP Community 1299:999)
  • Cogent Communications (triggered from 19782:911)

Local-preference modification (member default is 400, R&E Peer default is 300, Settlement-Free Peers 275, Commodity Peer default is 250 and Commodity Transit default is 200):

19782:150 Causes the Gigapop to add 50 to the default BGP Local-Preference applied to Member prefixes.
19782:50 Causes the Gigapop to subtract 50 from the default BGP Local-Preference applied to Member prefixes
19782:600 Causes the Gigapop to set BGP Local-Prefernece to 600
19782:500 Causes the Gigapop to set BGP Local-Prefernece to 500
19782:200 Causes the Gigapop to set BGP Local-Prefernece to 200

Export modification: default behavior is to send all learned GigaPOP prefixes to all BGP peers. The following communities are available per-prefix to allow members and peers to modify the default behavior (XXXXX=remote ASN. i.e. 65100:87 to block prefix to IU):

65000:XXXX Do not send this prefix to R&E Peer network.
65000:65000 Do not send this prefix to any R&E Peer network.
65001-65003:XXXX GigaPOP should prepend one-three ASs to R&E Peer network.
65100:XXXX Do not send this prefix to GigaPOP member.
65100:65100 Do not send this prefix to any GigaPOP member.
65101-65103:XXXX GigaPOP should prepend one-three ASs to GigaPOP member.
65400:XXXX Do not send this prefix to GigaPOP transit.
65400:65400 Do not send this prefix to any GigaPOP transit.
65401-65403:XXXX GigaPOP should prepend one-three ASs to GigaPOP transit.

Note: The Gigapop may need to change the BGP Communities it tags to prefixes in the future. If this is necessary, we will provide sufficient notification so that any routing policy members may have based on these communities can be updated.


Indiana GigaPOP NOC Support - DDoS Mitigation (Scrubbing) Service Procedure

Keywords: attack, DDoS, DDoS Attack Mitigation, I-Light, Indiana GigaPOP, mitigation, notifications, scrubbing, scrutinizer

Indiana GigaPOP will implement a DDoS Mitigation (Scrubbing) Service via Internet2.  The service is provided by Zenedge to Internet2 Network Participants and their members.  All Indiana GigaPOP Members may take advantage of this service.

There are four key components to the service.

*Service Desk task items displayed in the highlighted areas below.

Detection 

Indiana GigaPOP has implemented Scrutinizer, a NetFlow analytic tool by Plixer. Scrutinizer provides the Indiana GigaPOP with the ability to analyze netflow data with a sampling rate of 1:100 packets. The software includes algorithms to detect patterns in traffic flows, one of those algorithms provide the capabilities to detected DDoS attacks against an IP address or a network. Scrutinizer will notify the Indiana GigaPOP NOC when the DDoS algorithm detects an attack.  An example of the notification to be received from "Scrutinizer" is below:

  • DDOS - Possible - Inbound DDOS Attack 34263 external hosts sent a combined total of 368868 flows with flow characteristics within a byte/packet standard deviation of 1000/1000 in 5 minutes. This is often indicative of a DDOS attack. Excessive DDoS Detection seen on exporter(s): EnabledExporters. Targeted host: 198.62.98.33. 

The Service Desk will engage the State Engineering Team via phone during and after business hours to assess the legitimacy and IP netblock identification of the received notification.  *The Service Desk will rely on the State Engineering Team for IP netblock identification until information is populated within DB.

Existing Indiana GigaPOP and I-Light IP Addressing information is located here.

Notification

Notifications consist of three phases.  The notification and template used and sent via the Service Desk will be dependent upon whether or not the member being attacked is for Indiana GigaPOP or I-Light.  *An additional notification will be sent to members when an attack lasted less than the defined time.

  • A DDoS Attack is detected - lasting longer than fifteen (15) minutes.
    • Members will receive a notification an attack has occurred.
    • The Service Desk will use the email template "Member XXXX DDoS Attack Mitigation Service Investigation"
  • Indiana GigaPOP Engineers activated mitigation (scrubbing) in response to an attack.
    • Members will receive a notification mitigation (scrubbing) has occurred.
    • The Service Desk will use the email template "Member XXXX DDoS Attack Mitigation Service (initiated)"
  • A DDoS Attack has subsided.
    • Members will receive a notification when an attack has subsided.
    • The Service Desk will use the email template "Member XXXX DDoS Attack Mitigation Service (ended)"
  • *A DDoS Attack was short-lived - lasted less than fifteen (15) minutes.
    • Members will receive a notification when an attack was short-lived.
    • The Service Desk will use the email template "Member XXXX DDoS Attack Mitigation Service (short-lived)"

Members who detect or suspect a DDoS threat that is undetected by the Indiana GigaPOP and control their own BGP configuraiton, may initiate mitigation (scrubbing). Indiana GigaPOP asks that members who initate mitigation (scrubbing) on their own, also notify the Indiana GigaPOP NOC.  Members who do not control their own BGP configuration and suspect a DDoS threat, may contact the Indiana GigaPOP NOC and request an Engineer to assit.

Mitigation

Indiana GigaPOP Engineers or members who control their own BGP configuration, can trigger mitigation (scrubbing) of prefixes up to /24 for IPv4 or /48 for IPv6 by modifying their BGP announcements to add a BGP community tag to the specific prefix.  Please refer to the Indiana GigaPOP BGP Communities documentation.  Members can inquire with Indiana GigaPOP Engineers through the NOC for status of DDoS threats.  Zenedge provides a portal which will indicate both clean and attack traffic.  Most importantly, when the attack traffic has subsided and the Indiana GigaPOP Engineers or members can withdraw the additional BGP announcement.

Please be aware, adding the BGP community to prefix annoucement triggers two actions within the Indiana GigaPOP routing policies.  These actions occur simultaenously.  Please use cautiously.

  1. Suppress prefix to Commodity Transit and Commodity Peer Networks.  Prefixes tagged with the DDoS BGP Community that equal normal announcements will be suppressed to Commodity Transit and Commodity Peer Networks.  Prefixes tagged with the DDoS BGP Community that are longer than normal annoucements will not suppress the normal announcement, instead only advertise the more specific prefix to Internet2/Zenedge.
  2. Announce prefix to Internet2/Zenedge DDoS Mitigation (Scrubbing) Service.  Prefixes announced to Internet2/Zenedge will force traffic from Commodity Transit and Commodity Peer Networks to ingress through Zendge, to Internet2, to Indiana Gigapop.

Reporting

Indiana Gigapop and I-Light are working on a method to convey DDoS attack occurrances to members.  Due to the more sensitive nature of the information, publicly posting reports would be the last option.  More information on where to access reports of DDoS attacks will be available soon.


Indiana GigaPOP NOC Support - DDoS Mitigation (Scrubbing) Service FAQ

GigaPOP / I-Light DDoS Mitigation (Scrubbing) Service FAQ

 


Q: What BGP Community triggers mitigation (scrubbing)?

A: Please refer to the Indiana GigaPOP BGP Communities documentaiton.


Q: What is the cost to Indiana GigaPOP and I-Light Members?

A: The cost for the scrubbing service will be shared between Indiana GigaPOP members, including I-Light.   Once we know how many of the GigaPOP members are going to participate we will spread the costs proportionally.  We’ll be InTouch with the individual GigaPOP members to discuss these costs in more detail in the near future.


Q: Can members Opt-In and Opt-Out?

A: We think so, initially the scrubbing service will be a manual action by either the Indiana GigaPOP and I-Light Network Engineers or members themselves via BGP Communities.  A future automated process would need to take into account prefixes which would be excluded.  Considerations may need to be made for frequency of Opt-In/Opt-Out changes as well as time required to process changes.


Q: How do members Opt-In and Opt-Out?

A: Members may Opt-In or Opt-Out by notifying the appropriate NOC; Indiana GigaPOP or I-Light.  **Please note the anticipated turnaround time to be three (3) days to change a members status.


Q: Is BGP required to participate in the scrubbing service?

            A: No.  BGP allows the Indiana GigaPOP or I-Light member to initiate the scrubbing service independently of Indiana GigaPOP or I-Light Engineers.  Without BGP, a manual process will be implemented by Indiana GigaPOP or I-Light Engineers.


Q: Will Indiana GigaPOP and I-Light Engineers be available to assist members with BGP administration and verification?

A:  Yes!  Indiana GigaPOP and I-Light Engineers can help in three ways.

  1. Documentation of examples.  Please refer to DDoS Mitigation (Scrubbing) Service Scenario and Configuration Examples.
  2. Proactive discussion on BGP Configuration
  3. Reactive assistance for BGP Configuration Verification


Q: How will members be notified when a DDoS attack is occurring against the network or the member?  How will members be notified when DDoS scrubbing is active?

            A:  The NOC for Indiana GigaPOP or I-Light will send targeted notifications to member institutions when an attack is detected, when mitigation (scrubbing) is implemented and when the attack has subsided.


Q: How will the DDoS Scrubbing service affect Multicast?

            A: The DDoS Scrubbing service only functions on commodity traffic from transit providers.  Currently no active transit providers support Multicast.  Multicast traffic should not be affected internal to the Indiana GigaPOP or I-Light, although public senders or receives can still be attacked and external traffic to those hosts may be scrubbed.


Q: Will scrubbed traffic induce latency?

            A: Yes.  Measured increased RTT (round trip time) was between 22 and 55ms, dependent on the external endpoint and path to Indiana GigaPOP or I-Light member.  To provide reference, latency from Indiana to the West coast is approximately 48ms round trip, to the East coast is approximately 22ms round trip.


Q: Will Indiana GigaPoP and I-Light make attack analytics against members available to the entire member community?

      A:  At this time, our thought is yes.  We feel sharing attack analytics will be valuable to the overall community on several fronts.  Sharing the attack information in some form or another will allow the member community to gain exposure to different attack types and will aid in helping the member community learn how to protect their network and users.  We recognize that the attack information is sensitive and we understand the importance of this information not getting into the wrong hands to be used for nefarious purposes.  With that being said, our work continues on how to distribute the information securely.  We’ll be sure to provide more information on this front at things begin to solidify.


Q: Will Indiana GigaPoP and I-Light mitigate or scrub attacks lasting less than 15 minutes?

      A:  No.  Initial attack analytics indicate most of the attacks span about 1 minute and sometimes repeat 2-3 times within a 15-minute window.  In these cases, the attack has subsided before anyone could actually react.  Reports on short attacks will be available to members.


Q: How will members be notified with respect to DDoS attacks?

    A:  Our plan is to notify members if an attack is occurring or has occurred lasting longer than 15 minutes (see previous question).  The notification to the members will be sent by the GlobalNOC.  The standard policy and language around notification is still being ironed out by the Engineering group and the Service Desk, but our thought is that we’ll notify on the following:

  1. When an attack occurs and subsides within 15 minutes.
  2. When an attack occurs lasting longer than 15 minutes.
  3. When mitigation/scrubbing was initiated.
  4. When the attack has subsided.

* For attacks lasting less than 15 minutes, members will receive a notification that an attack occurred and subsided, no mitigation/scrubbing occurred.


Q: What happens if an attack is so severe that it takes down connectivity for other GigaPoP/I-Light members?

      A:  If the attack creates a situation where the commodity links become saturated, and if we get multiple complaints from other members notifying us that are being effected, then it’s our thought that we should disconnect the targeted member from the network until the issue is resolved.


Indiana GigaPOP NOC Support - DDoS Mitigation (Scrubbing) Service - Scenario and Configuraiton Examples

Please refer to these scenarios and configuraiton examples for assistance in configuration to trigger the DDoS Mitigation (Scrubbing) Service by adding a BGP community to your announcement.  These examples are applicable to membes who administer their own BGP configuraiton.  Indiana GigaPOP Engineers are available to assist with configuriaotn, either proactively or reactively, providing recommendations as needed to our members.

 

Scenario #1

Member typically announces a /16 prefix to the Indiana GigaPOP with a DDoS detection against a specific host. In this scenario, the /16 would remain announced to the Indiana GigaPOP Commodity Transit and Peer providers with a more specific /24 announced through Internet2/Zendge for DDoS Mitigation (Scrubbing).  More specific prefixes will force traffic to the specific host within the prefix through the DDoS Mitigation (Scrubbing) Service.

Using private RFC1918 address as an example, a member may have a BGP configuration similar to this.

Cisco

!
ip prefix-list MEMBER seq 5 permit 192.168.0.0/16
!
route-map OUT permit 10
 match ip address prefix-list MEMBER
route-map OUT deny 100
!

Juniper

policy-options {
    prefix-list MEMBER {
        192.168.0.0/16;
    }
    policy-statement OUT {
        term ANNOUNCE {
            from {
                prefix-list MEMBER;
            }
            then accept;
        }
        term REJECT {
            then reject;
        }
    }
}

Using private RFC 1918 addresing, assume a DDoS attack is occurring against 192.168.73.33. The member would announce 192.168.73.0/24 with the added DDOS BGP community.

Cisco

!
ip prefix-list MEMBER seq 5 permit 192.168.0.0/16
ip prefix-list MEMBER-DDOS seq 5 permit 192.168.73.0/24
!
route-map OUT permit 5
 match ip address prefix-list MEMBER-DDOS
 set community 19782:666 additive
!
route-map OUT permit 10
 match ip address prefix-list MEMBER
!
route-map OUT deny 100
!

Juniper

policy-options {
    community DDOS {
        members 19782:666;
    }
    prefix-list MEMBER-DDOS {
        192.168.73.0/24;
    }
    policy-statement OUT {
        term ANNOUNCE-DDOS {
            from {
                prefix-list MEMBER-DDOS;
            }
            then {
            community add DDOS;
            accept;
            }
        }
        term ANNOUNCE {
            from {
                prefix-list MEMBER;
            }
            then accept;
        }
        term REJECT {
            then reject;
        }
    }
}

 

Scenario #2

Member typically announces a /24 prefix to the Indiana GigaPOP with a DDoS detection against a specific host.  In this scenario, the Indiana GigaPOP routing polices would suppress the announcement to Commodity Transit and Commodity Peer Networks and announce the /24 through Internet2/Zendge for DDoS Mitigation (Scrubbing).

Using private RFC1918 address as an example, a member may have a BGP configuration similar to this.

Cisco

!
ip prefix-list MEMBER seq 5 permit 192.168.89.0/24
!
route-map OUT permit 10
 match ip address prefix-list MEMBER
!
route-map OUT deny 100
!

Juniper

policy-options {
    prefix-list MEMBER {
        192.168.89.0/24;
    }
    policy-statement OUT {
        term ANNOUNCE {
            from {
                prefix-list MEMBER;
            }
            then accept;
        }
        term REJECT {
            then reject;
        }
    }
}

Using private RFC 1918 addresing, assume a DDoS attack is occurring against 192.168.89.33. The member would add the DDoS BGP Community to their announcement of 192.168.73.0/24.

Cisco

!
ip prefix-list MEMBER seq 5 permit 192.168.0.0/16
!
route-map OUT permit 10
 match ip address prefix-list MEMBER
 set community 19782:666 additive
!
route-map OUT deny 100
!

Juniper

policy-options {
    community DDOS {
        members 19782:666;
    }
    policy-statement OUT {
        term ANNOUNCE {
            from {
                prefix-list MEMBER;
            }
            then {
            community add DDOS;
            accept;
            ]
        }
        term REJECT {
            then reject;
        }
    }
}

Indiana GigaPOP Cache Node Information

Indiana GigaPOP Caching Information

Akamai

Akamai Peering - 193.108.92.62
Akamai V6 Peering - 2600:1400:5:15::c16c:5c3e
Akamai IPv6 Cache Node - 2001:18E8:FFFF:C::/64
Akamai Server Farm - 149.165.180.0/26
Akamai BGP point-to-point peering to ICTC T640 - 149.165.255.132/30
Akamai Cache Server Farm - 149.165.240.64/26

Google

Google V6 Caching Service - 2001:18E8:FFFF:48::/64
Google Caching Service - 149.165.180.64/26
Google Caching Service - 149.165.183.72/30

Netflix

Netflix Cache 20G Node - 2001:18E8:FFFF:9::/64
Netflix Commodity Cache Appliance Gateway RTR.ICTC - 149.165.180.129/32
Netflix Commodity Cache Appliance #1 - 149.165.180.132/32
Netflix Commodity Cache Appliances - 149.165.180.128/26
Netflix Commodity Cache Appliance #3 - 149.165.180.133/32
Netflix Commodity Cache Appliance BGP Peering - 149.165.183.92/30
Netflix Open Connect Cache IPv4 ICTC #3 - 2001:18E8:FFFF:D::/64
Netflix Commodity Cache Appliance v6 BGP Peering - 2001:18e8:ffff:6::/64
Netflix OpenConnect Cache 20G Node #2 - 2001:18E8:FFFF:A::/64
Netflix Cache #3 - 2001:18E8:FFFF:5::4/128
Netflix Commodity Cache Appliance v6 - 2001:18e8:ffff:5::/64
Netflix Commodity Cache Appliance #2 - 149.165.180.134/32
Netflix Cache #1 - 2001:18E8:FFFF:5::2/128
Netflix Cache #2 - 2001:18E8:FFFF:5::3/128
RTR.ICTC Netflix Cache Gateway - 2001:18E8:FFFF:5::1/128

Apple

Apple CDN via 350 Cermak Chicago - 17.1.148.92 on rtr.ictc.indiana.gigapop.net with ASN 714

Apple CDN IPv6 via 350 Cermak Chicago - 2620:149:bb:9520::1 on rtr.ictc.indiana.gigapop.net with ASN 714

Apple CDN received v4 routes - 17.0.0.0/8

Apple CDN received v6 routes:
2620:149:11::/48
2620:149:40::/44
2620:149:bb::/48
2620:149:bb:8000::/50
2620:149:f8::/46
2620:149:fc::/46
2620:149:140::/44
2620:149:a14::/48
2620:149:a15::/48
2620:149:ac0::/48
2620:149:ac2::/48
2620:149:ae0::/48
2620:149:ae7::/48

Apple CDN subnets:
17.0.64.0/18
17.0.128.0/18
17.1.64.0/18
17.1.128.0/18
17.34.0.0/16
17.35.240.0/20
17.38.0.0/15
17.40.0.0/15
17.42.0.0/16
17.47.0.0/20
17.47.1.0/24
17.96.0.0/16
17.97.0.0/16
17.98.0.0/16
17.99.0.0/16
17.110.0.0/15
17.110.112.0/20
17.120.0.0/16
17.121.0.0/18
17.121.128.0/17
17.124.0.0/16
17.125.0.0/16
17.128.100.0/22
17.128.106.0/23
17.128.108.0/22
17.128.112.0/22
17.128.124.0/23
17.128.126.0/24
17.129.0.0/16
17.131.0.0/16
17.132.4.0/22
17.132.4.0/24
17.132.5.0/24
17.132.16.0/24
17.132.18.0/24
17.132.19.0/24
17.132.20.0/23
17.132.22.0/23
17.132.24.0/24
17.132.25.0/24
17.132.26.0/24
17.132.27.0/24
17.132.28.0/22
17.132.28.0/24
17.132.29.0/24
17.133.0.0/16
17.134.0.0/16
17.135.0.0/16
17.136.0.0/16
17.137.0.0/16
17.138.0.0/15
17.140.0.0/19
17.140.32.0/21
17.140.40.0/24
17.140.48.0/20
17.140.64.0/19


Your request has been completed.