--- 1/draft-ietf-grow-bgp-graceful-shutdown-requirements-05.txt 2010-10-25 20:14:44.000000000 +0200 +++ 2/draft-ietf-grow-bgp-graceful-shutdown-requirements-06.txt 2010-10-25 20:14:44.000000000 +0200 @@ -4,24 +4,24 @@ Intended status: Informational P. Francois UCL C. Pelsser IIJ Z. Ahmad Orange Business Services A. J. Elizondo Armengol Telefonica I+D T. Takeda NTT - October 11, 2010 + October 22, 2010 Requirements for the graceful shutdown of BGP sessions - draft-ietf-grow-bgp-graceful-shutdown-requirements-05.txt + draft-ietf-grow-bgp-graceful-shutdown-requirements-06.txt Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from @@ -40,23 +40,23 @@ and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. - This Internet-Draft will expire on April 09, 2011. + This Internet-Draft will expire on April 20, 2011. -Requirements for the graceful shutdown of BGP sessions +Internet-Draft Requirements for the graceful shutdown of BGP sessions Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents @@ -86,29 +86,29 @@ 1. Conventions used in this document...........................3 2. Introduction................................................3 3. Problem statement...........................................4 3.1. Example of undesirable BGP routing behavior.................4 3.2. Causes of packet loss.......................................5 4. Terminology.................................................6 5. Goals and requirements......................................7 6. Reference Topologies........................................9 6.1. E-BGP topologies............................................9 6.2. I-BGP topologies...........................................11 - 7. Security Considerations....................................14 - 8. IANA Considerations........................................15 - 9. References.................................................15 - 9.1. Normative References.......................................15 - 9.2. Informative References.....................................15 - 10. Acknowledgments............................................15 - 11. Author's Addresses.........................................16 + 7. Security Considerations....................................15 + 8. IANA Considerations........................................16 + 9. References.................................................16 + 9.1. Normative References.......................................16 + 9.2. Informative References.....................................16 + 10. Acknowledgments............................................17 + 11. Author's Addresses.........................................17 -Requirements for the graceful shutdown of BGP sessions +Internet-Draft Requirements for the graceful shutdown of BGP sessions 1. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 2. Introduction The Border Gateway Protocol(BGP) [BGP-4] is heavily used in Service @@ -125,40 +125,40 @@ is available in the Autonomous System (AS), it should be made possible to reroute the customer or peer traffic on this backup path before the BGP session(s) is/are torn down, the nominal path withdrawn and the forwarding is interrupted on the nominal path. The requirements also cover the subsequent re-establishment of the BGP session as even this "UP" case can currently trigger route loss and thus traffic loss at some routers. Currently, BGP [BGP-4] and MP-BGP [MP-BGP] do not include any - operation to gracefully withdraw a prefix while traffic toward that - prefix could still be correctly forwarded. When a BGP session is - taken down, BGP behaves as if it was a sudden link or router failure - and withdraws the prefixes learnt over that session, which may - trigger traffic loss. There is no mechanism to advertise to its BGP - peers that the prefix will soon be unreachable, while still being - reachable. When applicable, such mechanism would reduce or prevent - traffic loss. It would typically be applicable in case of a - maintenance operation requiring the shutdown of a forwarding - resource. Typical examples would be a link or line card maintenance, - replacement or upgrade. It may also be applicable for a software - upgrade as it may involve a firmware reset on the line cards and - hence forwarding interruption. + operation to gracefully advertise or withdraw a prefix while traffic + toward that prefix could still be correctly forwarded using the old + path. When a BGP session is taken down, BGP behaves as if it was a + sudden link or router failure and withdraws the prefixes learnt over + that session, which may trigger traffic loss. There is no mechanism + to advertise to its BGP peers that the prefix will soon be + unreachable, while still being reachable. When applicable, such + mechanism would reduce or prevent traffic loss. It would typically be + applicable in case of a maintenance operation requiring the shutdown + of a forwarding resource. Typical examples would be a link or line + card maintenance, replacement or upgrade. It may also be applicable + for a software upgrade as it may involve a firmware reset on the line + cards and hence forwarding interruption. The introduction of Route Reflectors as per [RR] to solve scalability issues bound to IBGP full-meshes has worsened the duration of routing convergence as some route reflectors may hide the back up path. Thus depending on RR topology more IBGP hops may be involved in the IBGP convergence. -Requirements for the graceful shutdown of BGP sessions +Internet-Draft Requirements for the graceful shutdown of BGP sessions Note that these planned maintenance operations cannot be addressed by Graceful Restart extensions [GR] as GR only applies when the forwarding is preserved during the control plane restart. On the contrary, Graceful Shutdown applies when the forwarding is interrupted. Note also that some protocols are already considering such graceful shutdown procedure (e.g. GMPLS in [RFC5817]). A successful approach of such mechanism should minimize the loss of @@ -178,27 +178,27 @@ the traffic that was directed towards the removed next-hops may be lost until the end of the BGP convergence. As it is a planned operation, a make before break solution should be made possible. As maintenance operations are frequent in large networks [Reliability], the global availability of the network is significantly impaired by this BGP maintenance issue. 3.1. Example of undesirable BGP routing behavior - To illustrate these problems, let us consider the following example - where one customer router "CUST" is dual-attached to two SP routers - "ASBR1" and "ASBR2". + To illustrate these problems, let us consider the following simple + example where one customer router "CUST" is dual-attached to two SP + routers "ASBR1" and "ASBR2". ASBR1 and ASBR2 are in the same AS and owned by the same service provider. Both are IBGP client of the route reflector R1. -Requirements for the graceful shutdown of BGP sessions +Internet-Draft Requirements for the graceful shutdown of BGP sessions ' AS1 ' AS2 ' /-----------ASBR1--- / \ / \ CUST R1 \ / @@ -234,37 +234,37 @@ When multiple BGP routers are involved and plenty of prefixes are affected, the recovery process can take longer than applications requirements. 3.2. Causes of packet loss The loss of packets during the maintenance has two main causes: - lack of an alternate path on some routers, - transient routing inconsistency. -Requirements for the graceful shutdown of BGP sessions +Internet-Draft Requirements for the graceful shutdown of BGP sessions Some routers may lack an alternate path because another router is hiding the backup path. This router can be: - a route reflector only propagating its best path; - the backup ASBR not advertising the backup path because it prefers the nominal path. This lack of knowledge of the alternate path is the first target of this requirement draft. Transient routing inconsistencies happen during IBGP convergence - because all routers are not updating their RIB at the same time. This - can lead to forwarding loops and then packet drops. The duration of - these transient micro-loops may depend on the IBGP topology (e.g. - number of Route Reflectors between ingress and egress ASBR), - implementation differences among router platforms (e.g. speed to - update the RIB and FIB, possibly the order in which prefixes are + because all routers are not updating their RIB and FIB at the same + time. This can lead to forwarding loops and then packet drops. The + duration of these transient micro-loops may depend on the IBGP + topology (e.g. number of Route Reflectors between ingress and egress + ASBR), implementation differences among router platforms (e.g. speed + to update the RIB and FIB, possibly the order in which prefixes are modified), forwarding mode (hop by hop IP forwarding versus tunneling). Transient forwarding loops can be avoided by performing only one IP lookup on BGP routes in each AS and by using tunnels (e.g. MPLS LSP) to send packets between ASBRs. As such, BGP/MPLS VPNs should be immune to such micro forwarding loops. 4. Terminology g-shut: Graceful SHUTdown. A method for explicitly notifying the BGP @@ -286,21 +286,21 @@ Affected router: a router reaching an affected prefix via a peering link undergoing maintenance. Initiator AS: the autonomous system of the g-shut initiator router. Neighbor AS(es): the autonomous system(s) of the g-shut neighbor router(s). -Requirements for the graceful shutdown of BGP sessions +Internet-Draft Requirements for the graceful shutdown of BGP sessions 5. Goals and requirements When a BGP session of the router under maintenance is shut down, the router removes the routes and then triggers the BGP convergence on its BGP peers. The goal of BGP graceful shutdown is to initiate the BGP convergence to find the alternate paths before the nominal paths are removed. As a result, before the nominal BGP session is shut down, all routers learn and use the alternate paths. Then the nominal BGP session can be shut down. @@ -331,53 +331,59 @@ simplicity of implementation and operation as shown in some of the following requirements. b) An Internet wide convergence is OPTIONAL. However if the initiator AS and the neighbor AS(es) have a backup path, they SHOULD be able to gracefully converge before the nominal path is shut down. c) The proposed solution SHOULD be applicable to any kind of BGP sessions (EBGP, IBGP, IBGP route reflector client, EBGP confederations, EBGP multi hop, MultiProtocol BGP extension...) and - any address family. If a BGP implementation allows closing a sub-set - of AFIs carried in a MP-BGP session, this mechanism MAY be applicable - to this sub-set of AFIs. - - Depending on the session type (EBGP, IBGP...), there may be some - variations in the proposed solution in order to fulfill the - requirements. + any address family. If a BGP implementation allows closing or + enabling a sub-set of AFIs carried in a MP-BGP session, this + mechanism MAY be applicable to this sub-set of AFIs. -Requirements for the graceful shutdown of BGP sessions + Depending on the kind of session, there may be some variations in the + proposed solution in order to fulfill the requirements. The following cases should be handled in priority: + +Internet-Draft Requirements for the graceful shutdown of BGP sessions + - The shutdown of an inter-AS link and therefore the shutdown of an eBGP session; - The shutdown of an AS Border Router and therefore the shutdown of all its BGP sessions. Service Providers and platforms implementing a graceful shutdown solution should note that in BGP/MPLS VPN as per [VPN], the PE-CE routing can be performed by other protocols than BGP (e.g. static - routes, RIPv2, OSPF, IS-IS...). This is out of scope of this - document. + routes, RIPv2, OSPF, IS-IS). This is out of scope of this document. d) The proposed solution SHOULD NOT change the BGP convergence behavior for the ASes exterior to the maintenance process, namely ASes other than the initiator AS and it(s) neighbor AS(es). e) An incremental deployment on a per AS or per BGP session basis - SHOULD be made possible. In case of partial deployment the proposed - solution SHOULD incrementally improve the maintenance process. The - solution SHOULD bring improvements even when one of the two ASes does - not support graceful shutdown. In particular, large Service Providers - may not be able to upgrade all of the deployed customer premises - access routers (CPE). + MUST be made possible. In case of partial deployment the proposed + solution SHOULD incrementally improve the maintenance process. + It should be noted that in an inter domain relation, one AS may have + more incentive to use graceful shutdown than the other. Similarly, in + a BGP/MPLS VPN environment, it's much easier to upgrade the PE + routers than the CE mainly because there is at least an order of + magnitude more CE and CE locations than PE and PE locations. As a + consequence, when splitting the cost of the solution between the g- + shut initiator and the g-shut neighbour the solution SHOULD favour a + low cost solution on the neighbour AS side in order to reduce the + impact on the g-shut neighbour. Impact should be understood as a + generic term which includes first hardware, then software, then + configuration upgrade.. f) Redistribution or advertisement of (static) IP routes into BGP SHOULD also be covered. g) The proposed solution MAY be designed in order to avoid transient forwarding loops. Indeed, forwarding loops increase packet transit delay and may lead to link saturation. h) The specific procedure SHOULD end when the BGP session is closed following the g-shut and once the BGP session is gracefully opened @@ -386,27 +392,26 @@ The duration of the g-shut procedure, and hence the time before the BGP session is safely closed SHOULD be discussed by the solution document. Examples of possible solutions are the use of a pre- configured timer, of a message to signal the end of the BGP convergence or monitoring the traffic on the g-shut interface... i) The solution SHOULD be simple and simple to operate. Hence it MAY only cover a subset of the cases. (As a consequence, most of the above requirements are expressed as "SHOULD" rather than "MUST") +Internet-Draft Requirements for the graceful shutdown of BGP sessions + The metrics to evaluate and compare the proposed solutions are, in decreasing order of importance: - The duration of the remaining loss of connectivity when the BGP session is brought down or up - -Requirements for the graceful shutdown of BGP sessions - - The applicability to a wide range of BGP and network topologies, especially those described in section 6; - The simplicity; - The duration of transient forwarding loops; - The additional load introduced in BGP (eg BGP messages sent to peer routers, peer ASes, the Internet). 6. Reference Topologies In order to benchmark the proposed solutions, some typical BGP @@ -426,47 +431,48 @@ We describe here some frequent EBGP topologies that SHOULD be supported by the solution. 6.1.1. 1 ASBR in AS1 connected to two ASBRs in the neighboring AS2 In this topology we have an asymmetric protection scheme between AS1 and AS2: - On AS2 side, two different routers are used to connect to AS1. - On AS1 side, one single router with two BGP sessions is used. +Internet-Draft Requirements for the graceful shutdown of BGP sessions + ' AS1 ' AS2 ' /----------- ASBR2.1 / ' / ' ASBR1.1 ' \ ' \ ' \----------- ASBR2.2 ' ' AS1 ' AS2 ' Figure 2. EBGP topology with redundant ASBR in one of the AS. The requirements of section 5 should be applicable to: - -Requirements for the graceful shutdown of BGP sessions - - Maintenance of one of the routers of AS2; - Maintenance of one link between AS1 and AS2, performed either on an AS1 or AS2 router. Note that in case of maintenance of the whole router, all its BGP - session needs to be shutdown. + sessions need to be gracefully shutdown at the beginning of the + maintenance and gracefully brought up at the end of the + maintenance. 6.1.2. 2 ASBRs in AS1 connected to 2 ASBRs in AS2 In this topology we have a symmetric protection scheme between AS1 and AS2: on both sides, two different routers are used to connect AS1 to AS2. ' AS1 ' AS2 ' @@ -478,31 +484,32 @@ ' ASBR1.2----------- ASBR2.2 ' AS1 ' AS2 ' Figure 3. EBGP topology with redundant ASBR in both ASes The requirements of section 5 should be applicable to: - Maintenance of any of the ASBR routers (in AS1 or AS2); + +Internet-Draft Requirements for the graceful shutdown of BGP sessions + - Maintenance of one link between AS1 and AS2 performed either on an AS1 or AS2 router. 6.1.3. 2 ASBRs in AS2 each connected to two different ASes In this topology at least three ASes are involved. Depending on which routes are exchanged between these ASes, some protection for some of the traffic may be possible. -Requirements for the graceful shutdown of BGP sessions - ' AS1 ' AS2 ' ASBR1.1----------- ASBR2.1 | ' | ' '''''|'''''''''' | ' | ' ASBR3.1----------- ASBR2.2 @@ -518,81 +525,93 @@ For instance if ASBR2.2 requires a maintenance affecting ASBR3.1, then ASBR3.1 will be notified. However we do not require for ASBR1.1 to be notified of the maintenance of the eBGP session between ASBR3.1-ASBR2.2. 6.2. IBGP topologies We describe here some frequent IBGP topologies that SHOULD be supported by the solution. +Internet-Draft Requirements for the graceful shutdown of BGP sessions + 6.2.1. IBGP Full-Mesh In this topology we have a full mesh of iBGP sessions: P1 ------ P2 | \ / | | \ / | | \ / | AS1 | / \ | | / \ | ASBR1.1---ASBR1.2 \ / \ / ''''''\''''/'''''''''''' \ / AS2 ASBR2.1 Figure 5. IBGP full mesh - When the session between ASBR1.1 and ASBR2.1 undergoes - maintenance, it is required that all IBGP peers of ASBR1.1 reroute - -Requirements for the graceful shutdown of BGP sessions - - traffic to ASBR1.2 before the session between ASBR1.1 and ASBR2.1 - is shut down. + When the session between ASBR1.1 and ASBR2.1 is gracefully + shutdown, it is required that all routers of AS1 reroute traffic + to ASBR1.2 before the session between ASBR1.1 and ASBR2.1 is shut + down. + Symmetrically, when the session between ASBR1.1 and ASBR2.1 is + gracefully brought up, it is required that all routers of AS1 + preferring ASBR1.1 over ASBR1.2 reroute traffic to ASBR1.1 before + the less preferred path trough ASBR1.2 is possibly withdrawn. 6.2.2. Route Reflector In this topology, route reflectors are used to limit the number of - IBGP sessions. + IBGP sessions. There is a single level of route reflectors and the + route reflectors are fully meshed. + +Internet-Draft Requirements for the graceful shutdown of BGP sessions P1 RR----- P2 RR | \ / | | \ / | | \ / | AS1 | \ / | | / \ | | / \ | | / \ | ASBR1.1 ASBR1.2 \ / \ / ''''''\''''''/'''''''''''' \ / \ / AS2 ASBR2.1 Figure 6. Route Reflector - When the session between ASBR1.1 and ASBR2.1 undergoes - maintenance, it is required that all BGP routers of AS1 reroute + When the session between ASBR1.1 and ASBR2.1 is gracefully + shutdown, it is required that all BGP routers of AS1 reroute traffic to ASBR1.2 before the session between ASBR1.1 and ASBR2.1 is shut down. + Symmetrically, when the session between ASBR1.1 and ASBR2.1 is + gracefully brought up, it is required that all routers of AS1 + preferring ASBR1.1 over ASBR1.2 reroute traffic to ASBR1.1 before + the less preferred path trough ASBR1.2 is possibly withdrawn. 6.2.3. hierarchical Route Reflector In this topology, hierarchical route reflectors are used to limit - the number of IBGP sessions. + the number of IBGP sessions. There could me more than levels of + route reflectors and the top level route reflectors are fully + meshed. -Requirements for the graceful shutdown of BGP sessions +Internet-Draft Requirements for the graceful shutdown of BGP sessions P1/hRR -------- P2/hRR | | | | | | AS1 | | | | P3/RR P4/RR | | @@ -603,39 +622,44 @@ ASBR1.1 ASBR1.2 \ / \ / ''''''\'''''''''/'''''''''''' \ / \ / AS2 ASBR2.1 Figure 7. Hierarchical Route Reflector - When the session between ASBR1.1 and ASBR2.1 undergoes - maintenance, it is required that all BGP routers of AS1 reroute + When the session between ASBR1.1 and ASBR2.1 is gracefully + shutdown, it is required that all BGP routers of AS1 reroute traffic to ASBR1.2 before the session between ASBR1.1 and ASBR2.1 is shut down. + Symmetrically, when the session between ASBR1.1 and ASBR2.1 is + gracefully brought up, it is required that all routers of AS1 + preferring ASBR1.1 over ASBR1.2 reroute traffic to ASBR1.1 before + the less preferred path trough ASBR1.2 is possibly withdrawn. 6.2.4. Confederations In this topology, a confederation of ASs is used to limit the number of IBGP sessions. Moreover, RRs may be present in the member ASs of the confederation. Confederations may be run with different sub-options. Regarding the IGP, each member AS can run its own IGP or they can all share the same IGP. Regarding BGP, local_pref may or may not cross the member AS boundaries. - A solution should support the shutdown of EBGP sessions between - member-ASs in the confederation in addition to the shutdown of EBGP - sessions between a member-AS and an AS outside of the confederation. + A solution should support the graceful shutdown and graceful bring up + of EBGP sessions between member-ASs in the confederation in addition + to the graceful shutdown and graceful bring up of EBGP sessions + between a member-AS and an AS outside of the confederation. -Requirements for the graceful shutdown of BGP sessions +Internet-Draft Requirements for the graceful shutdown of BGP sessions ASBR1C.1 ---------- ASBR1C.2 | | | | | AS1C | | | | | """|"""""""""""""""""""|""" | " | ASBR1A.2 " ASBR1B.2 @@ -655,41 +679,44 @@ Figure 8. Confederation In the above figure, member-AS AS1A, AS1B, AS1C belong to a confederation of ASs in AS1. AS1A and AS1B are connected to AS2. In normal operation, for the traffic toward AS2, . AS1A sends the traffic directly to AS2 through ASBR1A.1 . AS1B sends the traffic directly to AS2 through ASBR1B.1 . AS1C load balances the traffic between AS1A and AS1B - When the session between ASBR1A.1 and ASBR2.1 undergoes - maintenance, it is required that all BGP routers of AS1 reroute - traffic to ASBR1B.1 before the session between ASBR1A.1 and - ASBR2.1 is shut down. + When the session between ASBR1A.1 and ASBR2.1 is gracefully shutdown, + it is required that all BGP routers of AS1 reroute traffic to + ASBR1B.1 before the session between ASBR1A.1 and ASBR2.1 is shut + down. + Symmetrically, when the session between ASBR1A.1 and ASBR2.1 is + gracefully brought up, it is required that all routers of AS1 + preferring ASBR1A.1 over ASBR1.2 reroute traffic to ASBR1A.1 before + the less preferred path trough ASBR1.2 is possibly withdrawn. 7. Security Considerations At the requirements stage, this graceful shutdown mechanism is expected to not affect the security of the BGP protocol, especially if it can be kept simple. No new sessions are required and the additional ability to signal the graceful shutdown is not expected to bring additional attack vector as BGP neighbors already have the ability to send incorrect or misleading information or even shut down the session. +Internet-Draft Requirements for the graceful shutdown of BGP sessions + Security considerations MUST be addressed by the proposed solutions. In particular they SHOULD address the issues of bogus g-shut messages and how they would affect the network(s), as well - -Requirements for the graceful shutdown of BGP sessions - as the impact of hiding a g-shut message so that g-shut is not performed. The solution SHOULD NOT increase the ability for one AS to selectively influence routing decision in the peer AS (inbound Traffic Engineering) outside the case of the BGP session shutdown. Otherwise, the peer AS SHOULD have means to detect such behavior. 8. IANA Considerations @@ -723,27 +750,26 @@ "Graceful Shutdown in MPLS and Generalized MPLS Traffic Engineering Networks", RFC 5817 April 2010. [GR] S. Sangli, E. Chen, R. Fernando, J. Scudder, Y. Rekhter "Graceful Restart Mechanism for BGP", RFC 4724 January 2007. [Reliability] Network Strategy Partners, LLC. "Reliable IP Nodes: A prerequisite to profitable IP services", November 2002. http://www.nspllc.com/NewPages/Reliable_IP_Nodes.pdf +Internet-Draft Requirements for the graceful shutdown of BGP sessions + 10. Acknowledgments Authors would like to thank Nicolas Dubois, Benoit Fondeviole, Christian Jacquenet, Olivier Bonaventure, Steve Uhlig, Xavier - -Requirements for the graceful shutdown of BGP sessions - Vinet, Vincent Gillet, Jean-Louis le Roux, Pierre Alain Coste and Ronald Bonica for the useful discussions on this subject, their review and comments. This draft has been partly sponsored by the European project IST AGAVE. Authors' Addresses Bruno Decraene @@ -773,26 +799,27 @@ Zubair Ahmad Orange Business Services 13775 McLearen Road, Oak Hill VA 20171 USA Email: zubair.ahmad@orange-ftgroup.com Antonio Jose Elizondo Armengol Division de Analisis Tecnologicos + +Internet-Draft Requirements for the graceful shutdown of BGP sessions + Technology Analysis Division Telefonica I+D C/ Emilio Vargas 6 28043, Madrid -Requirements for the graceful shutdown of BGP sessions - E-mail: ajea@tid.es Tomonori Takeda NTT Corporation 9-11, Midori-Cho 3 Chrome Musashino-Shi, Tokyo 180-8585 Japan Email: takeda.tomonori@lab.ntt.co.jp