Multipath Traffic Engineering

Introduction Operators managing traffic within their networks have several tools, among them: Equal-cost Multipath (ECMP): balance traffic along multiple paths. This yields some resilience and some traffic management, as traffic can be load-balanced across multiple paths. To use ECMP effectively, one may have to adjust link metrics to allow multiple paths to have the same overall distance. Traffic Engineering (TE): state constraints for a path from an ingress router to an egress router, and let a path computation engine compute it. This gives much greater control over the nodes and links traversed, but is usually limited to finding a single path from ingress to egress . Multi-egress: allow traffic from an ingress router to a destination dst to use several egress routers, all of which have routes to that destination. dst may be an Internet prefix , a VPN prefix , an EVPN address , a VPLS site , or some other service destination. For BGP-signaled destinations, this requires that the BGP tie-breaking algorithm yield multiple results (rather than a single one), all of which become candidates for egress. Multi-ingress: consider multiple ingress routers as "equivalent" with respect to some of the traffic they sent to one or more egress routers. For example, an eBGP peer router or a VPN site may be multi-homed to several ingress routers, all of which would send such traffic to the same set of egress routers. describes requirements for MPLS-based TE, and thus is relevant to this memo. However, the authors focus on paths. This is further emphasized by the notion of a Label Switched Path, which is used to implement MPLS-based TE. RSVP-TE (), the protocol designed to meet the requirements of , builds a path from one ingress to one egress (for unicast traffic).

Existing Approaches to Multipathing in TE Networks Multipathing has traditionally been realized in TE networks by establishing multiple TE paths between a pair of endpoints and distributing traffic across them. For example, Section 5.6.5 of describes the use of multiple parallel traffic trunks between two nodes, with each trunk carrying a proportion of the aggregate traffic. In MPLS TE networks, this can be realized using multiple parallel TE LSPs between the same ingress and egress. Such LSPs may be explicitly provisioned by an operator or automatically created and maintained by an implementation and grouped under a common "container" construct. also notes that a TE tunnel may be supported by multiple parallel LSPs for purposes including load balancing. Traffic arriving at the ingress can then be distributed across these LSPs, while each LSP may be independently computed subject to its TE objectives and constraints. Segment Routing (SR) Policies provide another approach for realizing multiple TE paths with a common intent. An SR Policy is instantiated at an ingress and may have an active candidate path comprising a set of segment lists, where each segment list represents a source-routed path from the ingress toward the policy endpoint. Each segment list is associated with a weight, and traffic steered onto the SR Policy is distributed across the segment lists using weighted load balacing according to their relative weights. also defines a composite candidate path that can group multiple constituent SR Policies and distribute traffic among them according to their relative weights. (Note that SR-MPLS and SRv6 do not inherently support bandwidth reservations and management, which is a key requirement of ; this can be augmented using an external controller to do the entire task of bandwidth management. This function is an intrinsic aspect of RSVP-TE.) A common characteristic of these approaches is that for a given packet (or flow), one of the explicitly represented TE alternatives is selected, and traffic is sent to it, at the ingress. In the parallel-LSP model, the ingress distributes traffic across a set of TE LSPs. In the SR Policy model, the ingress distributes traffic across the segment lists of the active candidate path, or across constituent SR Policies in the case of a composite candidate path. Thus, the TE multipath decision is primarily (or entirely) an ingress function. The multipathing described in this document differs from this model. Rather than representing multipathing as a collection of parallel end-to-end TE paths with traffic distribution primarily at the ingress, MPTE represents the TE forwarding construct as a Directed Acyclic Graph (DAG). A junction within the DAG may have multiple outgoing branches and may independently distribute traffic among those branches according to the load shares associated with the DAG. Consequently, weighted multipath forwarding can occur at multiple junctions within the TE construct, rather than only when traffic enters it. This distinction allows paths through the TE construct to diverge and subsequently converge while naturally sharing forwarding state and network resources within the DAG. It also avoids requiring each possible end-to-end path through the multipath construct to be represented and managed as an independent TE path. In order to satisfy the constraints, TE often uses non-shortest paths. To do so without looplng packets, a tunnel is used. Such tunnels have to be signaled. RSVP-TE is a signaling protocol for MPLS-based tunnels. In this memo, we introduce a new tool: multipath TE (MPTE). This allows an operator to specify constraints for paths (as in TE), specify multiple ingresses and egresses, and use multiple paths from ingress to egress. Effectively, MPTE combines the advantages of the four tools above. The resulting set of paths from ingresses to egresses is a Directed Acyclic Graph (DAG), here called an MPTE DAG or MPTED. Finally, this memo allows the use of multiple types of tunnels. The main contribution of this memo is the notion of a (multipath) unicast tunnel across an MPTED, called an MPTE tunnel or MPTET, and an overview of how they are created. Protocols for provisioning such tunnels will be specified in companion documents. Another companion document defines how to distribute MPTE capabilities in an IGP so that entities computing MPTEDs can know which nodes to include in the DAG.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Definition of Commonly Used Terms This section provides definitions for terms and abbreviations that have a specific meaning to the MPTE protocol and that are used throughout this memo.

constraints:: desired properties of paths between ingresses and egresses.
constrained shortest path first (CSPF):: A modification to SPF to take into account TE constraints.
directed acyclic graph (DAG):: a directed graph that has no cycles. The result of a multipath SPF or CSPF computation is a DAG.
directed graph:: a set of nodes and directed links. A network is represented by a directed graph.
egress:: an end node of an MPTE DAG.
equal-cost multipath (ECMP):: a DAG consisting of shortest paths from an ingress to an egress.
flow-aware load balancing (FALB):: load balancing that maps packets that belong to a given flow onto the same path. What consitutes a flow depends on the type of traffic; for IP traffic, a flow is typically defined by the 5-tuple <source IP, dest IP, protocol type, source port, dest port>.
ingress:: a starting node of an MPTE DAG.
label-switched path (LSP):: an MPLS tunnel from an ingress to one or more egresses.
link:: A (directed) edge between two nodes. A pair of nodes may have 0 or more links between them. A link between nodes u and v will be denoted by (u, v, i), where i is u's oif for the link. A link may have associated attributes, in particular, a metric.
load balancing (LB):: a method whereby traffic to an egress is distributed among multiple next hops at each node along the DAG.
metric:: a positive number describing the contribution of a link to the oveall path length.
MC:: MPTED computer: the entity computing the MPTED, typically the ingress (if there is a single ingress) or a Path Computation Element
MPTE:: multipath TE with path constraints (including a slack) using nECMP paths from an ingress to one or more egresses.
MPTED:: an MPTE DAG resulting from CSPF-type computation on MPTE constraints.
MPTEP:: MPTE protocol: the protocol used to signal MPTETs.
MPTET:: MPTE tunnel: the signaled (and hence, forwarding) entity associated with an MPTED. Note: this terminology dates back to , where the signaled entity was a simple path with one ingress and one egress. MPTE constructs a DAG, not a path. This document will use the term DAG (MPTED) (rather than path) for the result of a CSPF computation, and the term tunnel (MPTET) for the signaled entity. If it helps visualization, imagine a multi-branched tunnel in a cave with multiple entry and exit points.
non-equal-cost multipath (nECMP) (generally qualified by "with slack s"):: a DAG of paths from an ingress u to an egress v that are within s of min(u, v).
node:: a vertex of a graph. A node may have associated attributes.
outgoing interface (oif):: a unique number (oif) assigned by a node for each outgoing link it has.
Path Computation Element (PCE):: an entity capable of performing CSPF on behalf of another node, the path computation client.
path length:: the sum of the metrics of the links that constitute path p, denoted by len(p)
shared risk group (SRG):: nodes and/or links that share "risk" (e.g., have a common power feed, or use a common fiber conduit)
shortest path:: a path between a pair of nodes u, v with minimum length. The set of shortest paths between u and v is a DAG, denoted by sp(u, v). The length of a shortest path from u to v is denoted by min(u, v)

shortest path first (SPF): an algorithm for computing the shortest path DAG from an ingress to an egress; typically refers to Dijkstra's algorithm for computing shortest paths between a given pair of nodes, or pairwise between all nodes.

signaling source (SS):: an entity responsible for signaling an MPTET
slack:: a path p from u to v has slack s if len(p) = min(u, v) + s.

traffic engineering (TE): a methodology for mapping traffic trunks to given paths or DAGs across a network.

traffic trunk:: a unidirectional aggregate of traffic flows from an ingress to a set of egresses that is treated identically in the forwarding plane.
tunnel originator (TO):: entity having the specifications of the MPTET

Overview Consider :

Multipathing

ECMP (slack 0) from node 0 to node 5 There are 4 ECMP paths from node 0 to node 5: 0-2=3-5 (2 paths) 0-2-4-5 0-4-5 These 4 distinct paths all have length 300.

nECMP from node 0 to node 5 with slack 10 There are 7 nECMP paths with slack 10 to node 5: 0-2=3=5 (4 paths) 0-2-4-5 0-4-5 0-6-7-5 These 7 paths have lengths 300 or 310. Thus, allowing nECMP paths a slack of 10 has yielded 3 additional paths, which provide increased diversity and load balancing, and possibly decreased congestion.

Multipathing from node 0 to egresses {5, 8} If, for some traffic trunk that starts at node 0, nodes 5 and 8 are equally good as egresses, then one can compute an ECMP DAG from 0 to {5, 8}; this yields 4 paths to 5 and 6 paths to 8, for a total of 10 paths this traffic trunk can take. Similarly, a nECMP DAG to {5, 8} with slack 10 has 15 paths, whereas one with slack 5 has the same 10 paths as with slack 0.

MPTED from ingresses {0, 1} to egresses {5, 8} If traffic from node 0 to nodes {5, 8} and from node 1 to nodes {5, 8} have common characteristics, it may make sense to compute a single DAG from {0, 1} to {5, 8}. Doing so allows the operator to view this entire DAG as one logical entity; a nice side benefit is reduced control and data plane state due to state sharing.

Load balancing Nodes in a netword have a Forwarding Information Base (FIB). A FIB maps a packet's destination address da to one or more "next hops". When a packet with address da arrives at n, n sends the packet to one of the next hops. n typically will distribute packets in a given ratio among the next hops. This is load balancing. The main goal of ECMP/nECMP is to supply as many nodes as possible in the MPTED with multiple next hops on which to forward the traffic trunk. At such nodes, traffic belonging to the trunk can be distributed among the next hops instead of going to a single next hop. This has the potential to reduce congestion and provide better utilization of available links.

Flow-aware load balancing When load balancing packets from a traffic trunk, it is often required that packets from a given flow be sent to the same next hop. This improves the probability of in-order delivery of packets in that flow, which is important for certain types of traffic. This is called flow-aware load balancing (FALB). The most common flow in IP traffic is defined by a 5-tuple (typically from the innermost IP header) consisting of the source IP address, the destination IP address, the protocol, the source port and the destination port. A hash (usually 16- or 20-bit) of this 5-tuple is called the packet's entropy. There are two common ways to achieve FALB of IP traffic. One is to do a "deepish" packet inspection (dPI), find the relevant 5-tuple, and use that to compute the packet's entropy. The entropy is then used to ensure that packets in the flow are sent to the same next hop. This memo suggests sending TE traffic over a tunnel (see {tunnels}); this can make the identification of IP flows expensive and error-prone. Another way of accomplishing this is to insert the entropy in the tunnel header. Many of the tunnels suggested in this memo have such a field. The ingress is in a good position to identify flows, and, when encapsulating the packet into the tunnel, can insert the entropy in the header. The heavy lifting of identifying flows is thus placed on the ingress. Transit nodes can simply use the entropy field to correctly map packets in a flow to the same next hop, thus ensuring FALB.

Per-packet load balancing FALB is often required and is a good default behavior, especially as end applications may be expecting packets in a flow to be delivered in order. However, FALB has the issue that it attempts (statistically) to place roughly the number of flows in the given ratio on the outgoing links; that may not place traffic in the same ratio, as flows need not carry the same amount of traffic. In some cases (typically when configured to), one can do per-packet load balancing (PPLB), meaning that load balancing is no longer flow aware. This can be done when the end applications do not require packets in a flow to be in order, or if some (bookended) devices outside the network put the packets back in order before delivering them to the applications (typically by addind a sequence number). When feasible, PPLB gives much better load distribution, and is currently the subject of investigation, implementation and standardization. One can achieve this by configuring each router in the DAG to do PPLB for the traffic trunks in the MPTET, or more simply by the ingress router assigning entropy at random to each packet it places in the MPTET. The latter approach keeps the decision of which DAGs (and corresponding traffic trunks) should be flow-aware and which not at the ingress; all other nodes simply do what the entropy fields tells them to do.

Constraints Constraints are an intent-based specification of acceptable paths that a traffic trunk may take from ingress(es) to egress(es). Constraints are thus an abstract way to control the resources that a particular traffic trunk uses. One way to do this is to add "resource class attributes" or "colors" to links, and then specify "include" and "exclude" sets. An include set means that all links that a path traverses must contain at least one element of the include set. An exclude set means that no link in the path can contain any color from the exclude set. Another way is to specify a (maximum) bandwidth that a traffic trunk can carry. This means that all links in the path must have that much available capacity. Packets exceeding the bandwidth can be forwarded normally, marked as droppable, or dropped. Let's add some simple constraints to our DAG. We associate the color red to one of the links from 2 to 3, and to the shorter of the links from 6 to 7. Then, we constrain the paths to "include red or blue", meaning only use links with either color red or blue. This yields the following: ECMP from node 0 to node 5 with constraints "include red or blue" yields a single path.

Protection One very useful aspect of TE is the ability to specify that a path must be link- or node- or shared-risk-disjoint from another path. That means that the two paths do not have links or nodes or "shared risk groups". Additionally, one can build protection paths for an existing path to protect against link or node failures . This is especially important as traditional TE currently takes a single path through the network, meaning that a link or node failure will result in dropped traffic until the TE path is restored. While not quite as crucial in the case of an MPTET, since ideally, there will be multiple nexthops at each node, there will be cases where a node has a single next hop, or all next hops share a common failure mode. Identifying these cases and building protection paths for such nodes will be described in a future version of this memo.

Tunnels The shortest path first algorithm is an easy-to-implement and very efficient algorithm whereby all routers in a network can agree on the path that a packet to a particular destination should take. That means if all routers are agreed (roughly) on the topology and metrics of the network, they will forward packets in a loop-free manner to all destinations -- without the need for signaling or tunnels. However, an MPTED will not contain the same paths -- some paths may be rejected as they don't satisfy the constraints; other paths may be used even though they are not shortest paths. Thus, to route packets in a traffic trunk over a computed MPTED, a tunnel is typically used. This tunnel will have to be signaled to the MPTED nodes. The tunnel may be MPLS- or IP-based. A few things are important about tunnel headers: whether they carry an entropy field (EF), whether they have a "discriminator" (D) that allows multiple tunnels between an ingress-egress pair, whether they allow multiple egresses (ME), and whether they allow multiple ingresses (MI). These will be discussed in the description of the tunnels below. In the memo, we consider the following tunnel types: IP-in-IP: encapsulation allows the creation of an "outer" IP header (IPv4 or IPv6) to carry a payload packet (which is typically an IP payload). The outer IP header's protocol field indicates the "protocol" of the inner payload packet. The outer header of IP-in-IPv4 tunnel doesn't contain an EF; transit nodes can either spray packets across outgoing next hops, attempt to do dPI, or use the same next hop for all packets. For IP-in-IPv6, the outer header's flow label can be used for the EF. To accommodate ME, the egresses have to have the same (anycast) IP address which would be used as the destination IP of the tunnel. MI is not possible. GRE: Generic Routing Encapsulation. We include in this definition and with the Key Present (bit 2) set to 0. This is similar to IP-in-IP; however, the payload is not required to be IP. There is no EF in the header. D, ME and MI same as for IP-in-IP. GRE-E: GRE with Key Present; the Key value is the EF. D, ME and MI same as for IP-in-IP. GRE6: GRE with IPv6 addresses. The EF is the Flow Label field of the IPv6 header. D, ME and MI same as for IP-in-IP. G-in-U: GRE-in-UDP . The UDP source port is the EF; the GRE Key, if present, can be ignored from a load balancing point of view. D, ME and MI as in IP-in-IP. MPLS-in-UDP . The UDP source port is the EF; D, ME and MI as in IP-in-IP. SRv6 . The EF is the Flow Label in the IPv6 header. SigLab (signaled label switching). The labels to be used are signaled. Signaling proceeds from egress(es) to ingress(es). An entropy label can be used as the EF. At each node, a different label is used for each MPTED; this is the discriminator. ME and MI are both allowed. StatLab (static label). A single statically-assigned label defines the tunnel throughout the MPTED. Here, a block of MPLS labels is given to a label allocator; these labels MUST NOT be allocated by any node in the network. EF, D, ME and MI are as for SigLab. The MPTED computer (MC) must interact with the allocator when creating or deleting an MPTED.

Backward Compatibility Introducing a new idea to the network (and thus new protocols, new extenstion and new software) is typically done incrementally. Thus, in a network transitioning to MPTE, there will be some nodes that are MPTE-capable, and others that are not. In above, if node 4 is not MPTE-capable, it can either be left out of the MPTED, or a "classical" tunnel can be constructed from (say) node 2 to node 5, allowing hybrid paths 0-2-(4)-5 and 0-2-(4)-5-8 for a DAG from {0} to {5, 8}. The signaling specs will say whether this is possible, and if so, how it can be achieved.

Operation Here are the steps to create an MPTE tunnel: Define the traffic trunk for the MPTET. Examples include "BGP destinations with community xyz" or "gold class traffic belonging to VPN foo" or "AI workload bar from DC A to DC B". Define the constraints of the traffic trunk, including: ingresses, and the bandwidth entering the DAG at each ingress; egresses; metric to minimize (perhaps with slack) -- this could capture delay or fiber length; criteria of acceptable nodes and links for the DAG, including link colors and shared risk groups (SRGs). This information is given to the Tunnel Originator (TO). The TO sends this information to the MPTE Computer (MC). The MC computes a DAG that satisfies the constraints. The DAG consists of a set of junctions; these are sent to the Signaling Source (SS). The SS instantiates the MPTET by sending signaling messages to all the junctions. When ready, the SS tells each ingress that the MPTET meeting the DAG constraints is ready for traffic. The ingresses map traffic matching the traffic trunk to the MPTET. Computation (possibly using a variant of CSPF) of an MPTED is done by the MC, which may be an ingress or a PCE . (This memo does not specify such an algorithm.) Signaling primarily occurs between the SS and each junction node. Auxiliary signaling may occur among junction nodes.

MPTED In this memo, a node is identified by its IP loopback address. A link from node u to node v is identified by u's loopback address and its (4-octet) outgoing interface index (oif), a unique identifier for the link allocated by u. oifs are usually exchanged in the TE extensions of an IGP. (A link also has a (4-octet) incoming interface index, the iif. For neighbors u and v, the correlation between u's oif and v's iif is typically done by the IGP. iifs are not used in this memo.) For now, this memo only deals with point-to-point links; a future revision will describe the use of multi-access links. An MPTED is identified by a unique (4-octet) ID (the MID) assigned to the MPTED by the MC. As an MPTED can change over its lifetime, it is assigned a version number starting at 0 and incremented every time the MPTED is recomputed. Thus, a full MPTED ID (the FID) consists of <MC, MID, version>. An MPTED consists of two or more "junction nodes". A junction node can have one of five types: a pure ingress node has zero incoming links and one or more outgoing links in the MPTED. Traffic routed on a MPTED enters at the ingress. a pure egress node has one or more incoming links and zero outgoing links in the MPTED. Traffic routed on a MPTED leaves at an egress. a transit ingress node where traffic can either enter the MPTED or arrive from another ingress node to continue on in the MPTED. a transit egress node where traffic can either exit the MPTED or go on to another egress node. a "regular" junction node has one or more incoming links and one or more outgoing links. Traffic does not enter or leave at such a node: it comes from a phop and goes to an nhop. A junction node v consists of v, its previous hops (phops) and its next hops (nhops). A phop is specified by an incoming link of v: (u, v, oif1); an nhop by an outgoing link of v: (v, w, oif2). Note that, since links are point-to-point, it is sufficient to specify (u, oif1) ((v, oif2)) for a phop (nhop, respectively). The nodes u (and w) are loosely referred to as a phop (and nhop) of v, although strictly speaking the link should be included. A pure ingress has no phops and a pure egress has no nhops. The MPTED is broken down into a set of junction nodes. A junction node v is specified by: bandwidth (coming in to and going out of v) a list of phops of v a list of nhops of v, with corresponding load balancing shares

Tunnel Provisioning A designated entity, the Tunnel Originator (TO), is given the specifications of the MPTET: the ingresses, the egresses and the constraints. The TO is typically one of the tunnel ingresses or a PCE. The TO sends the tunnel specification to the MC. The MC computes the MPTED (as a list of junctions) and returns this to the TO. The TO then sends the list of junctions to the Signaling Source (SS) which provisions the tunnel. Note that TO, MC and SS are functional blocks; they may reside on separate nodes or co-reside on the same node. For example, a single node X may be the TO and SS but decide to delegate computation to a (remote) PCE. X then gets the results via PCEP and signals the tunnel. Other permutations are possible.

Signaling Overview The SS signals the creation or update of an MPTE tunnnel by sending to each junction node v a JUNCTION message consisting of: the MPTET ID the junction node specification the tunnel type some flags After v parses this specification, for all tunnel types other than SigLab, it installs FIB state for the junction. For tunnel type SigLab, v allocates an incoming MPLS label L_u for each phop u (u, oif), and sends a LABEL message to u containing: the MPTET ID the phop (u's loopback + u's oif for the link) the allocated label L_u u records label L_u as part of its own junction state. When v receives a LABEL message from all its nhops, it installs swap state in its LFIB.

Signaling Several signaling protocols are being extended to provision MPTETs: RSVP-TE , PCEP and BGP , among others. The details of each will be specified in companion documents; this memo restricts itself to an overview of the common elements.

Message Flow Provisioning messages (to create, update and delete a tunnel) are sent from the Signaling Source (SS) to each junction node (including possibly other ingresses). Notifications are sent from each junction node to the SS to send updates on the state of that node with respect to the MPTET. Label messages (when needed) are sent hop-by-hop from egresses to their phops and further upstream in an ordered fashion. In special scenarios, a node may send a message to one or more of its nhops.

Message Types

JUNCTION A JUNCTION message contains the following information elements:

MPTET ID:: a unique identifier for an MPTE tunnel. This usually consists of the TO ID and a unique ID in the namespace of the TO. It also includes a version number to distinguish among instances of a tunnel as it is undergoes updates. The companion signaling documents will describe the MPTET ID in more detail.
Tunnel Type:: various types of tunnels are used, so each node must be told which type of tunnel this MPTET consists of.
Tunnel Information:: provides details for the MPTET. For example, for an MPLS tunnel with a statically assigned label, the Tunnel Information is the label. For IP-based tunnels, the Tunnel Information is the source and destination IP addresses (plus optional other information).
Junction Bandwidth:: specifies the bandwidth incoming to the junction in Megabits per second (Mbps).
nhop share:: a 2-octet share of the outgoing bandwidth per nhop. A Junction should attempt to send a ratio of (share n)/(sum (share i)) of the incoming bandwidth to nhop #n.

LABEL A LABEL message is used to let each junction know what to use to forward packets in the MPTET. A LABEL message is sent from an egress junction node to each of its phops. A pure ingress node never sends a LABEL message as it has no phops. The LABEL message carries the MPTET ID and a label, which can be an MPLS label or an IP destination address.

NOTIFY A NOTIFY is sent from a junction node to the SS to let the SS know the state of the MPTET at that node. This could be the labels it assigned to its phops, or error conditions.

Forwarding State From a forwarding point of view, an ingress's job is to: identify the traffic trunk, i.e., the set of packets that are to be sent via the MPTET; encapsulate the packets in the signaled tunnel type; forward the packet to the ingress's next hops, in accordance with the computed weights. FIB entries have a lookup portion (the "routes") and a next hop portion. In all cases, the next hop at junction J is a weighted list of J's nhops as specified by the SS in the JUNCTION message. For an ingress node, the routes define the traffic trunk meant to be carried by the MPTET. For a non-ingress node v, the routes identify the MPTET from its phops.

IP Tunnels If the MPTET is an IP tunnel, all junctions will need to know the MPTET type, the MPTET source IP src and destination IP dst.

MPTET Type MPTET signaling determines the tunnel type. This is typically configured at the TO.

MPTET Destination IP Address dst Consider first the case that the MPTET has a single egress E. Here are two ways to support multiple MPTETs to the egress: E has an Egress Address Block (EAB) consisting of a set of IPv4 and/or IPv6 prefixes and/or SRv6 locators . E chooses an element of the appropriate EAB, dst, and signals dst in its LABEL message to its phops. E uses an IP tunnel that has a discriminator field D, and uses that in conjunction with its loopback dst. The LABEL message consists of <dst, D>. If there are multiple egresses, dst MUST be a common (anycast) address amomg all the egresses. One of the egresses, E, is chosen to pick dst (using either of the methods above), and signals this to its phops, as well as all other egresses.

MPTET Source IP Address src If there is a single ingress I, src is set to I's loopback. If there are multiple ingresses, and dst was chosen from an EAB, each ingress sets src to its own loopback. If there are multiple ingresses, and a discriminator D is being used, there MUST be a designated ingress I who picks src, and sends it to all other ingresses.

IP MPTET FIB Entries In the case where an EAB was used to pick dst, the route consists simply of dst. In the case where a discriminator D is being used for the MPTET, the route consists of <dst, src, D> at each junction. Otherwise, the route is <dst, src>. (Note: this last case only allows one MPTET between a src-dst pair, for each tunnel type.)

Static Label For a MPTET with a "statically assigned" label (typically by a PCE), the route consists of the assigned label, and the nexthops are a set of interfaces. In the simplest case, the entire DAG has a single label; if so, the label operation is null. A variant allows for different controller-assigned labels for each junction node; in this case, the forwarding state is as for "signaled" labels, where the incoming label is swapped to the correct outgoing label.

Signaled Label For signaled labels, the routes for node v are the labels v sent to its phops. Each nexthop is a swap of the incoming label to the label sent by v's nhops.

Graceful Restart A node N is capable of Graceful Restart if a) it can maintain control plane state across restarts; and b) it can maintain forwarding state across restarts. If N is capable of Graceful Restart, an MPTE DAG going through N can continue functioning while N restarts. While N is restarting, new JUNCTION/LABEL messages will be dropped or ignored; new MPTE DAGs passing through N will not be established. Once restart is complete, N will send an OPEN message and re-establish connections will all its peers (or all the MPTEP Reflectors). Thereafter, N can participate in new DAGs passing through it by processing received JUNCTION messages. More details will be described in a future version.

IANA Considerations None. The related protocol documents will have IANA requirements.

Security Considerations TBD

Acknowledgements Many thanks to Kostas Zorbadelos for his careful reading and useful suggestions, and his continued interest in this technology. Many thanks as well to the MPTE group at HPE Networking for the active discussions that advance the technology and make it implementable.