Network Working Group K. Majumdar Internet Draft Upscale AI Intended status: Standards Track L. Dunbar Expires: February 24, 2027 Futurewei V.Kasiviswanathan NextHop AI A. Ramchandra Google A. Choudhary Cisco August 24, 2026 Multi-segment SD-WAN via Cloud Backbone draft-ietf-rtgwg-multisegment-sdwan-16 Abstract This document describes a method for seamlessly interconnecting geographically separated SD-WAN segments via a Cloud Backbone without requiring Cloud Gateways (GWs) to decrypt and re-encrypt traffic. By encapsulating IPsec- encrypted payloads within GENEVE headers (RFC 8926), the approach enables Cloud GWs to forward encrypted traffic directly between distant Customer Premises Equipment (CPEs). This reduces processing overhead, improves scalability, and preserves the confidentiality of enterprise data while ensuring secure and efficient multi-segment SD-WAN connectivity. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. xxx, et al. Expires February 24, 2027 [Page 1] Internet-Draft Multi-segment SD-WAN Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on February 21, 2027 . Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction..............................................3 2. Conventions and Definitions...............................5 3. Use Cases.................................................6 3.1. Multi-segment SD-WAN via a Single Cloud GW...........6 3.2. Multi-segment SD-WAN via Cloud Backbone..............8 3.3. Traffic Steering Challenges in Multi-Segment SD-WAN..9 4. Data Plane encoding for SD-WAN Transit...................10 4.1. Multi-Segment SD-WAN Option Class...................11 4.2. SD-WAN Tunnel Endpoint Sub-TLV......................12 4.3. SD-WAN Tunnel Originator Sub-TLV....................14 Dunbar, et al. Expires Dec 24, 2027 [Page 2] Internet-Draft Multi-segment SD-WAN 4.4. Egress GW Sub-TLV...................................15 4.5. Permitted Regions Sub-TLV...........................16 4.6. Excluded Regions Sub-TLV............................18 4.7. HMAC Authentication Value Sub-TLV...................19 5. Packet Header Processing.................................21 6. Error Handling...........................................22 7. Control Plane considerations.............................23 7.1. Control Plane for CPEs..............................23 7.2. Control Plane between CPEs and Cloud GWs............24 8. Observability Consideration..............................24 9. Security Considerations..................................25 9.1. Threat Analysis.....................................25 9.2. HMAC-based Integrity and Authentication.............26 9.3. Additional Integrity and Authentication.............28 10. Manageability Considerations............................29 11. Applicability Considerations............................30 12. IANA Considerations.....................................31 13. References..............................................32 13.1. Normative References...............................32 13.2. Informative References.............................33 14. Acknowledgments.........................................35 Appendix A: Illustration of Packets through Cloud GWs.......35 A.1 Single Hop Cloud GW.....................................35 A.2 Multi-hop Transit GWs...................................38 1. Introduction Enterprises are increasingly turning to Software-Defined Wide Area Network (SD-WAN) to connect Customer Premises Equipment (CPE)s with cloud services, as discussed in detail in [Net2Cloud]. Each SD-WAN segment typically connects a CPE to its nearest Cloud Gateway (GW). Some of this traffic terminates at the cloud services and must be decrypted by the Cloud GW. Other traffic is destined for remote CPEs located in different geographic regions and only require forwarding across a Cloud Backbone, without decryption. Figures 1 and 2 in Sections 3.1 and 3.2 illustrate the CPEs, Cloud GWs, Cloud Backbone, and the connectivity considered in this document. Multi-segment SD-WAN refers to the architecture in which two or more SD-WAN segments are interconnected via a Cloud Backbone. This model enables traffic that originates in one Dunbar, et al. Expires Dec 24, 2027 [Page 3] Internet-Draft Multi-segment SD-WAN SD-WAN segment to reach a distant CPE through transit Cloud GWs without decryption. It supports hybrid traffic handling: local cloud-bound traffic is decrypted by the Cloud GW, while CPE-to-CPE traffic is forwarded securely across the backbone. Interconnecting these SD-WAN segments via a Cloud Backbone provides several key benefits: a) Seamless connectivity - Enterprises can integrate geographically dispersed SD-WAN segments into a unified network without complex manual configurations. b) Scalability - The Cloud Backbone's elasticity accommodates increased traffic demands without requiring extensive on- premises infrastructure. c) Simplified operations - Centralized orchestration streamlines policy enforcement and network management across all segments. The challenges and motivations for this architecture are further detailed in [Net2Cloud], which outlines issues enterprises face when interconnecting branch sites with dynamic workloads in third-party Cloud Datacenters (DC), particularly when leveraging existing VPN infrastructure. A key requirement for Cloud Backbone interconnecting (stitching) multiple SD-WAN segments is the ability to forward encrypted traffic across the Cloud Backbone without requiring decryption at Cloud GWs. Since IPsec Security Associations (SAs) are established end-to-end between CPEs, Cloud GWs cannot access the payload for routing. Introducing an additional IPsec tunnel layer between CPE and Cloud GW just for routing purposes is inefficient-it adds processing overhead, increases latency due to decryption and re- encryption, and imposes scalability limits due to cloud provider restrictions on IPsec capacity per GW instance. This document defines a GENEVE-based method that avoids these inefficiencies. SD-WAN CPEs encapsulate IPsec-encrypted packets with GENEVE headers [RFC8926] that include Sub-TLVs to signal when traffic should transit the Cloud Backbone without decryption. This enables Cloud GWs to forward encrypted Dunbar, et al. Expires Dec 24, 2027 [Page 4] Internet-Draft Multi-segment SD-WAN traffic efficiently to remote CPEs, without accessing the payload. The result is secure, low-latency, and scalable interconnection of geographically distributed SD-WAN segments using the Cloud Backbone. 2. Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. The IP addresses used in this document are for illustrative purposes only. IPv4 examples use addresses from the documentation blocks defined in [RFC5737] (192.0.2.0/24 and 198.51.100.0/24). All mechanisms described apply equally to IPv6, where equivalent examples would use addresses from the documentation prefix defined in [RFC3849] (2001:db8::/32). The following acronyms and terms are used in this document: Cloud Backbone: The global, private network infrastructure operated by a cloud provider that interconnects its regions, zones, and points of presence. Cloud DC: Off-Premises Data Center, managed by the third party, that hosts applications, services, and workload for different organizations or tenants. CPE: Customer (Edge) Premises Equipment. Ingress Cloud GW: The Cloud GW through which traffic enters the Cloud Backbone. Egress Cloud GW: The Cloud GW through which traffic exits the Cloud Backbone toward the destination CPE. OnPrem: On Premises data centers and branch offices. RR Route Reflector. Dunbar, et al. Expires Dec 24, 2027 [Page 5] Internet-Draft Multi-segment SD-WAN Region: A geographic or administrative area defined by the Cloud Backbone provider and used for placement, routing, or policy enforcement. SA IPsec Security Association SD-WAN Software-Defined Wide Area Network. An overlay connectivity service that optimizes transport of IP Packets over one or more Underlay Connectivity Services and determining forwarding behavior by applying Policies to them. [MEF-70.1] Transit Node: A node within the Cloud Backbone through which traffic may transit between the Ingress and Egress Cloud GWs. VPN Virtual Private Network. Zone: A subdivision of a Region representing a distinct deployment or failure domain within the Cloud Backbone. 3. Use Cases 3.1. Multi-segment SD-WAN via a Single Cloud GW Enterprise branches, such as CPE1 and CPE2 in Figure 1, with established SD-WAN paths to a Cloud GW for accessing cloud services can also use the Cloud GW to interconnect with one another. For customers that have already chosen SD-WAN, stitching SD-WAN segments through a Cloud Gateway provides a way to extend policy enforcement and traffic control across branches using their existing SD-WAN-to-cloud connectivity, while avoiding the need to establish separate direct branch to branch SD-WAN paths. This approach is beneficial for several reasons: Dunbar, et al. Expires Dec 24, 2027 [Page 6] Internet-Draft Multi-segment SD-WAN - The public Internet path from a branch to its nearby Cloud GW is typically shorter and more controlled than a public Internet path between branches, which can reduce exposure to unpredictable performance, limited bandwidth, and security risks. - Centralized enforcement of enterprise security policies can be enabled through cloud-hosted services. Traffic destined to cloud-resident applications can be decrypted for full inspection (e.g., firewall, threat detection), while CPE- to-CPE traffic that remains IPsec-encrypted can still benefit from header- or flow-based functions-such as DDoS mitigation, rate limiting, anomaly detection, and SLA/usage analytics-especially when the same CPE also sends traffic terminating in the cloud. - Cloud platforms often offer enhanced monitoring, proprietary threat detection tools, and analytics services that can inspect and respond to suspicious traffic crossing segments. Dunbar, et al. Expires Dec 24, 2027 [Page 7] Internet-Draft Multi-segment SD-WAN +-----------------+ | Cloud | | +----+ +----+ | +--------|Edge+--+ GW | | Direct | +-----+----+ +-/\-+ | Connect | | | / \ | | | +---------/----\--+ | | / \SD-WAN Path CPE<->GW PrivateLines / \ | +----------/-----+ IPsec Tunnel +-------+ / | \ | / | \ ++--/+ | +-\--+ |CPE1| +----+CPE2| +----+ +----+ Figure 1 Multi-Segment SD-WAN stitching via a Cloud GW Note: For clarity, each line in this figure represents connectivity that may consist of multiple parallel paths. Multiple paths are not shown to avoid excessive complexity in the illustration. 3.2. Multi-segment SD-WAN via Cloud Backbone For geographically distant enterprise branches that have established SD-WAN paths to their respective Cloud GWs for accessing cloud services, the Cloud Backbone provides an efficient way to interconnect these branches, as shown in Figure 2. As outlined in the Introduction section, this approach enhances network integration, supports dynamic scaling, and simplifies overall management, making it well- suited for multi-segment SD-WAN deployments across different regions. Dunbar, et al. Expires Dec 24, 2027 [Page 8] Internet-Draft Multi-segment SD-WAN +---------------------------------------+ | Cloud Backbone | +-------+-----+ +----+ +-----+ | | +-----| Edge+--+ GW1|============| GW2 | | Direct| | | +-----+ +|--|+ +--|--+ | Connect| | | | | | | | | +------------|--|----------------|------+ | | IPsec +-----+ Private Lines / \ |CPE10| | +-------------/--+ \ +-----+ | / | \ | +----+ | +----+ +---------+CPE1| +--+CPE2| +----+ +----+ Figure 2 Multi-Segment SD-WAN Stitching via Cloud Backbone 3.3. Traffic Steering Challenges in Multi-Segment SD-WAN Traffic engineering mechanisms, such as SR Policy [RFC9256] and RSVP-TE [RFC3209], can be used to steer traffic through specific network nodes. However, in typical SD-WAN deployments, CPE-to-CPE traffic is carried as best-effort over the public Internet or other shared transport. Forwarding in the underlay is destination- based, and the on-premises CPEs cannot directly control the specific path that packets take. This limits the ability to enforce precise traffic engineering (TE) to reach destination CPEs. This lack of predictable routing makes traffic steering between branch offices highly challenging. Unlike private MPLS networks or provider-controlled backbones, SD-WAN cannot inherently dictate the intermediate paths for branch-to- branch traffic. As a result, policies intended to optimize performance, enforce security, or ensure compliance can be difficult to implement. Dunbar, et al. Expires Dec 24, 2027 [Page 9] Internet-Draft Multi-segment SD-WAN To address this issue, this document describes a method where Cloud GWs explicitly interconnect SD-WAN segments, ensuring that branch-to-branch traffic is steered through the Cloud Backbone rather than taking unpredictable Internet routes. This approach provides greater control over traffic flows, improving reliability, security, and policy enforcement. Note: The mechanism described in this document does not alter the forwarding behavior of the underlay network. Traffic from the source CPE to the ingress Cloud GW and from the egress Cloud GW to the destination CPE continues to follow normal underlay forwarding. Since these are typically short hops, the more useful traffic engineering (TE) occurs across the longer-range Cloud Backbone. In this model, the overlay steering defined here enables predictable selection of ingress and egress Cloud GWs, while TE within the backbone is offloaded to the Cloud Backbone provider. 4. Data Plane encoding for SD-WAN Transit To enable Cloud GWs to distinguish between packets requiring decryption for internal cloud services and transit packets that should be forwarded to destination CPEs, proper packet marking is essential. Many encapsulation methods, such as VLAN Tags [IEEE802.1Q], IP-in-IP [RFC2003], GRE [RFC2784], etc., can be used to steer traffic from a CPE to its nearest (or chosen) Cloud GW. However, GENEVE encapsulation [RFC8926] offers significant advantages, including flexible option Sub- TLVs that can signal routing and policy preferences, such as Permitted Regions, Excluded Regions, preferred egress Cloud GWs, and other service specific requirements. In addition, GENEVE Encapsulation [RFC8926] is widely supported by major Cloud Service Providers, which allows Cloud GWs to efficiently steer IPsec-encrypted packets between CPEs via Cloud Backbone without decryption, reducing processing overhead and improving performance while maintaining end-to- end encryption. Dunbar, et al. Expires Dec 24, 2027 [Page 10] Internet-Draft Multi-segment SD-WAN 4.1. Multi-Segment SD-WAN Option Class The Multi-Segment SD-WAN Option uses the GENEVE tunnel option format defined in Section 3.5 of [RFC8926]. This document uses the GENEVE Option Class value 0x0163, which has been assigned by IANA to identify the Multi-Segment SD-WAN Option. This enables Cloud GWs to interpret and process SD-WAN transit packets efficiently without decrypting the encapsulated SD-WAN traffic. Figure 3 reproduces the GENEVE option header for context and illustrates how the Multi- Segment SD-WAN Sub-TLVs are carried in the option data. The GENEVE option header format and the semantics of its fields are unchanged from [RFC8926]. The Sub-TLVs MAY appear in any order, and other optional Sub-TLVs defined for this option MAY also be included. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Multi-seg-SD-WAN Option Class |C| Type |R|R|R| Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Optional SD-WAN Tunnel Endpoint Sub-TLV ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Optional SD-WAN Tunnel Originator Sub-TLV ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Optional Egress GW Sub-TLV ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // // // Optional Type Length Value objects (variable) // // // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3 Multi Segment SD-WAN Option Class - Multi-seg-SD-WAN Option Class: value 0x0163 (assigned by IANA). - C-bit: MUST be set to ensure that a receiving node drops the packet if it does not recognize the option, as per [RFC8926]. Dunbar, et al. Expires Dec 24, 2027 [Page 11] Internet-Draft Multi-segment SD-WAN - Type (7 bits): Specifies the multi-segment SD-WAN forwarding model. Values are assigned from the "Multi- Segment SD-WAN GENEVE Option Types" registry defined in Section 12: Type = 1: Single-hop transit SD-WAN Type = 2: Multi-Hop transit SD-WAN with an explicitly specified egress Cloud GW (via Egress GW Sub-TLV). Type = 3: Multi-hop transit SD-WAN without an explicitly specified egress Cloud GW. - R-bits (3 bits): Reserved for future use. These bits MUST be set to zero on transmission and ignored on receipt. - Length (5 bits): Indicates the length of the option data, containing the Sub-TLVs defined below, in 4-byte units, excluding the 4-byte option header [RFC8926]. A value of zero indicates that no Sub-TLVs are present. If the total encoded length of the included Sub-TLVs is not a multiple of 4 octets, zero-valued padding octets MUST be appended at the end of the option data to achieve 4-octet alignment. The padding is not part of any Sub-TLV. The option data is limited to 124 octets. Therefore, the total length of all included Sub-TLVs and any padding MUST NOT exceed 124 octets. An encapsulator MUST NOT include a Sub-TLV that would cause this limit to be exceeded. As specified in [RFC8926], the GENEVE Protocol Type is the EtherType of the encapsulated PDU and MUST be set to 0x0800 for IPv4 or 0x86DD for IPv6. For ESP, the IPv4 Protocol or IPv6 Next Header field is set to 50. 4.2. SD-WAN Tunnel Endpoint Sub-TLV The SD-WAN Endpoint sub-TLV indicates the destination CPE, which is the endpoint of the IPsec Tunnel between branch CPEs. This Sub-TLV is used by the Cloud Backbone to determine the optimal egress Cloud GW for forwarding the encrypted traffic. The SD-WAN Endpoint Sub-TLV is OPTIONAL. When it is absent, the destination CPE is determined from the destination address of the encapsulated IP packet. Dunbar, et al. Expires Dec 24, 2027 [Page 12] Internet-Draft Multi-segment SD-WAN For example, in an SD-WAN deployment where CPE1 establishes an IPsec SA with CPE2 (as shown in Figure 1), this Sub-TLV within the GENEVE header contains CPE2's IP address, ensuring that encrypted traffic is correctly routed to the terminating CPE of the IPsec tunnel while enabling the Cloud Backbone to steer the packet to the most suitable egress Cloud GW. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |SD-WAN Endpoint| length | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SD-WAN Endpoint Addr Family | Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable) + ~ ~ | SD-WAN Endpoint Address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4 SD-WAN Endpoint Sub-TLV - SD-WAN Endpoint (8 bits): Identifies the SD-WAN Tunnel Endpoint Sub-TLV with a Type value of 1. - Length (8 bits): Specifies the total length of the value field in octets. - Reserved (16 bits): Reserved for future use. These bits MUST be set to zero on transmission and ignored on receipt. - SD-WAN Endpoint Addr Family (16 bits): Identifies the address family of the SD-WAN Endpoint Address. Values are taken from the IANA "Address Family Numbers" registry [IANA-AF]. A value of 1 indicates an IPv4 address and a value of 2 indicates an IPv6 address. Other Address Family values are not supported by this specification. - SD-WAN Endpoint Address: Contains the destination endpoint address. The field is 4 octets for Address Family 1 (IPv4) and 16 octets for Address Family 2 (IPv6). If the Address Family is unsupported or the Sub- TLV Length is inconsistent with the address length indicated by the Address Family, the packet MUST be discarded as malformed. Dunbar, et al. Expires Dec 24, 2027 [Page 13] Internet-Draft Multi-segment SD-WAN 4.3. SD-WAN Tunnel Originator Sub-TLV The SD-WAN Tunnel Originator Sub-TLV is an optional Sub-TLV within the multi-seg-SD-WAN Option Class to indicate the originating CPE of the IPsec Tunnel. For example, in an SD-WAN deployment where CPE1 establishes an IPsec SA with CPE2 (as shown in Figure 1), this Sub-TLV within the GENEVE header carries CPE1's address, allowing transit nodes and Cloud GWs to recognize the source of the encrypted traffic. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |SDWAN Origin | Length | reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SD-WAN Org Addr Family | Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable) + ~ ~ |SD-WAN Tunnel Originator Addr +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5 SD-WAN Tunnel Originator Sub-TLV - SDWAN Origin (8 bits): Identifies the SDWAN Tunnel Originator Sub-TLV with a Type value of 2. - Length (8 bits): Specifies the total length of the value field in octets, excluding the first 4 bytes, which include the SD-WAN Origin (1 byte), Length (1 byte), and Reserved (2 bytes) fields. - Reserved (16 bits): Reserved for future. These bits MUST be set to zero on transmission and ignored on receipt. - SD-WAN Org Addr Family (16 bits): Identifies the address family of the originator. Values are taken from the IANA "Address Family Numbers" registry [IANA-AF]. A value of 1 indicates an IPv4 address and a value of 2 indicates an IPv6 address. Other Address Family values are not supported by this specification. - SD-WAN Tunnel Originator Addr: Contains the originator address. The field is 4 octets for Address Family 1 (IPv4) and 16 octets for Address Family 2 (IPv6). If the Address Family is unsupported or the Sub-TLV Length is Dunbar, et al. Expires Dec 24, 2027 [Page 14] Internet-Draft Multi-segment SD-WAN inconsistent with the address length indicated by the Address Family, the packet MUST be discarded as malformed. This Sub-TLV allows Cloud GWs and transit nodes to identify the packet's source, allowing them to apply source-specific policies for forwarding. Such policies may include traffic engineering rules specific to the originating CPE, security enforcement tailored to the source, or path selection constraints based on the origin. Any such policy, however, MUST be applied only after the ingress Cloud GW authenticates the sender and validates the Originator value against authorized policy state. A false, unauthorized, or unverifiable Originator value MUST NOT be used to obtain preferred treatment and such packets MUST be discarded. 4.4. Egress GW Sub-TLV In a multi-segment SD-WAN deployment over the Cloud Backbone, the originating CPE can use the Egress GW Sub-TLV to explicitly specify the egress Cloud GW responsible for forwarding traffic to the destination CPE. This ensures predictable routing behavior and enables policy-driven packet delivery across the Cloud Backbone. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |SDWAN EgressGW | length | reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Egress GW Addr Family | Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable) + ~ ~ | Egress GW Address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 6 SD-WAN Egress GW Sub-TLV - SDWAN EgressGW (8 bits): Identifies Egress GW Sub-TLV with a Type value of 3. - Length (8 bits): Specifies the total length of the value field in octets, excluding the first 4 bytes, which include the SD-WAN EgressGW Sub-TLV Type (1 byte), Length (1 byte), and Reserved (2 bytes) fields. - Reserved (16 bits): Reserved for future. MUST set to 0. MUST be ignored by recipients. Dunbar, et al. Expires Dec 24, 2027 [Page 15] Internet-Draft Multi-segment SD-WAN - Egress GW Addr Family: Identifies the address family of the Egress GW. A value of 1 indicates an IPv4 address and a value of 2 indicates an IPv6 address. - Egress GW Address: Contains the address of the egress Cloud GW. The field is 4 octets for Address Family 1 (IPv4) and 16 octets for Address Family 2 (IPv6). If the Address Family is unsupported or the Sub-TLV Length is inconsistent with the address length indicated by the Address Family, the packet MUST be discarded as malformed. The Egress GW Sub-TLV allows the originating CPE to specify the Egress Cloud GW responsible for forwarding traffic to the destination CPE. This Egress GW address can be either preconfigured or dynamically discovered through a control plane protocol exchange with the destination CPE. By explicitly defining the egress GW, this Sub-TLV ensures predictable traffic steering, reducing reliance on destination-based routing and optimizing packet delivery across the Cloud Backbone. The details of the control plane protocol used for GW discovery are beyond the scope of this document. 4.5. Permitted Regions Sub-TLV Some enterprises may require that traffic across the Cloud Backbone is strictly confined to a specific set of regions. This Sub-TLV allows the ingress SD-WAN CPE to express such restrictions as part of the encapsulation metadata. Traffic MUST be discarded if the Ingress Gateway or Egress Gateway belongs to a region not listed in this Sub-TLV. For intermediate transit nodes or regions, enforcement is based on the Cloud Backbone provider's knowledge of its topology and region boundaries and the policy agreed with the enterprise. This restriction is commonly used to enforce regulatory, security, or latency-based geographic constraints, where data must remain confined to specified regions. Format of the Permitted Regions Sub-TLV: Dunbar, et al. Expires Dec 24, 2027 [Page 16] Internet-Draft Multi-segment SD-WAN 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |PermittedReg | Length | Reserved (16 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Region Len | UTF-8 encoding of Region Name or ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Region Len | UTF-8 encoding of Region Name or ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7 Permitted Regions Sub-TLV - PermittedReg (8 bits): Identifies the Permitted Regions Sub-TLV with a Type value of 4. - Length (8 bits): Total length of the Value field (everything after the Type and Length fields), in octets. - Reserved (16 bits): Reserved for future use. MUST be set to zero and ignored on receipt. - Region Len (8 bits per region entry): Length of the UTF- 8 encoding of the Region Name or identifier, in octets. - UTF-8 encoding of the Region Name (e.g., "us-west", "eu- central"). Region Names are compared case-insensitively. Multiple regions MAY be present, each starting with its own Region Len field. Processing notes: - The receiving Cloud GW MUST apply the region checks specified above. Enforcement of restrictions on internal Cloud Backbone transit regions depends on the Cloud Backbone provider's knowledge of its topology and region boundaries and is outside the scope of this document. - If any component of the path falls outside the listed regions, the packet MUST be discarded. - Region Names and their interpretation are based on prior agreement between the enterprise and the Cloud Backbone provider and may use commonly recognized or provider- specific naming conventions. Region Names are compared case-insensitively Dunbar, et al. Expires Dec 24, 2027 [Page 17] Internet-Draft Multi-segment SD-WAN Note: It is beyond the scope of this document to specify how the Cloud Backbone enforces this restriction. Mechanisms for identifying region boundaries, enforcing region-based constraints, and generating alerts or alarm notifications when traffic violates region restrictions are subject to implementation decisions and based on prior agreement between the Cloud Backbone provider and the enterprise. 4.6. Excluded Regions Sub-TLV Excluded Regions Sub-TLV is an optional field used to specify a list of Cloud Availability Regions or Zones that must be avoided when forwarding packets across the Cloud Backbone. This can be used for: - Regulatory compliance, ensuring traffic does not traverse non-compliant regions. - Risk mitigation, preventing traffic from passing through regions with known security, performance, or geopolitical concerns. Multiple region entries MAY be specified in a single Sub-TLV. Each region is identified by a variable length UTF-8 encoded name or numeric ID, preceded by a length field. This Sub-TLV expresses explicit exclusions and supports both soft and hard enforcement. 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |ExcludedRegion | length |E| Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Region Len | UTF-8 encoding of Region Name or ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Region Len | UTF-8 encoding of Region Name or ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8 Excluded Regions Sub-TLV - ExcludedRegion (8 bits): identifies the Excluded Regions Sub-TLV with a Type value of 5. Dunbar, et al. Expires Dec 24, 2027 [Page 18] Internet-Draft Multi-segment SD-WAN - Length (8 bits): Total length of the Value field in octets (everything after the first 2 bytes). - E-bit (1 bit) - Exclusion severity indicator: o 0: Soft exclusion - The listed region SHOULD be avoided when feasible. o 1: Hard exclusion - The listed region MUST NOT be traversed. If no path satisfying the exclusion is available, the packet MUST be discarded, and an alert or alarm MAY be generated. - Reserved (15 bits): MUST be set to zero on transmission and ignored on receipt. - Region Len (8 bits per region entry): Length of the UTF- 8 encoding of the Region Name or identifier, in octets. - UTF-8 encoding of the Region Name (e.g., "us-west", "eu- central"). Region Names, based on prior agreement between the enterprise and the Cloud Backbone provider, are compared case-insensitively. Multiple region entries MAY be listed, each beginning with a Region Len byte. Processing Notes: The E-bit determines how strictly the exclusions are enforced. A value of 1 (hard exclusion) requires that the listed region MUST NOT be traversed. If no compliant path is available, the packet MUST be discarded and an alert or alarm MAY be generated. A value of 0 (soft exclusion) indicates that the Cloud Backbone SHOULD avoid the listed region when feasible. If avoidance is not feasible, the packet MAY traverse the listed region and no alert or alarm is required. The meaning and granularity of region identifiers MUST be agreed upon between the enterprise and the Cloud Backbone provider. It is beyond the scope of this document to define how enforcement or alerting is implemented. These are subject to operator policies and implementation specifics. 4.7. HMAC Authentication Value Sub-TLV To support integrity protection and authentication of the Multi-Segment SD-WAN GENEVE metadata, this document defines Dunbar, et al. Expires Dec 24, 2027 [Page 19] Internet-Draft Multi-segment SD-WAN the HMAC (Hash-based Message Authentication Code) Authentication Value (HMAC-Auth-Val) Sub-TLV. The Sub-TLV carries the HMAC authentication value computed over the GENEVE header, including the Multi-Segment SD-WAN Option Class, allowing a receiving Cloud GW to verify that the steering metadata has not been modified in transit. The selection of the HMAC algorithm, authentication value length, key establishment, and verification procedures are described in Section 9.2. The HMAC Authentication Value Sub-TLV has the following format: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | HMAC-Auth-Val | length | reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ ~ | HMAC Authentication Value | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9 Multi Segment SD-WAN HMAC Sub-TLV - HMAC-Auth-Val (8 bits): HMAC Authentication Value Sub- TLV Type = 6 (Assigned by this document). - Length (8 bits): Total length of the value field, which is the length of the HMAC Authentication Value in bytes plus 2 reserved bytes. - Reserved (16 bits): Reserved for future use. These bits MUST be set to zero on transmission and ignored on receipt. - HMAC Authentication Value (variable): Contains the HMAC computed over the GENEVE header, including the Multi- Segment SD-WAN Option Class and all included Sub-TLVs, excluding the HMAC-Auth-Val Sub-TLV itself (i.e., its Type, Length, Reserved, and Authentication Value fields). The HMAC algorithm, keying method, and authentication value length are specified in Section 9.2. Dunbar, et al. Expires Dec 24, 2027 [Page 20] Internet-Draft Multi-segment SD-WAN 5. Packet Header Processing The procedures described in this section apply only to packets that carry the SD-WAN Option Class in the GENEVE header. Packets without this option are processed using default forwarding behavior. As illustrated in Figure 1, when a Cloud GW receives a GENEVE-encapsulated packet (i.e., Dst Port = 6081 (GENEVE); MultiSeg-SDWAN Option Class; encapsulated IPv4 Protocol or IPv6 Next Header = 50 (ESP)), it processes the packet as follows: Processing at the Ingress Cloud GW: - Authenticate the packet or its originating CPE, and validate the GENEVE metadata, using the authentication mechanism configured for the deployment. - Process the Multi-Segment SD-WAN Option Type: o For Type = 1, the Cloud GW performs the applicable policy, security, and analytics processing and forwards the packet toward the destination CPE. An Egress GW Sub-TLV MUST NOT be present. o For Type = 2, an Egress GW Sub-TLV MUST be present. The Cloud Backbone uses the Sub-TLV to identify the egress Cloud GW. o For Type = 3, an Egress GW Sub-TLV MUST NOT be present. The Cloud Backbone determines the egress Cloud GW based on the destination CPE address. o A packet containing a Type value inconsistent with the presence or absence of the Egress GW Sub-TLV MUST be discarded as malformed. - For Type = 1, the Cloud GW forwards the packet toward the destination CPE after performing the applicable policy, security, and analytics processing. For Type = 2 or Type = 3, update the outer IP header for forwarding across the Cloud Backbone by setting the source address to the ingress Cloud GW and the destination address to the egress Cloud GW identified by the Egress GW Sub-TLV (Type = 2) or selected by the Cloud Backbone (Type = 3). The ingress Cloud GW MUST Dunbar, et al. Expires Dec 24, 2027 [Page 21] Internet-Draft Multi-segment SD-WAN update the IPv4 header checksum, when applicable, and recompute the UDP checksum as required by [RFC8926]. - Forward the packet to the egress Cloud GW. To prevent unauthorized access, Cloud GW MUST drop any packets containing unrecognized source addresses or invalid values in the GENEVE Sub-TLVs, ensuring that only registered entities can utilize Cloud services. Processing at the Egress Cloud GW: - Decapsulate the GENEVE header to extract the IPsec- encrypted payload. - Validate that the SD-WAN Tunnel Endpoint Sub-TLV corresponds to a registered destination CPE. - Ensure the source Cloud GW is an authorized forwarding node to prevent unauthorized traffic injection. - Forward the IPsec-encrypted payload to the destination CPE, preserving the end-to-end encryption. - Drop any packet that lacks a valid destination CPE or originates from an untrusted source. By enforcing these processing steps at both the ingress and egress Cloud GWs, the system ensures secure, efficient, and policy-compliant forwarding of SD-WAN traffic across the Cloud Backbone. 6. Error Handling To ensure secure and efficient traffic forwarding through the Cloud Backbone, Cloud GW MUST enforce the following error handling measures: - Drop packets with unregistered or invalid source/destination addresses to prevent unauthorized access. - Validate the SD-WAN Endpoint Sub-TLV, when present, and drop packets if the destination CPE is unauthorized, unreachable, or mismatched. Dunbar, et al. Expires Dec 24, 2027 [Page 22] Internet-Draft Multi-segment SD-WAN - Discard packets with incorrect GENEVE headers, invalid Sub-TLV formats, or authentication failures. The receiver MUST verify that each Sub-TLV fits within the remaining option data and that the total encoded length of all Sub-TLVs, together with any trailing padding, does not exceed the length of the enclosing Multi- Segment SD-WAN Option. A packet that fails either check MUST be discarded as malformed. An unrecognized Sub-TLV Type MUST be skipped using its Length field, and processing MUST continue with the next Sub-TLV. - Reject misrouted packets if the Cloud Backbone cannot determine an optimal egress Cloud GW or if the specified egress GW is unreachable. - Enforce rate limits on excessive traffic from a single source to prevent congestion and abuse. - Verify compliance with region restrictions specified by the Permitted Regions and Excluded Regions Sub-TLVs. Service subscription validation, rate limiting, and other provider-specific admission and traffic-management policies are deployment-specific and are outside the scope of this document. By implementing these error handling mechanisms, Cloud GWs ensure network stability, security, and efficient resource utilization while preventing misconfigurations and forwarding errors. 7. Control Plane considerations 7.1. Control Plane for CPEs The control plane enables SD-WAN CPEs to discover their network attributes, establish connectivity, and exchange routing information. In an SD-WAN deployment, on-premises CPEs and virtual CPEs (vCPEs) in Cloud DCs may be managed under a common iBGP administrative domain, facilitating route propagation and policy enforcement. Mechanisms such as BGP-based SD-WAN Edge Discovery [SD-WAN-Edge-Discovery] allow CPEs to dynamically discover each other's properties, improving automation and reducing manual configurations. Additionally, IPsec SA parameters Dunbar, et al. Expires Dec 24, 2027 [Page 23] Internet-Draft Multi-segment SD-WAN between CPEs can be exchanged through the iBGP control plane using a RR to simplify security policy management. The IPsec SAs themselves are established separately, for example via IKEv2 or out-of-band management systems. The iBGP sessions among CPEs described in this section are a logical control plane relationship among enterprise managed nodes, using the enterprise's own management/control connectivity. They are independent of the eBGP sessions between a CPE and its attached Cloud GW described in Section 7.2. 7.2. Control Plane between CPEs and Cloud GWs This section describes the control plane relationship between a CPE and its attached Cloud GW, which is distinct from the iBGP based control plane among CPEs described in Section 7.1. There are typically eBGP sessions between a CPE and a Cloud GW for exchanging routing information related to services that terminate within the cloud. This allows the CPE to learn routes to cloud-hosted resources and enables the Cloud GW to learn routes to the CPE's on-premises networks. This control- plane relationship is separate from the CPE-to-CPE encrypted traffic that transits the Cloud Backbone, which remains end- to-end encrypted and is not decrypted at the Cloud GWs. When the connection between a CPE and a Cloud GW traverses a public or otherwise untrusted network, an IPsec tunnel may also be established to secure that traffic. In such cases, the IPsec Security Association (SA) parameters between the CPE and its corresponding Cloud GW are established out-of- band (e.g., via management or automation systems) or negotiated dynamically using IKEv2. Control plane mechanisms MUST ensure that Cloud GWs can identify and authenticate SD-WAN CPEs, validate SD-WAN metadata, and apply appropriate routing policies based on dynamic network conditions. This ensures that route exchanges are trustworthy, policy-compliant, and adaptive to changing operational requirements. 8. Observability Consideration Observability considerations encompass monitoring, analysis, and reporting mechanisms to gain insights into the behavior Dunbar, et al. Expires Dec 24, 2027 [Page 24] Internet-Draft Multi-segment SD-WAN and performance of the multi-segment SD-WAN infrastructure. Key observability aspects include: - Performance Metrics: Monitor and collect performance metrics related to link utilization, latency, and packet loss across the SD-WAN segments and Cloud DC backbone. This data provides insights into the overall health and efficiency of the network. IP Flow Information Export (IPFIX) [RFC7011] is one of the standardized methods to expose traffic flow over the network. - Global Network Topology Visualization: Utilize visualization tools to depict the global network topology, showcasing the interconnections and traffic flows between different SD-WAN segments and Cloud DCs. - Control Plane Monitoring: Monitor the control plane for both CPEs and the communication between CPEs and Cloud GWs. This includes tracking route discovery, path selection, and any changes in network state to ensure proper functioning of the SD-WAN control plane. - Security Event Logging: The security event logging is to capture and analyze security-related events, including threat detection, authentication failures, and any unauthorized access attempts. Syslog [RFC5424] is a valuable tool for security monitoring and auditing. These considerations contribute to the overall success of the multi-segment SD-WAN deployment connecting edge devices via a Cloud DC backbone. 9. Security Considerations 9.1. Threat Analysis The GENEVE header used for steering is not encrypted, making it susceptible to on-path attacks between CPEs and Cloud GWs. Key risks include: a) Eavesdropping: Attackers can learn branch and Cloud GW locations, though payload remains protected by IPsec. Dunbar, et al. Expires Dec 24, 2027 [Page 25] Internet-Draft Multi-segment SD-WAN b) Header Manipulation: Altered Sub-TLVs may cause misrouting or packet drops. c) Bandwidth Theft: A malicious or misconfigured CPE could spoof SD-WAN metadata to use Cloud Backbone resources without authorization. Mitigation the above risks requires authenticating and validating SD-WAN metadata to ensure it originates from authorized CPEs. 9.2. HMAC-based Integrity and Authentication The HMAC-Auth-Val Sub-TLV defined in Section 4.7 provides integrity protection and source authentication for the Multi- Segment SD-WAN steering metadata carried in the GENEVE header. It allows Cloud GW to verify that the steering metadata originated from an authorized SD-WAN CPE and has not been modified while in transit. HMAC is a widely deployed cryptographic mechanism that provides message integrity and data origin authentication by combining a shared secret key with a cryptographic hash function. Upon receipt of a packet, the receiving Cloud GW computes the HMAC over the received GENEVE header using the shared key and compares the result with the authentication value carried in the HMAC-Auth-Val Sub-TLV. A successful comparison confirms that the protected metadata has not been altered and was generated by an entity possessing the shared key. This mechanism authenticates the GENEVE metadata exchanged between SD-WAN CPEs and the Cloud GWs. The HMAC key is a dedicated symmetric key shared between the CPE and its attached Cloud GW. The key MAY be provisioned by the SD-WAN controller or another secure management mechanism. When the key is distributed over an existing IPsec-protected CPE-to-GW channel, no additional IKEv2 exchange is required solely for protecting the GENEVE metadata. The HMAC authentication mechanism defined in this document provides integrity and source authentication of the GENEVE steering metadata but does not by itself provide replay protection. Replay protection, when required by a deployment, is provided by mechanisms outside the scope of this document. Dunbar, et al. Expires Dec 24, 2027 [Page 26] Internet-Draft Multi-segment SD-WAN The anti-replay mechanism of the encapsulated IPsec ESP SA protects the ESP traffic according to the applicable IPsec processing but does not constitute replay protection for the GENEVE metadata defined in this document. Unless otherwise configured, HMAC-SHA-256 [RFC4868] is the default algorithm for computing the authentication value. Other HMAC algorithms MAY be used when explicitly configured by the SD-WAN controller for both the CPE and its attached Cloud GW, together with the corresponding keying parameters. For HMAC-SHA-256, a 16-octet (128-bit) truncated authentication value, consistent with [RFC4868], is RECOMMENDED. The Multi-Segment SD-WAN GENEVE Option has a maximum option-data length of 124 octets, which MUST accommodate all included Sub-TLVs. The HMAC protects the GENEVE steering metadata exchanged between a CPE and its attached Cloud GW, while the encapsulated payload is independently protected by IPsec. Deployments in which the CPE-to-Cloud-GW path is trusted or otherwise protected MAY configure a shorter authentication value, or MAY omit the HMAC Authentication Value Sub-TLV. Deployments using shorter authentication values MUST consider the resulting reduction in forgery resistance. The HMAC defined in this section provides the following security properties: - Data Integrity: Detects unauthorized modification of the Multi-Segment SD-WAN GENEVE metadata. - Source Authentication: Allows Cloud GWs to verify that the steering metadata originated from an authorized SD-WAN CPE possessing the shared authentication key. - Low Overhead: Supports truncated authentication values to minimize packet overhead while providing adequate protection for steering metadata. - Algorithm Agility: Supports different HMAC algorithms through controller configuration. - Operational Simplicity: Reuses existing IPsec trust relationships and key management, avoiding additional Security Associations between Cloud GWs and CPEs. Dunbar, et al. Expires Dec 24, 2027 [Page 27] Internet-Draft Multi-segment SD-WAN Although truncating the HMAC authentication value reduces the security margin against forgery compared to the full HMAC output, the HMAC protects only the GENEVE steering metadata, while the payload remains independently protected by IPsec. An attacker without the shared secret key may attempt to forge an authentication value by guessing; the probability of successful forgery increases as the authentication value is shortened. Deployments using truncated authentication values MUST therefore select a length appropriate to their threat model and operational environment. 9.3. Additional Integrity and Authentication Some deployments may require stronger or more comprehensive integrity protection than a truncated HMAC, such as when mandated by security policy, regulatory compliance, or risk management practices. In these cases, an additional integrity layer can be applied using ESP with NULL encryption (ESP- NULL) [RFC2410] [RFC6071] or Authentication Header (AH) [RFC4302] on top of the existing IPsec encryption between CPEs. Consistent with [RFC8221], ESP-NULL is preferred over AH for authentication-only protection because of its compatibility with NAT traversal. AH and ESP-NULL provide cryptographic integrity for the entire IP packet, not just the GENEVE metadata. All approaches (including the HMAC) require cryptographic keys. The operational difference is that AH/ESP-NULL require dedicated IPsec SAs and IKE state between each Cloud GW and CPE, increasing per-peer state and processing. By contrast, the HMAC Sub-TLV (Type = 6, defined in this document) can use controller-distributed symmetric keys (e.g., per-tenant or per-CPE) without establishing additional IPsec SAs between Cloud GWs and CPEs. NAT Considerations: AH is not compatible with NAT traversal because it authenticates the outer IP header, and any address change will cause verification to fail. ESP-NULL avoids this issue but still incurs additional per-packet processing. When AH or ESP-NULL is used, the required IPsec SAs and associated keying material SHOULD be established using IKEv2 [RFC7296]. Deployments MAY use manual or controller based key provisioning when supported by the operational environment, but such provisioning MUST provide equivalent authentication, Dunbar, et al. Expires Dec 24, 2027 [Page 28] Internet-Draft Multi-segment SD-WAN authorization, replay protection, key freshness, and key rollover properties. 10. Manageability Considerations In multi-segment SD-WAN deployments where the Cloud GW and CPEs belong to different administrative domains, manageability must address the challenges of secure, interoperable, and policy-compliant operation across organizational boundaries, consistent with the service framework defined in MEF 70.1 [MEF-70.1]. Key considerations include: - Cross-Domain Authentication and Authorization: Ensure that CPEs connecting to the Cloud GW are authenticated using mutually agreed methods, and that authorization policies are enforced to prevent unauthorized use of Cloud Backbone resources. - Metadata Validation and Policy Enforcement: Cloud GWs MUST validate SD-WAN metadata (e.g., GENEVE Sub-TLVs) against the registered information for each CPE. This prevents spoofing, misrouting, and cross- tenant traffic leakage. - Operational Coordination and Fault Handling: Define inter-organization procedures for troubleshooting and incident response. This should include point-of- contact directories, escalation processes, and shared logging formats for event correlation. - Coordination of Configuration Changes: Coordinate configuration changes-such as policy updates, region restrictions, or authentication parameters-so that both the Cloud GW and CPEs apply them consistently, avoiding mismatches that disrupt traffic. - Policy Automation Using I2NSF Principles ([RFC8192]): Where feasible, leverage I2NSF concepts to automate policy configuration, exchange, and enforcement between domains, reducing manual coordination and improving operational consistency. Dunbar, et al. Expires Dec 24, 2027 [Page 29] Internet-Draft Multi-segment SD-WAN 11. Applicability Considerations The GENEVE encapsulation defined in this document is used to carry Multi-Segment SD-WAN steering metadata between a CPE and its attached Cloud GW and across the Cloud Backbone. The CPE-to-Cloud-GW path may traverse the public Internet or other shared underlay networks. In such cases, the public Internet serves only as the underlay for reaching the Cloud GW; Internet transit nodes do not interpret or process the Multi-Segment SD-WAN GENEVE metadata. Traffic steering based on this metadata is performed by the Cloud GWs across the controlled Cloud Backbone. When GENEVE is carried over the general Internet, implementations MUST follow the UDP usage and congestion considerations specified in Section 4.1 of [RFC8926]. GENEVE traffic sent over the general Internet MUST either carry congestion-controlled traffic or use an appropriate tunnel- level congestion-control mechanism. Within a controlled Cloud Backbone, traffic engineering, capacity management, and congestion handling are provided by the Cloud Backbone operator. The encapsulator MUST account for the maximum encapsulation overhead that it can add, including the outer IP header, UDP header, GENEVE header, and Multi-Segment SD-WAN option data. As specified in Section 4.4.1 of [RFC8926], the maximum possible GENEVE option length MUST be included when calculating the tunnel MTU. As specified in Section 4.1 of this document, the Multi-Segment SD-WAN Option data, including all Sub-TLVs and any trailing padding, is limited to 124 octets. An encapsulator MUST NOT include a set of Sub- TLVs whose total encoded length, including any trailing padding, exceeds this limit. An encapsulator MUST NOT emit an encapsulated packet that exceeds the effective tunnel MTU. If an inner packet is too large to be encapsulated without exceeding the effective tunnel MTU, the encapsulator MUST drop the packet unless IPv4 fragmentation is permitted and performed. For IPv4, if fragmentation is not permitted, the encapsulator SHOULD return an ICMP Destination Unreachable, Fragmentation Needed message toward the source when possible. For IPv6, routers do not fragment packets; therefore, the encapsulator SHOULD return an ICMPv6 Packet Too Big message toward the source when possible. Dunbar, et al. Expires Dec 24, 2027 [Page 30] Internet-Draft Multi-segment SD-WAN Implementations SHOULD use Path MTU Discovery (PMTUD) and SHOULD avoid fragmentation where possible. Upon receiving a valid ICMP Destination Unreachable, Fragmentation Needed message for IPv4, or an ICMPv6 Packet Too Big message for IPv6, the encapsulator SHOULD reduce the effective tunnel MTU as appropriate. MTU-related ICMP or ICMPv6 messages generated on the Cloud- GW-to-Cloud-GW path are delivered to the ingress Cloud GW rather than directly to the originating CPE. Deployments SHOULD therefore configure the effective tunnel MTU to account for the complete encapsulation overhead and the MTU of the Cloud Backbone path. 12. IANA Considerations IANA has assigned the following GENEVE Option Class from the First Come First Served range of the "Geneve Option Class" registry: Option Class Description Assignee/Contact Reference ------ ------------------- ------------- ----------- 0x0163 Multi Segment SD-WAN IETF [this document] IANA is requested to create a new registry in the "Network Virtualization Overlays (NVO3)" registry group for Sub-TLV types used specifically by the Multi-Segment SD-WAN GENEVE Option Class (0x0163): Registry Name: Multi Segment SD-WAN Sub-TLVs Applicable Option Class: 0x0163 (Multi-Segment SD-WAN) Registration Procedure: IETF Review Change Controller: IETF Reference: [this document] Sub-TLV Type Description Reference ------------ ---------------------- --------------- 0 Reserved 1 SD-WAN Endpoint [Section 4.2] 2 SD-WAN Originator [Section 4.3] 3 SD-WAN Egress GW [Section 4.4] 4 Permitted Regions [Section 4.5] 5 Excluded Regions [Section 4.6] 6 Multi SD-WAN-HMAC [Section 4.7] Dunbar, et al. Expires Dec 24, 2027 [Page 31] Internet-Draft Multi-segment SD-WAN 7-254 Unassigned 255 Reserved IANA is requested to create a new registry in the "Network Virtualization Overlays (NVO3)" registry group for the 7-bit Type field of the Multi-Segment SD-WAN GENEVE Option Class (0x0163). Registry Name: Multi-Segment SD-WAN GENEVE Option Types Applicable Option Class: 0x0163 (Multi-Segment SD-WAN) Registration Procedure: IETF Review Change Controller: IETF Reference: [this document] Type Description Reference ---- ------------ -------------- 0 Reserved [Section 4.1] 1 Single-Hop Transit SD-WAN [Section 4.1] 2 Multi-Hop Transit SD-WAN with Explicit Egress Cloud GW [Section 4.1] 3 Multi-Hop Transit SD-WAN without Explicit Egress Cloud GW [Section 4.1] 4-127 Unassigned The values in this registry are applicable only to the Multi-Segment SD-WAN GENEVE Option Class (0x0163). 13. References 13.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC4302] Kent, S., "IP Authentication Header", RFC 4302, DOI 10.17487/RFC4302, December 2005. [RFC4868] Kelly, S. and S. Frankel, "Using HMAC-SHA-256, HMAC-SHA-384, and HMAC-SHA-512 with IPsec", RFC 4868, May 2007. Dunbar, et al. Expires Dec 24, 2027 [Page 32] Internet-Draft Multi-segment SD-WAN [RFC7296] Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., and T. Kivinen, "Internet Key Exchange Protocol Version 2 (IKEv2)", RFC 7296, October 2014. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8221] Migault, D. and T. Guggemos, "Cryptographic Algorithm Implementation Requirements and Usage Guidance for Encapsulating Security Payload (ESP) and Authentication Header (AH)", RFC 8221, DOI 10.17487/RFC8221, October 2017. [RFC8926] J. Gross, et al., "GENEVE: Generic Network Virtualization Encapsulation", RFC8926, Nov 2020. [IANA-AF] IANA, "Address Family Numbers", Internet Assigned Numbers Authority, https://www.iana.org/assignments/address-family- numbers/ 13.2. Informative References [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, October 1996. [RFC2410] R. Glenn and S. Kent, "The NULL encryption Algorithm and Its Use with IPsec", RFC2410, Nov. 1998. [RFC2784] D. Farinacci, T. Li, S. Hanks, D. Meyer, and P. Traina, "Generic Routing Encapsulation (GRE)", RFC2784, March 2000. [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001. Dunbar, et al. Expires Dec 24, 2027 [Page 33] Internet-Draft Multi-segment SD-WAN [RFC3849] S. Huston, M. Thomson, S. Kent, and J. Seo, "IPv6 Address Prefix Reserved for Documentation", RFC3849, July 2004. [RFC5424] R. Gerhards, "The Syslog Protocol", RFC5424, March 2009. [RFC5737] J. Arkko, M. Cotton, and L. Vegoda, "IPv4 Address Blocks Reserved for Documentation", RFC5737, January 2010. [RFC6071] S. Frankel and S. Krishnan, "IP Security (IPsec) and Internet Key Exchange (IKE) Document Roadmap", Feb. 2011. [RFC7011] B. Claise, B. Trammell, and P. Aitken, "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information", RFC7011, Sept 2013. [RFC8192] S. Hares, et al., "Interface to Network Security Functions (I2NSF) Problem Statement and Use Cases", July 2017 [RFC9256] Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and P. Mattes, "Segment Routing Policy Architecture", RFC 9256, DOI 10.17487/RFC9256, July 2022. [MEF-70.1] MEF 70.1 SD-WAN Service Attributes and Service Framework. Nov. 2021. [Net2Cloud] L. Dunbar and A. Malis, "Dynamic Networks to Hybrid Cloud DCs Problem Statement", draft-ietf- rtgwg-net2cloud-problem-statement-43, Jan, 2025. [SD-WAN-Edge-Discovery] L. Dunbar, et al., "BGP UPDATE for SD-WAN Edge Discovery", draft-ietf-idr-sdwan-edge- discovery-29, July. 2026. Dunbar, et al. Expires Dec 24, 2027 [Page 34] Internet-Draft Multi-segment SD-WAN [IEEE802.1Q] IEEE, "IEEE Standard for Local and Metropolitan Area Networks--Bridges and Bridged Networks", IEEE Std 802.1Q. 14. Acknowledgments The authors would like to thank Adrian Farrel, Joel Halpern, Donald Eastlake, Stephen Farrell, Ajeet Gill, Gunter Van De Velde, Mike Bishop, Roman Danyliw, for their extensive review and suggestions. This document was prepared using 2-Word-v2.0.template.dot. Appendix A: Illustration of Packets through Cloud GWs This appendix illustrates the encapsulation of traffic between SD-WAN CPEs as it traverses one or more Cloud GWs. The Cloud GWs act as intermediate forwarding points that stitch together the SD-WAN segments based on the Multi- Segment SD-WAN metadata; they do not terminate the enterprise traffic or the end-to-end IPsec protection between the communicating CPEs. A.1 Single Hop Cloud GW For the single-hop Cloud GW scenario shown in Figure 1, assume the following: - There is a bidirectional IPsec tunnel between CPE1 and Cloud GW; with IPsec SA1 for the traffic from the CPE1 to the Cloud-GW; and IPsec SA2 for the traffic from the Cloud-GW to the CPE1. This tunnel can carry CPE1 traffic to and from cloud services. The existing IPsec-protected channel can also be used to securely exchange configuration, control, and keying information, such as the HMAC key used to protect the GENEVE metadata. This tunnel is separate from the end- to-end CPE1-to-CPE2 IPsec tunnel described below. - There is a bidirectional IPsec tunnel between CPE2 and Cloud GW; with IPsec SA3 for the traffic from the CPE2 to the Cloud-GW; and IPsec SA4 for the traffic from the Cloud-GW to the CPE2. This tunnel can similarly Dunbar, et al. Expires Dec 24, 2027 [Page 35] Internet-Draft Multi-segment SD-WAN carry CPE2 traffic to and from cloud services and can also be used to securely exchange configuration, control, and keying information. It is separate from the end-to-end CPE1-to-CPE2 IPsec tunnel. - All the CPEs are under a common administrative control and participate in a common iBGP control plane, with routing information exchanged through a Route Reflector (RR). The CPEs can exchange information about their corresponding Cloud GW addresses through the control plane; the mechanism used to distribute this information is outside the scope of this document. When CPE1 (192.0.2.0/26) and CPE2 (192.0.2.64/26) need to communicate with each other, they establish a separate bidirectional end-to-end IPsec Tunnel, with SA5 for the traffic from CPE1 to CPE2 and SA6 for the traffic from CPE2 to CPE1. The Cloud GW forwards this IPsec-protected traffic without terminating or decrypting the CPE1-to-CPE2 IPsec tunnel. Assume the IPsec ESP Tunnel Mode is used. A packet from 192.0.2.1 to 192.0.2.65 has the following outer header: Dunbar, et al. Expires Dec 24, 2027 [Page 36] Internet-Draft Multi-segment SD-WAN +---------------------------------------------------+ | Outer IP Header | | Protocol = 17 (UDP) | | Src IP = CPE1 (underlay address) | | Dst IP = Cloud GW (underlay address) | +---------------------------------------------------+ | UDP Header | | Src Port = xxxx (ephemeral) | | Dst Port = 6081 (GENEVE) | +===================================================+ | GENEVE Header | | Protocol Type = 0x0800 (IPv4) or 0x86DD (IPv6) | | [Indicates the payload is an IP packet] | +---------------------------------------------------+ | MultiSeg-SDWAN Option Class | | SD-WAN EndPt Sub-TLV (CPE2 address) | | [Optional other SD-WAN Sub-TLVs] | | HMAC-Auth-Val Sub-TLV (GENEVE Hdr Authentication)| | (validated by GW) | +===================================================+ | ESP Outer IP Header (Tunnel Mode) | | Src IP = CPE1 (tunnel IP) | | Dst IP = CPE2 (tunnel IP) | | Protocol = 50 (ESP) | +---------------------------------------------------+ | ESP Header | | SPI (Security Parameters Index) | | Sequence Number | +---------------------------------------------------+ | Encrypted Payload | | Inner IP Header | | Src = 192.0.2.1 (host behind CPE1) | | Dst = 192.0.2.65 (host behind CPE2) | | Protocol = TCP | | TCP Header | | Application Payload | | Padding | | Pad Length | | Next Header | +---------------------------------------------------+ | Integrity Check Value (ICV) | | (Generated by CPE1, validated by CPE2) | +---------------------------------------------------+ Figure 10 Packet Encapsulation through a Single Cloud GW Dunbar, et al. Expires Dec 24, 2027 [Page 37] Internet-Draft Multi-segment SD-WAN A.2 Multi-hop Transit GWs Traffic between geographically separated CPEs can traverse multiple Cloud DCs over the Cloud backbone. Using Figure 2 as an example, assume the following: - There is a bidirectional IPsec tunnel between CPE1 and the Cloud GW1; with IPsec SA1 for the traffic from the CPE1 to the Cloud-GW1; and IPsec SA2 for the traffic from the Cloud-GW1 to the CPE1. This tunnel can carry CPE1 traffic to and from cloud services. The existing IPsec-protected channel can also be used to securely exchange configuration, control, and keying information, such as the HMAC key used to protect the GENEVE metadata. This tunnel is separate from the end- to-end CPE1-to-CPE2 IPsec tunnel described below. - There is a bidirectional IPsec tunnel between CPE10 and the Cloud GW2; with IPsec SA3 for the traffic from the CPE10 to the Cloud-GW2; and IPsec SA4 for the traffic from the Cloud-GW2 to the CPE10. These SAs can carry CPE10 traffic to and from cloud services and can also be used to securely exchange configuration, control, and keying information. It is separate from the end-to-end CPE1-to-CPE10 IPsec tunnel. - All the CPEs are under a common administrative control and exchange routing information through an iBGP Route Reflector (RR). The CPEs can exchange information about their corresponding Cloud GW addresses through the control plane; the mechanism used to distribute this information is outside the scope of this document. When CPE1(192.0.2.0/26) and CPE10(192.0.2.128/25) need to communicate with each other, CPE1 and CPE10 establish a bidirectional IPsec Tunnel, with SA5 for the traffic from CPE1 to CPE10 and SA6 for the traffic from CPE10 to CPE1. Assuming the IPsec ESP Tunnel Mode is used, a packet from 192.0.2.1 to 192.0.2.129 has the following encapsulation: Dunbar, et al. Expires Dec 24, 2027 [Page 38] Internet-Draft Multi-segment SD-WAN +---------------------------------------------------+ | Outer IP Header | | Protocol = 17 (UDP) | | Src IP = CPE1 (underlay address) | | Dst IP = Cloud GW1 (underlay address) | +---------------------------------------------------+ | UDP Header | | Src Port = xxxx (ephemeral) | | Dst Port = 6081 (GENEVE) | +===================================================+ | GENEVE Header | | Protocol Type = 0x0800 (IPv4) or 0x86DD (IPv6) | | [Indicates payload is an IP packet] | +---------------------------------------------------+ | MultiSeg-SDWAN Option Class | | SD-WAN EndPt Sub-TLV (CPE10 address) | | EgressGW Sub-TLV (explicit egress Cloud GW2) | | HMAC-Auth-Val Sub-TLV (GENEVE Hdr Authentication) | | (validated by GW) | +===================================================+ | ESP Outer IP Header (Tunnel Mode) | | Src IP = CPE1 (tunnel IP) | | Dst IP = CPE10 (tunnel IP) | | Protocol = 50 (ESP) | +---------------------------------------------------+ | ESP Header | | SPI (Security Parameters Index) | | Sequence Number | +---------------------------------------------------+ | Encrypted Payload | | Inner IP Header | | Src = 192.0.2.1 (host behind CPE1) | | Dst = 192.0.2.129 (host behind CPE10) | | Protocol = TCP | | TCP Header | | Application Payload | | Padding | | Pad Length | | Next Header | +---------------------------------------------------+ | Integrity Check Value (ICV) | | <- validated by CPE10 | +---------------------------------------------------+ Figure 11: Packet Encapsulation through Multiple Cloud GWs Dunbar, et al. Expires Dec 24, 2027 [Page 39] Internet-Draft Multi-segment SD-WAN Authors' Addresses Kausik Majumdar Upscale AI Email: kmajumdar@upscaleai.com Linda Dunbar Futurewei Email: ldunbar@futurewei.com Venkit Kasiviswanathan Nexthop AI Email: venkit@nexthop.ai Ashok Ramchandra Google Email: archiashok@gmail.com Dunbar, et al. Expires Dec 24, 2027 [Page 40] Internet-Draft Multi-segment SD-WAN Aseem Choudhary Cisco Email: asechoud@cisco.com Contributors' Addresses Ajeet Pal Singh Gill Microsoft Azure Email: ajeetgill@microsoft.com Dunbar, et al. Expires Dec 24, 2027 [Page 41]