VXLAN with BGP EVPN [Part 3] – Features Comparison between Dell EMC OS10 + OS9, Cumulus Linux, Pluribus NetVisor Linux

This is the third post in the “VXLAN with EVPN” Series of posts. The focus is on features available for various control plane options with VXLAN, across the Dell EMC Networking Eco System. I had originally planned to follow up this post with a further two, focusing on the configuration. However, I feel it would be best if I commit those posts following the release of our next code, which brings several key enhancements to the solution. As such, you will probably see a break of a few weeks before the next posts arrive.

Please treat the content here as indicative only. With new releases for the different platforms, portions or sections of this post may not hold true by the time you read/visit this page. It should still be useful as a baseline for what is supported/required, for the intended solutions.

VXLAN Control Plane Options

Dell EMC OS10 VXLAN

• Protocol Based : BGP EVPN (10.4.2)
• Controller Based : NSX
• Static/Manual Config

Dell EMC OS9 VXLAN

• Controller Based : NSX
• Static/Manual Config
• No BGP EVPN option for OS9

Cumulus VXLAN

Cumulus Linux includes native Linux VXLAN kernel support. It supports :

• EVPN
• Static
• Controller Based – NSX + Midonet + OpenStack
• LNV : deprectaed as of 3.7.4, Removed from 4.0. 

Pluribus NetVisor Linux

• Fabric Control plane nvOSd (L2) is based on shared endpoint database (vPort)
• EVPN for Multi-fabric Federation (Roadmap)

Big Switch BCF

• Multicast
• BGP EVPN : Roadmap

HW Support

Features for

• L3/Routing Gateways

  • Routing traffic across VNIs.
  • Execution :
    • Dual pass refers to : Routing first on the outer header IP address, and then again on the decapsulated packet. (Dual pass is supported on OS9 only. No support on OS10)
    • RIOT Stands for Routing in and Out of Tunnels. It indicates pipeline or native support on the chip, for routing function. Within Dell EMC family of switches, this support is found on Broadcom Trident 2+, Maverick and Qumran chips. 
  • Broadcom Tomahawk : Dual Pass, Internal (supported only on OS9)
  • Broadcom Trident 2 : Dual Pass, External (supported only on OS9)
  • Broadcom Trident 2+ : Native/RIOT Profiles (OS10 support available)

• L2/Bridging Gateways

  • Bridge Traffic within a VNI

Dell EMC OS10 VXLAN

• with BGP EVPN

  • L3/Routing

    • 10.4.3 : S41xx, S42xx, S52xx

  • L2/Bridging: 

    • 10.4.3 : S41xx, S42xx, S52xx, S40xx, S60xx, S61xx, Z91xx, Z92xx

• with Static Config

  • L3/Routing

    • 10.4.3 : S41xx, S42xx, S52xx

  • L2/Bridging: 

    • 10.4.3 : S41xx, S42xx, S52xx, S40xx, S60xx, S61xx, Z91xx, Z92xx

• with NSX (Controller Based)

  • L3/Routing

    • Future

  • L2/Bridging: 

    • 10.4.3 : S41xx, S42xx, S52xx, S40xx, S60xx, S61xx, Z91xx, Z92xx

Fabric Scope/Atomic Commands

Dell EMC OS10 VXLAN with EVPN

• SFS – Smart Fabric Support, Roadmap

Dell EMC OS10 VXLAN with Static

• No

Cumulus VXLAN with EVPN

• No

Pluribus NetVisor Linux

• Yes

Big Switch BCF

• Yes

Peering Choices

BGP EVPN details/behaviour

• Underlay : Exchange IP Routes (CLOS, Non-EVPN addresses)
• Overlay : Exchange EVPN Routes (b/w VTEPs, EVPN addresses)

Options

• IGP/iBGP
• eBGP/iBGP
• eBGP (Dual Peering)
• eBGP (Single Peering, Separate address family adverts)

Dell EMC OS10 VXLAN with EVPN

• OSPF/iBGP : 10.4.2
• eBGP/iBGP : 10.4.3
• eBGP (Dual Peering): 10.4.2
• eBGP (Single Peering): 10.5.x

Dell EMC OS10 VXLAN with Static

• No Routing/Control in Overlay
• OSPF/None : 10.4.2
• eBGP/None : 10.4.2

Cumulus VXLAN with EVPN

• OSPF/iBGP
• iBGP/Route Reflectors between Spines
• eBGP (Single Peering)

Pluribus NetVisor Linux

• Underlay : OSPF, i/eBGP
• Overlay : BGP configured only on Border Leaf.
• Pluribus does not rely on underlay routing protocol for exchanging overlay information, such as location of endpoints etc. Fabric Control Plane communication does happen over IP underlay, using in-band int.

eBGP Single Peering support

Dell EMC OS10 VXLAN with EVPN

• 10.5.x

Dell EMC OS10 VXLAN with Static

• N/A. Overlay is not using Protocol based control plane to signal FDB.

Cumulus VXLAN with EVPN

• Yes

Pluribus NetVisor Linux

• Single Session N/A.
• In EVPN, Overlay info exchange happens over eBGP, AFI/SAFI : L2VPN/EVPN.
• In Pluribus, Overlay info exchange happens through the Fabric control Plane. Only underlay eBGP (or OSPF) peering needed.

Multi-Tenancy Support

BGP EVPN details/behaviour

Route Distinguisher & Route Target, enable overlapping addresses.

Dell EMC OS10 VXLAN with EVPN

• VRF and P,V mode : 10.4.3
• L2 Multi Tenancy : Port+VLAN to VNI Map = 10.4.2
• L3 Multi Tenancy : VNI-to-VRF (Interface Virtual Network mapping to VRF) = 10.4.3
• Auto RD + RT : With eBGP underlay, “Auto” mode for RD/RT is only available if you have the same AS for Spine & Leaf layers
• Manual RD + RT : If Spine/Leaf each have unique AS, we need to manually configure RD & RT.

Dell EMC OS10 VXLAN with Static

• VRF and P,V mode = 10.4.3
• L3 Multi-tenancy = 10.5.0

Cumulus VXLAN with EVPN

• Yes

Pluribus NetVisor Linux

• Private vNETs. A Per tenant, fabric-wide container object, with isolated data, control & management planes. Full 4K vlans per tenant.
• VRFs are always assigned to/within a vNet.

Route Types [Type 1-6]

BGP EVPN details/behaviour

• Minimum Required : Type 2, 3, 5 .
• RT-1 – Ethernet Auto-Discovery Route
• RT-2 – [MAC + IP] Host Route + MAC Mobility Seq. number
• RT-3 – VTEP Discovery + VNI Membership per VTEP (Inclusive multicast Ethernet tag)
  • + Replication list for BUM Traffic
  • + BUM Replication Model signalling.
• RT-4 – BGP Multi-homing DF Election – (Ethernet Segment Route)
• RT-5 – Advertise IP Subnet Prefixes – IP Prefix Route, Summarized. Route to external destinations.
• RT-6 – Optimize Overlay Multicast – (Selective Multicast Ethernet Tag Route)

Dell EMC OS10 VXLAN with EVPN

• Type 1 : Future – 10.5.x
• Type 2
  • MAC only = 10.4.2,
  • MAC+IP = 10.4.3
• Type 3 : 10.4.2
• Type 4 : 10.5.x
• Type 5 : Future – 10.5.x
• Type 6 : Future

Dell EMC OS10 & OS9 VXLAN with Static

• N/A. No Protocol based exchange. Remote MAC + IP Per VNI, learnt by flooding

Cumulus VXLAN with EVPN

• RT 2, 3 : Supported
• RT 5 : Supported
• RT 6 : 4.1

Pluribus NetVisor Linux

• RT-1 : No Support.
• RT-2 : Fabric Control Plane nvOSd updates the virtualized, shared address database, (vPort), with host/reachability info.
• RT-3 : VTEP config, VNI <> VTEP Map are manual. Tunnel Creation & VNI add to tunnel is automatic.
• RT-4 : No Support
• RT-5 : VRF/Subnet

Fabric translates BGP EVPN routes into vPort messages to:

1. Automate tunnel creation to external VTEPs
2. Create entries for any learned L2 or L3 entry

Extended BGP Communities 

BGP EVPN details/behaviour

• MAC Mobility : Tables updated to point to current location. advertises :
  • Sequence numbers, along with
  • MAC+IP address, in
  • Type 2 routes.
• Default Gateway : Advertise
  • Default Gateway (Requisite for Centralized Routing. Or use Static/Default Route Entries, if not available)

Dell EMC OS10 VXLAN with EVPN

• MAC Mobility : Needs Validation
• Default Gateway Extended Community : Future – 10.5.x

Dell EMC OS10 & OS9 VXLAN with Static

• N/A

Cumulus VXLAN with EVPN

• Yes

Pluribus NetVisor Linux

• Yes, via vPort forwarding & ARP optimization.
  • When VMs move, the process of re-registration in the vPort database happens automatically

VXLAN HA (High Availability)

Example Details/behaviour

• Switch Virtualization : Cluster
• L2 HA (L2 Multi-path) : MLAG/VLT
• L3 without VLT : BGP Multi-home
• L3 Routing with L2 HA
• L3 HA in underlay : VRRP
• L3 HA & Routing Optimization in Overlay : Distributed Anycast Gateways

Type 1 & 4 Routes are used for multi-homed nodes. RT 1 = Common Devices/Ethernet Segments between Switches.

Dell EMC OS10 VXLAN with EVPN

• L2 (VLT) 10.4.3
• L3 Routing with L2/VLT HA in Symmetric Model : 10.5.x
• L3 Routing with L2/VLT HA in Asymmetric Model : 10.4.3
• HA Without VLT = BGP Multi-home : 10.5.x

Dell EMC OS10 VXLAN with Static

• L2 (VLT) 10.4.3
• L3 Routing with L2/VLT HA : 10.4.3

Dell EMC OS10 VXLAN with Controller (NSX)

• Stand-alone Mode : 10.4.3.
• L2/VLT Based HA : Future – 10.5.x

Dell EMC OS9 VXLAN with Static

• L2 (VLT) 9.13

Cumulus VXLAN with EVPN

• L2 HA = MLAG
• EVPN Multi-homing : 4.1

Pluribus NetVisor Linux

• L2 HA (Dual Active Control Plane, Multi-Chassis LAG) : VLAG/Cluster
• L3 HA for Underlay/VTEP : VRRP
  • VTEP IP = VRRP VIP
  • VTEP object can also reference a Physical IP (PIP) for both ip and virtual-ip properties, but in this case the tunnel would cease to operate if the switch hosting PIP, fails.
• L3 HA for Overlay + Routing Optimization : Distributed Anycast Gateways (2.6.0)

Discovery, Mapping of VNI, VTEP etc. 

Example BGP EVPN details/behaviour

• Local VLAN <> VNI Mapping = Manual.
• Remote VTEP Discovery : Auto/ eBGP. Type 3 Routes.
• VNI <> Remote VTEP Mapping = Type 3

Outside EVPN, VTEP Flood lists can be configured statically or via Controller.

Dell EMC OS10 VXLAN with EVPN

• Local VLAN <> VNI Mapping = Manual.
• Remote VTEP Discovery = Auto/ eBGP. Type 3 Routes.
• VNI <> Remote VTEP Mapping = Type 3

Dell EMC OS10 & OS9 VXLAN with Static

• Static Map Profiles <> VTEPs.
• VTEP Flood list [VTEP <> VNI Maps] Statically Configured.

Cumulus VXLAN with EVPN

• Local VLAN <> VNI Mapping = Manual.
• Remote VTEP Discovery : Automatic, eBGP. Type 3 Routes.
• VNI <> VTEP Mapping : Type-3

Pluribus NetVisor Linux

• Static VLAN <> VNI Mapping
• Static VNI <> VTEP Mapping

Once VTEP created & VNI <> VTEP mapping executed,

• Auto Remote VTEP discovery
• Auto Tunnel Creation to other VTEPs (2.6.0)
• Auto VNI Add to tunnels (3.0.4)

End host detection (Address Registration and Tracking)

Details/behaviour

• Traditional Switch – Localized Flood & Learn. Initial MAC address learning over data plane e.g. via GARP/ARP.
• Behaviour consistent across all the solutions being discussed.

Table Installation & Aging

BGP EVPN Details/behaviour

• Install all entries in Hardware, regardless of active or not.
• Kernel does not ARP refresh or Age out entries learnt via a protocol, the protocol removes these when the advertisements are withdrawn.

Dell EMC OS10 VXLAN with EVPN, OS10/9 Static VXLAN, Cumulus

• Behaviour consistent and identical, as mentioned above.

Pluribus NetVisor Linux

• Conversational/vPort Forwarding : Only active destination entries installed in HW. When needed, dormant entries can be retrieved from vport database, and installed.
• Fabric endpoints can scale in software-based vPort database to half a million, or 10 times higher than a typical switches. Entries in vPort db are persistent. Entries in HW table are aged out dynamically.

End Host Distribution ( Address Propagation ) + Remote Host Learning

BGP EVPN Details/behaviour

• After local host detection, entries are installed locally on a switch/VTEP.
• The local MP-BGP process learns MAC entries (HW Table) & IP Route (ARP/ND Table)
• the MP-BGP Process then (Unicast) advertises MAC+IP Route to remote VTEPs, via EVPN Route Type 2.
• As a consequence, Remote VTEPs learn host MACs which are not local to them.
• Outside EVPN, Host [MAC + IP Routes] can be configured via Controller.
• Remote end: Received MAC+IP routes placed in BGP table.
  • If route target community sent with route, matches a local VNI route target import, route will be placed in switch’s MAC forwarding table, with appropriate VXLAN tunnel as its destination.
  • IP address, if included, will be placed in the EVPN ARP cache (Used for ARP Suppression)

Dell EMC OS10 VXLAN with EVPN

• EVPN Route Type 2 (Unicast to Remote VTEPs). Remote Host MAC Learning = in Control Plane via Type 2 Routes

Dell EMC OS10 VXLAN with Static

• Underlay Multicast/Flood & Learn (Propagation via Flood/Learn MC, regardless of BUM handling being HER or underlay MC).
• Each VNID Maps to a MC Group in the underlay.
• Remote Host MAC Learning = data plane learning after de-cap of VXLAN header

Cumulus VXLAN with EVPN

• EVPN Route Type 2 (Unicast to Remote VTEPs). Remote Host MAC Learning = in Control Plane via Type 2 Routes

Pluribus NetVisor Linux

• Pluribus does not rely on underlay routing protocol for exchanging overlay information, such as location of endpoints etc.
• Detected hosts are registered in vPort db.
• Once installed locally, the table is always in Sync between all members, so entries are updated in all vPort copies across the fabric.
• From vPort, it will also be downloaded to the switch HW forwarding table, and remain there as long as active.

BUM Traffic Replication/handling

BGP EVPN Details/behaviour

There are two ways for handling the BUM traffic

1. Head End/Ingress Replication
   • Ingress NVE replicates
   • Unicast.
   • Uses Rep/flood lists.
   • No Multicast support needed from underlay
   • In HER/IR, be mindful of scale. 
2. L3 Underlay Multicast
   • Single Copy to Spine, which replicates.
   • Multicast support needed in Underlay

With regards to BGP EVPN, there is use of Route Type 3 for

• VTEP Replication List
• BUM Replication Model signalling.

Dell EMC OS10 VXLAN with EVPN

• Headend/Ingress Replication : 10.4.2
• Underlay Multicast : Future – 10.5.x
• Unknown destination : flood to every VTEP interested in the VNI.

Dell EMC OS10 VXLAN with Static

• Headend Replication 10.4.3
• Underlay Multicast : Future – 10.5.x.

Dell EMC OS9 VXLAN with Static

• Headend Replication 10.4.3
• No Multicast Support

Cumulus VXLAN with EVPN

• Headend (Ingress) Replication is EVPN’s only/Default option. (done in HW)
• Cumulus verified Scale : 128 VTEPs.
• Service Node (Spine) Replication was previously possible in Cumulus with LNV, but not with EVPN. It was done in SW. With LNV being depreciated, this option is irrelevant now.

Pluribus NetVisor Linux

• Headend/Ingress Replication.
• vPort (conversational) forwarding.
• For silent hosts, Flood DMACs throughout the fabric.
• No Multicast support.

Forwarding Optimizations

BGP EVPN Details/behaviour

• ARP optimization = ARP Suppression / Proxy ARP : Suppress ARP Flood/BC on the VXLAN tunnels. Switch handles Access Client ARP request for remote hosts, via ARP proxy.
• Distributed Anycast Gateway ( Same virtual MAC+IP gateway addresses on all first-hop switches). Route E-W Traffic in every leaf, reduce hair pinning.
• Split Horizon
• BFD (Enhance Failover times)

Dell EMC OS10 VXLAN with EVPN

• ARP/ND Suppression : 10.5.x
• IP Support in Type 2 MAC+IP Routes : With 10.4.3, there is now support for the IP portion of MAC+ IP Routes in Type 2. Previously, there was support only for the MAC portion. Also, the IP portion is what enables ARP suppression.
• Distributed AnyCast Gateway : 10.4.3
• BFD : Future – 10.5.0

Dell EMC OS10 VXLAN with Static

• Distributed AnyCast Gateway 10.4.3 (Statically Configured)

Cumulus VXLAN with EVPN

• ARP/ND Suppression : 3.2.1
• Distributed AnyCast Gateway : 3.5
• BFD : 2.5

Pluribus NetVisor Linux

• Conversational/vPort Forwarding : Efficient use of L3 HW Table entries (The actual endpoint conversations in progress which require HW forwarding, only install them in HW tables).
• ARP optimization (Proxy ARP – suppress ARP flood/BC as long as remote host ARP info present on switch [vPort db] connected to remote host)
• Distributed Anycast Gateway
• Split Horizon
• BFD : 3.1

Routing (Where executed)

BGP EVPN Details/behaviour

Two options (Covered in my previous post, 5b)

• Centralized
• Distributed (@ first Hop)

Dell EMC OS10 VXLAN with EVPN

• Distributed : 10.4.3
• Centralized : Future – 10.5.x

Dell EMC OS10 VXLAN with Static

• Distributed : 10.4.2
• Centralized : 10.4.3

Cumulus VXLAN with EVPN

• Distributed : Supported
• Centralized : Supported

Pluribus NetVisor Linux

• Distributed : Supported
• Centralized : No Support

Routing/IRB (How Executed)

BGP EVPN Details/behaviour

• Two Options – Symmetric or Asymmetric. Covered in my previous post, 5b.
• Asymmetric can work with Default VRF.
• Symmetric requires non-default VRF support as a must. Uses VRF VNI as transit between ingress & egress NVEs.

Dell EMC OS10 VXLAN with EVPN

• Symmetric = 10.5.0
  Asymmetric = 10.4.3

Dell EMC OS10 VXLAN with Static

• Asymmetric : 10.4.3

Cumulus VXLAN with EVPN

• Asymmetric : 3.4
• Symmetric : 3.5

Pluribus NetVisor Linux

• Symmetric IRB (Inter-op) : 3.2
• Asymmetric : Supported

---

In the next two posts, I intend to give an overview of the Configuration details. Hopefully, I would be able to find the time to work on the content, not too long after the next release is out.

until then, take care !