MX7000 Networking – Part 2 : Operating Modes, IOM Selection

Carrying on from the previous post;

The first post on MX Modular Chassis, gave an overview of the the Components and IOM Characteristics of the MX7000 Modular System. This post will  focus on the options, use cases and differences between the modules. The third post will review sample designs. The fourth will have some further good practices.

We will start with a discussion of the Switch Operating Modes below.

Switch Operating Modes

Two Modes are currently available with the switches, with a third one being a roadmap item.

1. Smart Fabric Mode:
   • Layer 2 Mode.
   • There is a strong focus on Automation, Converged Management & elimination of reliance on CLI.
   • In a Multi-Chassis design, Smart Fabric Mode requires MX9116 Fabric Switching Engine. In a single Chassis design, It can use Ethernet switch MX5108.
2. Full Switch Mode: 
   • Full Featured Switch, Fully enabled CLI. Configure everything required.
   • LCM (Life Cycle Management) is still via OME-M (Open Manage) Interface
3. Open Networking Mode
   • Dell EMC Networking’s rich ON/SDN Ecosystem Solutions, already available on rack switches, may become available with Blade switches as well in the future.

Smart Fabric Vs. Scalable Fabric

Just to ensure there is no confusion between the use of these two terms,

• Smart Fabric is a Switch Operating Mode focused on L2 Aggregation and Automation. The alternative to Smart Fabric is Full Switch Mode.
• Scalable Fabric on the other hand, is a Fabric Architecture that involves Master Switching (FSE) and Expander Modules (FEMs). It is comparable to the model of Cisco UCS Fabric Interconnect & FEX . It can span upto 10x Chassis. Scalable Fabric requires MX9116n Fabric Switching Engines. It is not applicable to MX5108n Switches. The MX9116 FSE can be in either Smart Fabric or Full Switch Mode, when used in a Scalable fabric.

Smart Fabric Vs. Full Switch : When to use Which

Smart Fabric Mode

Use Smart Fabric mode when the following is required or satisfactory

• IO Modules as L2 Aggregation only (No Layer 3).
• Supports only a Single FC Zone
• VLT is auto configured with SF Mode. Thus, it is a requisite.

Full Switch

Use Full Switch mode for any scenario where SF Mode is not adequate. Use it when the following is required or satisfactory

1. Topologies requiring more Control, Granularity & Full CLI access on the IOMs.
2. Supports Multiple FC Zones
3. If we require L3 on the IOMs e.g. hosting VLAN SVIs/L3 boundary on IOMs.
4. If VLT is not desired, Full switch must be used (SFS auto configures VLT)
5. If we require functions/configuration available in a Full/Proper switch, e.g. Routing Protocols, ACLs etc.
6. Some examples:
   • Full Switch without a Scalable Fabric : An L3 Leaf & Spine design where the Spine is external to the Modular infrastructure, and we are looking to push Leaf layer down to individual chassis’ fabric A, for e.g. This will not be a Scalable Fabric topology built with expander modules. Rather, each chassis/fabric would be populated with MX91/51 switches, for the Leaf function. So,
     • Chassis 1 Fab A Switch A1 & A2, are a Leaf pair; then Chassis 2 Fab A Switch A1 and A2 are a leaf pair, and so on.
     • The Leaf pair will offer VLT multi-pathing southbound to multi-homed servers (When using switch dependent Link Aggregation), and ECMP Routed links Northbound to the spine.
     • As L3 boundary lies on Leaf pair, VLANs terminate locally on each Leaf pair in this design.
     • With L3 ECMP between Leaf & Spine, individual leaf switches could have Unique BGP ASNs when using EBGP.
     • We are still free to build the second fabric (B) as smart, scalable etc.
   • Full Switch with a Scalable Fabric : Connecting two Scalable fabrics (Note we are talking of Scalable, not Smart Fabric) directly, without going through a transit Core/Aggregation. for e.g. Connecting Scalable Fabric B in DC 1, to Scalable Fabric B in DC 2, directly & without traversing Core or Border Edge Switch block.
     • This only addresses FEM/FSE topologies in Full Switch mode, and not Smart Fabric mode, which cannot uplink to another smart fabric, and must connect to proper/full switch(es).
     • Note that without FEMs, it is not a scalable (or even viable) solution if each chassis will link individually across DCs with a counterpart. Unlike the MXL switch on M1000e, we do not (yet) have stacking available as an alternative to consolidate IOMs logically, so the use of FEMs with FSEs is the only solution that scales to multiple chassis in each location.

IOM Selection – When to use Which

I have highlighted the deficiencies/limitations of each contender, in Red.

Ethernet Switching & Fabric Expansion

MX5108

• General
  • East West Switching within the Chassis (Switching Fabric).
  • 4 External Base-T ports (10G). These ports can operate as 1G BT
  • Not a Scalable Switch. Single Chassis Config.
  • No Quad port NIC (Future) Support. Dual Port NIC Only.
• Storage Specific:
  • FSB (FIP [FCoE Initialization Protocol] Snooping Bridge) Mode Available. FSB placement is between an FCoE Server Node & an NPG Switch.
  • FCoE transit only, So it will FCoE uplink to a SAN Gateway Switch.
  • No FC.
  • No NPG (NPIV Proxy Gateway or Access Gateway)

MX9116

• General
  • Scalable Switch. Fabric Switching Engine in a Multi-Chassis Fabric.
  • Embedded ToR capability (Available today in Full Switch, Smart Fabric mode is roadmap)
  • East West Switching within the Chassis (Switching Fabric).
  • Can support (Future) Quad Port NICs.
  • Dense & Varied Complement of External Port – Q28DD, Q28SD, Q28Unified.
• Storage Specific:
  • Direct FC Storage Connectivity (F_port, Zoning on switch) [port 43/44]
  • NPG/NPIV-P gateway to traditional Brocade/MDS FC SAN
  • Unified ports support 8G, 16G, 32G.
  • Full Fabric Features (to connect to multi switch fabric) not available 
    • No E_Port. Connect to another FC switch via NPG only.
    • No FSPF – Routing Protocol for FC Fabric.
  • No FC Down To Server, only FCoE (Thus, CNA requisite on server).

MX7116

• Multi-Chassis, Extender/Expander
• No Management. No Maintenance. Simplified Architecture.
• No East West Switching locally.

Pass Through Modules

(10/25G SFP28, 10GBT)

• No Management. No Maintenance. Non-Switched.
• Neither PTM will do 1GbE.

Storage

MXG610

• FC to compute/server
• E_Port (Connect into FC Fabric)
• Management support on OME Modular
• Cannot become part of scalable fabric.

MX5000

• SAS Switching only.

Scalable Fabric Topology

Scalable fabric requires a minimum of 2 Chassis, and can span to a maximum of 10. Recommended FSE placement is across two chassis for redundancy, as depicted below.

For the connectivity between Switching Engines and Expander Modules, currently only DAC/AOC are qualified. This imposes a max distance of 3m when using DACs, and 20m when using AOC. Thus, the chassis cannot be >20m apart. Support for SR optics for FSE<>FEM connection is in the roadmap, which will allow longer distances between them.

You can actually connect switches (not expander) across fabrics, for e.g. A1 to B1. However, this can only be done in Full Switch mode. Smart Fabric Mode will not allow this, as it imposes identical fabric requirement.

For VLT, Smart Fabric mode selects the VLTi ports (enforced). In Full Switch, we can use whichever ports we want.

MX7116 FEMs act as an Ethernet repeater. On MX7108, do not use the rightmost QSFP28-DD port 2. It is reserved for future use, to connect to servers with quad-port NICs.

To close off this post, as a reminder from my dedicated post on Smart Fabric Services, SFS Cluster exposes a single Rest API end-point, to facilitate management of all IOMs. All (Switching) nodes in the cluster share Group CPS, Distributed DB, and talk CPS to each other.

In the next post, we will have a look at a sample design and related considerations.