AI-Ready Infrastructure: GPU Fabric Design, Lossless Ethernet, and Validation

The decision is a function of cluster scale, procurement posture, and operational talent. NVIDIA InfiniBand NDR is the incumbent for NVIDIA DGX SuperPOD reference architectures of 128 GPUs and up where maximum training performance and SHARP v3 in-network reduction are worth running a second fabric.

NVIDIA Spectrum-X (Spectrum-4 SN5600 800G switches plus BlueField-3 DPUs, Cumulus Linux 5.10 control plane) is the NVIDIA-stack Ethernet answer — right for operators who standardize on Ethernet end-to-end and have already chosen NVIDIA GPUs.

Arista Etherlink (7280R3 leaf, 7800R3 spine) is the standards-based Ultra Ethernet Consortium-aligned path — right for operators who want NVIDIA GPUs but not a NVIDIA-only switching fabric.

Below 64 GPUs, standard enterprise Ethernet with careful RoCEv2 tuning is usually sufficient; above 1,024 GPUs, InfiniBand or Spectrum-X becomes more defensible as the operational overhead amortizes over larger scale.

WFHS is vendor-agnostic — the design recommendation reflects the client’s constraints, not a vendor margin.

Standard leaf-spine fabrics built for web or VM workloads typically run 3:1 or 4:1 oversubscription in the leaf uplink — that is fine when traffic is bursty north-south and packet loss is recoverable by TCP. GPU training collective patterns (all-reduce, all-gather) stall the entire job on the slowest link because every GPU is synchronized.

Rail-optimized topology dedicates a non-blocking rail to each GPU within a host, then stacks identical rails across the cluster so every collective traverses the same rail number — which collapses fan-in congestion into predictable deterministic paths and allows 1:1 non-blocking leaf-to-spine oversubscription across the entire rail.

The oversubscription ratio for a training fabric GPU-east-west path must be 1:1; any oversubscription in that path stalls NCCL.

Storage, management, and out-of-band traffic can ride more conservatively oversubscribed parallel fabrics. At 512 GPUs with a 64-port 400G leaf radix, this produces 32 downlinks to GPUs and 32 uplinks to spine per leaf — non-blocking by construction.

Design the cable plant for 800G and the optical module procurement for whichever generation your deployment window aligns with. 800G-FR4 (QSFP-DD800 or OSFP800) is shipping in volume across NVIDIA Spectrum-X SN5600, Arista 7280R3/7800R3, Cisco Nexus 9364E-SG2, and Juniper QFX5230-64CD. 1.6T optics are on vendor roadmaps through 2026 and beyond but volume availability and interop maturity lag 800G by a year or two.

Practical guidance: specify MPO-16 or MPO-12 APC connectors and OS2 single-mode fiber rated for the 1.6T transition so you do not re-trunk the cable plant at upgrade.

Consider Linear Pluggable Optics (LPO/XPO) for short-reach inside-row links to cut per-port optical power 30–50% where channel budget permits. Avoid locking procurement to a single optics vendor for a 3–5 year deployment where 1.6T parts will be the incumbent for the back half of that window.

Four layers, each tested independently before the cluster is handed off for production training. Layer 1: every optic and every link validated for pre-FEC and post-FEC BER against transceiver specification; any marginal link swapped before higher-layer testing begins. Layer 2: iperf3 and InfiniBand perftest (ib_send_bw, ib_write_lat) or RoCEv2 equivalents run point-to-point between every host pair at full line rate, with 99th and 99.9th percentile tail latency captured.

Layer 3: NVIDIA nccl-tests all_reduce_perf, all_gather_perf, reduce_scatter_perf across ring sizes up to the full cluster, compared against theoretical ring algorithmic bounds — deviation over 10–15% indicates PFC, adaptive routing, SHARP, or ECMP hash configuration issues.

Layer 4: a 30–60 minute synthetic training run with deliberate fault injection (link down, switch reboot, GPU throttle) to confirm convergence and observability pipelines.

The validation deliverable is a signed acceptance document enumerating each test, measured result, and design target — the baseline document the client operations team inherits.

For most enterprise AI training clusters in the 64- to 512-GPU range, Ethernet is sufficient when it is engineered correctly. Spectrum-X or Arista Etherlink with RoCEv2, properly tuned PFC and DCQCN congestion control, deep-buffer leaf switches (Arista 7280R3, Cisco Nexus 9364E-SG2), and rail-optimized 1:1 non-blocking topology delivers training performance within 5–10% of InfiniBand NDR for most transformer and CNN workloads.

The case for InfiniBand strengthens at 1,024 GPUs and beyond, where SHARP v3 in-network all-reduce reduction delivers 20–40% wall-clock improvement on collective operations and the operational overhead of a dedicated fabric amortizes over larger scale.

The case against InfiniBand is straightforward: it is a second fabric with its own management plane, cable plant, and operational talent pool.

For an operator standardizing on Ethernet end-to-end with a Kubernetes-centric inference stack, Spectrum-X or Arista Etherlink eliminates that second fabric while preserving AI performance. The right answer is the one that matches your team’s operational maturity and the cluster’s performance headroom — not a vendor preference.

Cleanly, with phased switch and optic replacement anchored to the existing cable plant. Most enterprise fabric migrations preserve the OS2 single-mode structured fiber plant and replace transceivers, linecards, and switches in discrete phases.

Phase one is leaf replacement with 800G-capable platforms (Arista 7280R3, Cisco Nexus 9364E-SG2, Juniper QFX5230-64CD, or NVIDIA SN5600) while spine remains 400G; new GPU pods attach at 800G through the new leaf, and existing 400G GPU pods continue on the old leaf until scheduled migration.

Phase two is spine replacement to a matching 800G platform once enough leaves have migrated that spine-side bandwidth is the constraint.

The cable plant survives both phases if the original MPO-16 or MPO-12 APC terminations were rated for 800G; if they were not, the migration window requires re-termination.

The risk is running a mixed-speed fabric during the migration where ECMP hash polarization creates unbalanced flows — we validate ECMP behavior at every phase boundary. This is a standard network architecture migration workstream and is scoped as a fixed-fee SOW with a defined rollback checkpoint at the end of each phase.

SHARP v3 is NVIDIA’s Scalable Hierarchical Aggregation and Reduction Protocol — an InfiniBand switch-side feature that offloads the NCCL reduction operation from the GPU fabric traffic pattern into the switch ASIC itself. On a traditional ring all-reduce, every GPU sends and receives data from its ring neighbors across multiple hops; the total time scales with the ring size.

With SHARP, the reduction aggregation happens in the Quantum-2 switch, so each GPU exchanges data with the switch once rather than traversing the ring.

The measured wall-clock improvement on transformer training all-reduce operations is typically 20–40%, with larger gains at larger cluster sizes where ring hop count would otherwise dominate. SHARP v3 extends the feature set with streaming aggregation and expanded collective operation support.

The operational requirement is NVIDIA Quantum-2 InfiniBand switches and ConnectX-7 or ConnectX-8 NICs with SHARP licensing enabled; it is not available on the Ethernet fabric. For clusters where all-reduce is the dominant operation (most dense transformer training), SHARP is a material reason to choose InfiniBand over Ethernet at scale.

Power and cooling planning starts the same day as the fabric design, not after the switch order ships. An 8-GPU NVIDIA HGX H100 node draws 10.2 kW at sustained load; an HGX H200 is similar; an HGX B200 lands near 14 kW per node.

A 4-node rack is 40–56 kW — three to five times the density of a legacy enterprise rack. That density crosses the 30 kW threshold where direct-to-chip liquid cooling (DLC) is mandatory per ASHRAE TC 9.9 technical guidance, with ASHRAE liquid cooling Class W32 or W45 supply temperatures covering most deployments.

The network SOW includes a power budget per rack for switches, optics, and DPUs; a separate management-fabric budget that must remain on UPS; and coordination deliverables with the facility engineer and the DLC vendor so rack-top cable management is compatible with coolant manifold routing before the first cable is pulled.

WFHS does not price PDUs, chillers, or CDUs — we are network engineers — but we size the network power envelope, flag conflicts between DLC and cabling paths, and hand the facility engineer a parts-counted power budget rather than a hand-wave.

Where structured cabling requires greenfield design or remediation, that is scoped as a parallel workstream.

The SN5600 delivers 64 OSFP ports in a 2RU form factor (3.46″ high, 17.2″ wide, 28.3-29.3″ deep) with an aggregate switching capacity of 51.2 Tbps. Each port supports 10, 25, 50, 100, 200, 400, and 800G Ethernet, which lets a single SN5600 back-end leaf connect 64 GPU hosts at 800G or fan out to higher port counts at 400G or 200G.

Typical ATIS power draw with passive cables is 940 W.

The control plane runs on an Intel Xeon E-2276ME hexa-core Coffee Lake CPU with 32 GB DDR4 and a 160 GB SSD, giving Spectrum-X telemetry and RoCE congestion-control features enough headroom for large fabrics.

The Quantum-X800 Q3400 provides 144 ports of 800 Gb/s XDR InfiniBand connectivity per chassis. It includes hardware-based, in-network computing using SHARP v4 for collective offload, along with adaptive routing and telemetry-based congestion control tuned for training traffic. A dedicated port connects the switch to Unified Fabric Manager, which is the NVIDIA management platform for InfiniBand scale-out fabrics.

For training clusters where deterministic tail latency matters more than Ethernet interoperability, the Quantum-X800 is the InfiniBand option we design against when a customer specifies IB.

We stay vendor-agnostic and will also design a Spectrum-X Ethernet fabric if Ethernet is the operational requirement.

NVLink 5 delivers 1,800 GB/s of bidirectional throughput per Blackwell GPU across 18 NVLinks, which NVIDIA cites as over 14x the bandwidth of PCIe Gen5. NVLink 4 on Hopper provides 900 GB/s per GPU, so NVLink 5 exactly doubles per-GPU bandwidth generation over generation. That matters at fabric design time because tensor-parallel and pipeline-parallel split sizes change when the scale-up domain runs at 1.8 TB/s instead of 900 GB/s.

We design the back-end Ethernet or InfiniBand fabric knowing where NVLink ends and scale-out begins so you do not spend 800G ports on traffic that should have stayed inside the NVLink domain.

A GB200 NVL72 rack connects 72 Blackwell GPUs and 36 Grace CPUs in a single NVLink domain with 130 TB/s of aggregate low-latency GPU-to-GPU communication. Nine NVLink switch trays sit in the rack, and each tray carries 144 ports at 100 GB/s. The unified memory pool is approximately 30 TB per rack, with HBM3E providing 576 TB/s of aggregate memory bandwidth.

NVFP4 Tensor Core performance reaches 1,440 PFLOPS per rack.

Because the NVLink domain is 72 GPUs, everything beyond that goes out over the scale-out fabric, which is where our Ethernet and InfiniBand back-end design work lives.

The HGX B200 baseboard carries 8 NVIDIA Blackwell SXM GPUs, fifth-generation NVLink, and an on-board NVLink 5 Switch providing 14.4 TB/s of total NVLink Switch bandwidth. Total GPU memory across the eight-GPU configuration is 1.4 TB. HGX B200 is the 8-GPU reference platform most enterprise AI builds start from before scaling up to NVL72 or scaling out across multiple baseboards with a Spectrum-X or Quantum-X800 back-end.

When we scope your rack and fabric design, HGX B200 vs GB200 NVL72 changes the back-end port count, rail-optimized spine sizing, and the cooling profile you plan for.

The Arista 7800R4 family has four chassis sizes for AI spine roles. The 7816LR4 is a 16-slot chassis with 576 ports of 800 GbE and 460 Tbps of switching capacity, quoted as 920 Tbps full-duplex, with 173 Bpps forwarding. The 7812R4 is 12-slot, 432 ports of 800 GbE, 345 Tbps. The 7808R4 is 8-slot, 288 ports of 800 GbE, 230 Tbps.

The 7804R4 is 4-slot, 144 ports of 800 GbE, 115 Tbps.

Per-linecard buffer is 32 GB, with 512 GB total in the 16-slot chassis. Three 36-port 800G linecards are available: 7800R4C-36PE for accelerated compute, 7800R4-36PE for general data center, and 7800R4K-36PE for service-provider scale.

The 7800R4 enables advanced RDMA load balancing, optimized DCQCN, ECN, and PFC congestion management, and accelerated sFlow for AI workload visibility. Arista’s stated design goal is “RDMA Aware QoS and load balancing for reliable RoCE packet delivery,” which is the specific behavior RoCEv2 fabrics need when collective operations pile up at the spine.

The Etherlink family is explicitly forward-compatible with Ultra Ethernet Consortium standards, so a fabric designed around 7800R4 today does not become stranded when UEC 1.0.2 features move from specification to production silicon.

We design the DCQCN thresholds and PFC watchdogs against the specific GPU scale-out pattern, not against a generic data-center template.

The Arista 7060X6-64PE and 7060X6-64PE-B are 2RU switches with 64 ports of 800G OSFP800 and 51.2 Tbps of switching capacity. The 7060X6-32PE is a 1RU variant with 32 ports of 800G and 25.6 Tbps. Latency starts at 700 nanoseconds, which is the class of latency you want at the AI leaf when the east-west fabric is carrying NCCL AllReduce traffic.

OSFP800 ports support breakout to 400G, 200G, 100G, 50G, and 10G, and the switches support hitless speed changes so a rack can transition from 400G to 800G GPU hosts without tearing down the leaf.

We pick 7060X6 for rail-optimized leaf roles where port density and sub-microsecond latency matter more than deep buffering.

The Arista 7280R4-32DE and 7280R4-32PE each carry 32 ports of 800G with breakout to 256 ports of 100G. The 7280R4-64QC-10PE combines 64 ports of 100G QSFP with 10 ports of 800G OSFP. Non-blocking bandwidth on the 32-port models is 25.6 Tbps,

with 9.6 Bpps line-rate forwarding and class-leading 3.5 microsecond latency. Dynamic deep buffer capacity reaches 32 GB per system, and Virtual Output Queues prevent head-of-line blocking under burst conditions.

The Etherlink for AI feature set adds the AI Analyzer powered by AVA plus advanced RDMA load balancing and DCQCN, ECN, and PFC tuning.

We deploy 7280R4 where the leaf must absorb mixed storage and scale-out traffic without microburst drops.

CloudVision Network Data Lake (NetDL) provides real-time state streaming for network telemetry and analytics across the Arista fabric. NetDL stores both live state for real-time monitoring and historical state for post-event analysis,

and it is the training substrate for the AI and ML models that CloudVision uses to flag anomalies. Autonomous Virtual Assist (AVA) runs proactive risk analysis of configuration changes before deployment, which catches fabric-wide regressions before they hit production.

CloudVision is deployable either as cloud-native SaaS or as an on-premises appliance, which matters for regulated tenants.

We integrate NetDL with your existing observability stack so the AI fabric does not become a telemetry island.

The current downloadable Ultra Ethernet Consortium specification is UEC 1.0.2, with a 1.0 whitepaper and supporting video published on ultraethernet.org. UEC scope is stated as “a complete architecture that optimizes Ethernet for high performance AI and HPC networking,” targeting improvements in bandwidth, latency, tail latency, and scale. It matters at design time because an AI Ethernet fabric built today needs a forward-compatibility path to UEC.

Arista has publicly stated Etherlink products are forward-compatible with UEC standards, which is one of the reasons we lean toward Arista 7060X6, 7280R4, and 7800R4 when the customer has ruled out InfiniBand and wants a multi-generation Ethernet investment.

ECN is defined in IETF RFC 3168 and uses a 2-bit field in bits 6 and 7 of the TOS octet in IPv4, or the Traffic Class field in IPv6. The four codepoints are Not-ECT (00), ECT(0) (10), ECT(1) (01), and CE (11). When Active Queue Management on a router detects congestion on an ECT-marked packet, the router may set the Congestion Experienced codepoint rather than dropping the packet.

TCP adds ECE and CWR flags for sender and receiver ECN negotiation and response.

In AI RoCEv2 fabrics, ECN marking drives DCQCN, which is the feedback loop that slows down NIC transmit rate before the fabric starts dropping traffic and triggering retransmissions.

NCCL is NVIDIA’s inter-GPU communication library that provides topology-aware primitives across PCIe, NVLink, InfiniBand, and standard IP. It accelerates eight primary collectives: AllReduce, Broadcast, Reduce, AllGather, ReduceScatter, AllToAll, Gather, and Scatter. It also supports point-to-point send and receive for custom communication patterns. NCCL integrates with CUDA streams and its C API mirrors MPI conventions, which is why most training frameworks expose NCCL as the default backend.

When we design a back-end fabric, the NCCL traffic pattern (AllReduce dominates most training workloads) drives rail-optimized topology decisions and the spine-to-leaf oversubscription ratio.

The Arista 7700R4 Distributed Etherlink Switch supports over 30,000 400 GbE accelerators in a single-hop system. The architectural intent is a single-hop fabric for very large AI clusters, which reduces collective-operation latency by collapsing what would otherwise be a three-tier Clos into one logical hop.

For sites already planning a 10,000+ GPU buildout on Ethernet, 7700R4 changes the topology math because the back-end fabric stops looking like a traditional spine-leaf and starts looking like a chassis-extended distributed switch.

We scope this against InfiniBand alternatives so the decision is made on operational economics, not on vendor preference.

The Juniper QFX5240 is the current flagship 800G AI data center switch, with up to 64 ports of 800 GbE in QSFP-DD or OSFP and up to 102.4 Tbps of bidirectional throughput. Breakout options include 128 ports of 400 GbE, 256 ports of 100 GbE, or 256 ports of 50 GbE. Juniper positions it explicitly as offering “up to 800GbE interfaces to support AI Data Center Networking deployments.”

The QFX5230 fills the 400G secondary AI role with 64 ports of 400 GbE QSFP56-DD and 51.2 Tbps.

We design against QFX5240 when a customer is standardized on Juniper Apstra or needs a JunOS operational model across the fabric, and we stay vendor-agnostic on the core decision.

The Arista 7800R3 family spans four chassis sizes. The 7816LR3 and 7816R3 are 16-slot with 460 Tbps switching capacity. The 7812R3 is 12-slot, 345 Tbps. The 7808R3 is 8-slot, 230 Tbps. The 7804R3 is 4-slot, 115 Tbps. Forwarding rate reaches 96 Bpps, with 14.4 Tbps of fabric per line card. Total buffer in a full 16-slot chassis is 384 GB, with 24 GB per 400G line card.

Class-leading latency is 3.5 microseconds.

The platform uses Virtual Output Queues and a cell-based redundant fabric to avoid head-of-line blocking, and integrated MACsec, IPsec, and VXLANsec via TunnelSec run at 10G through 400G for encrypted fabric segments.

IEEE Std 802.3df-2024 standardizes 400 Gb/s and 800 Gb/s Ethernet, and was approved on 16 February 2024 following IEEE-SA Standards Board ratification. The task force work completed at that milestone, which means every 800G product shipping into AI fabrics today references a fully ratified IEEE standard, not a pre-standard vendor extension. That is why we can design 800G leaf and spine layers with confidence that DR4, FR4, and LPO optics from different vendors will interoperate.

When a procurement team asks whether 800G is “production” or “early access,” the correct answer is: the IEEE standard has been approved, silicon is shipping, and optics are in volume.

NVIDIA ConnectX-7 supports up to four ports with 400 Gb/s aggregate throughput. Its feature set includes ASAP2 network acceleration, advanced RoCE support for lossless RDMA, GPUDirect Storage offload that moves storage traffic directly between NVMe and GPU memory, and hardware-accelerated TLS, IPsec, and MACsec encryption.

In an AI back-end fabric, ConnectX-7 is the NIC that does the RoCEv2 heavy lifting between the GPU host and the Spectrum-4 or Quantum-X800 switch, with DCQCN and ECN handling running in silicon rather than in the host stack.

We size the NIC-to-switch speed match so a 400G ConnectX-7 host does not end up behind an oversubscribed 200G leaf port.

Unified Fabric Manager is the NVIDIA management platform for InfiniBand scale-out computing fabrics. It provides real-time monitoring and control across the fabric, plus telemetry collection, threshold-crossing events, and alarms. UFM manages devices, ports, virtual ports, cables, groups, PKeys, and user access, which is the full set of operational objects an InfiniBand administrator touches.

It supports standalone and high-availability configurations, and runs either bare-metal or in Docker.

A plugin architecture covers SNMP, telemetry streaming, link maintenance, and cluster integration. We integrate UFM with customer observability and automation tooling so the IB fabric is not operated out of a separate console from the rest of the data center.

NVIDIA publishes current DGX SuperPOD reference architectures for DGX GB200, DGX B200, DGX B300, and DGX H200. The DGX B300 reference architecture ships in two variants: one with Spectrum-4 Ethernet and DC Busbar Power, and one with Quantum-X800 InfiniBand and AC Power.

Each RA documents how compute, networking switches, software, and storage components integrate in a SuperPOD configuration, which is the document set a design engineer works from when translating a GPU count target into a rack-by-rack build list.

We design customer deployments against these RAs when a customer wants to stay on the NVIDIA reference path, and we adapt them against Arista or Juniper fabrics when a customer has an existing network standard.