QYTANG VIP PROGRAM · Nvidia

NVIDIA AI Supercomputing Networking Comprehensive Practice (AIN2027)

重点课程Public preview1 videos363 curriculum items

COURSE INTRODUCTION

Course Introduction

A fully upgraded NVIDIA AI supercomputing networking course built around RDMA. It covers the RoCE protocol and lossless Ethernet (PFC/ECN), hands-on Cumulus and SONiC switching, InfiniBand with UFM-based operations, GPU topology and model parallelism, and NCCL collective communication testing and tuning, plus new demonstrations of DPU network offload, Kubeflow distributed training, and vLLM inference. Labs run on a Kubernetes platform with real NVIDIA switches, NICs, and GPU servers, emphasizing verifiable, troubleshootable AI network design and operations.

CURRICULUM

Course Curriculum

00
1. 0.1
  Networking, Hostnames, and SSH
  Plan hosts, VIPs and RoCE cabling; set up name resolution and passwordless SSH.
  1. 0.1.1
    Host and VIP Planning
  2. 0.1.2
    RoCE Physical Cabling
  3. 0.1.3
    Hostnames, /etc/hosts, and Passwordless SSH
  4. 0.1.4
    Network Checks and Port Planning
2. 0.2
  Master DNS
  Deploy dnsmasq on the master and point every node to it.
  1. 0.2.1
    DNS Record Planning
  2. 0.2.2
    Installing dnsmasq on the Master
  3. 0.2.3
    Pointing Nodes to Master DNS and Verifying
3. 0.3
  OS, containerd, and Kubernetes Tools
  Build the OS baseline, container runtime and kubeadm/kubelet tooling.
  1. 0.3.1
    Disabling Swap, Kernel Modules, and sysctl
  2. 0.3.2
    Installing containerd
  3. 0.3.3
    Installing Kubernetes Tools and Pinning node-ip
  4. 0.3.4
    Node Baseline Check
4. 0.4
  Harbor and Image Trust
  Make all four nodes trust the Harbor registry and verify course image pulls.
  1. 0.4.1
    Checking Harbor Service and Images
  2. 0.4.2
    Distributing the Harbor CA to Four Nodes
  3. 0.4.3
    Configuring containerd and Verifying Pulls
  4. 0.4.4
    Course Image Inventory
  5. 0.4.5
    Installing Harbor from Scratch
5. 0.5
  Initializing the Control Plane
  Pre-checks, control-plane images, kubeadm init and verification.
  1. 0.5.1
    Pre-Initialization Checks
  2. 0.5.2
    Control-Plane Images
  3. 0.5.3
    Initializing and Verifying the Control Plane
  4. 0.5.4
    Common Fixes
6. 0.6
  Joining the Three Workers
  Meet the join prerequisites, then join and register each worker.
  1. 0.6.1
    Join Prerequisites
  2. 0.6.2
    Joining and Registering Each Worker
7. 0.7
  Calico, MetalLB, and Ingress
  Install pod networking, load balancing and ingress for external access.
  1. 0.7.1
    Installing Calico and Helm
  2. 0.7.2
    Installing MetalLB
  3. 0.7.3
    Installing ingress-nginx
  4. 0.7.4
    Demo App and Windows Client Access
8. 0.8
  Kubeflow and CPU Notebooks
  Install Kubeflow and validate the dashboard, notebooks and HTTPS entry.
  1. 0.8.1
    StorageClass, kustomize, and inotify Preparation
  2. 0.8.2
    Installing the Kubeflow Community Distribution
  3. 0.8.3
    Dashboard and Notebook Validation
  4. 0.8.4
    Production HTTPS Entry
  5. 0.8.5
    Common Issues
9. 0.9
  Platform Acceptance and Snapshot
  Accept the platform item by item and snapshot it as the starting point for later chapters.
  1. 0.9.1
    DNS, Management Network, and SSH
  2. 0.9.2
    Kubernetes, Harbor, and Ingress
  3. 0.9.3
    Basic RoCE Reachability
  4. 0.9.4
    Pre-Snapshot Checks for UFM and NVMe-oF Server
  5. 0.9.5
    Reboot Recovery and Stop Point
01
View handout
1. 1.1
  Why AI Training Needs High-Speed Networks
  Synchronous data parallelism explains compute dependencies, communication waits, and stragglers.
  1. 1.1.1
    Training Advances Step by Step
  2. 1.1.3
    One Slow GPU Makes the Others Wait
  3. 1.1.5
    Overhead of Traditional Paths and the Value of RDMA
  4. 1.1.6
    The Two Mainstream RDMA Approaches
2. 1.2
  How RDMA Changes the Data Path
  Socket vs. RDMA: kernel bypass, zero copy, and protocol offload.
  1. 1.2.1
    Sockets vs. RDMA
  2. 1.2.2
    Three Key RDMA Mechanisms
3. 1.3
  Messaging and Remote Memory Access
  Semantics and costs of Send/Receive, Write, and Read.
  1. 1.3.1
    Send/Receive: The Receiver Must Sign for It
  2. 1.3.2
    RDMA Write: Placing Data Directly in the Remote Locker
  3. 1.3.3
    RDMA Read: Fetching Data Directly from the Remote Locker
4. 1.4
  How an RDMA Write Completes
  Follow a full request to learn QP, MR, CQ, WQE/CQE, and connection setup.
  1. 1.4.1
    A Write and Read in Ten Steps
  2. 1.4.2
    Mapping the Terms to the Flow
  3. 1.4.3
    Connection Setup and Memory Exchange
5. 1.5
  RDMA Transports: InfiniBand and RoCE
  An intuitive comparison of the two transports, their architectures, and costs.
6. 1.6
  From Host Memory to GPU Memory
  Which path GPUDirect RDMA shortens and what it requires.
  1. 1.6.1
    The Path GPUDirect RDMA Shortens
02
View handout
1. 2.1
  What RoCE Is and Why It Exists
  Why RDMA moved onto Ethernet and where RoCE fits in the AI/GPU era.
  1. 2.1.1
    RoCE in the AI/GPU Era
2. 2.2
  How InfiniBand and RoCE Relate
  RoCE keeps IB transport semantics while replacing the underlying network.
3. 2.3
  From InfiniBand to RoCEv1 and RoCEv2
  How outer encapsulation determines RoCE forwarding scope.
  1. 2.3.1
    RoCEv1
  2. 2.3.2
    RoCEv2
  3. 2.3.3
    Why the Course Uses RoCEv2
4. 2.4
  Reading RoCEv2 Packets
  Who reads BTH, RETH, ICRC and what each field proves.
  1. 2.4.1
    Full Encapsulation of an RC Write
  2. 2.4.2
    Inspecting Frame 16 in Wireshark
5. 2.5
  Management vs. RoCE Data Traffic in Captures
  Merge captures from two ports to separate TCP management and RoCE data traffic.
  1. 2.5.1
    Capturing on Two Ports and Merging
  2. 2.5.2
    Separating the Two Traffic Types in Wireshark
03
View handout
1. 3.1
  Lab Platform and Software Preparation
  Part 1 begins: the overall lab topology and DOCA-OFED driver and tool preparation.
  1. 3.1.1
    Driver and Tool Preparation
2. 3.2
  Host Environment: Four Layers from NIC to GID
  Map the physical, network, RDMA, and GID layers to each other.
3. 3.3
  Physical Layer: PCIe Hardware and Link State
  Confirm the NIC is detected and the physical link has negotiated.
  1. 3.3.1
    Confirming PCIe Detection
  2. 3.3.2
    Confirming Physical Link State
4. 3.4
  Network Layer: MAC, IP, and Reachability
  Verify data-port addresses and basic reachability.
  1. 3.4.1
    Confirming Data-Port Addresses
  2. 3.4.2
    Basic Reachability Test
5. 3.5
  RDMA Layer: Devices and Port State
  Check RDMA link type, MTU, port state, and netdev binding.
  1. 3.5.1
    Link Type and MTU
  2. 3.5.2
    Port State and Netdev Binding
6. 3.6
  GID Layer: The Global Identifier Table
  Read the GID table, choose the GID index, and compare with Chapter 2 captures.
  1. 3.6.1
    Checking the Host GID Table
  2. 3.6.2
    Why This Lab Uses GID[3]
  3. 3.6.3
    Comparing with the Chapter 2 Capture
7. 3.7
  Switches and Link Components
  Part 2 begins: console access and reset, switch model, ASIC, OS, and breakout cabling.
  1. 3.7.1
    Console Access, Factory Reset, and First Login
  2. 3.7.2
    Confirming Hardware Model and ASIC
  3. 3.7.4
    Physical Cabling and Breakout
8. 3.8
  Cumulus Linux Initial Setup
  Basic switch networking and passwordless SSH for counter collection.
  1. 3.8.1
    Basic Network Configuration
  2. 3.8.2
    Passwordless SSH for Counter Collection
9. 3.9
  Enabling Lossless RoCE
  ECN and PFC principles and cases, queue design, and enabling RoCE lossless mode.
  1. 3.9.1
    ECN: Telling the Sender to Slow Down
  2. 3.9.4
    PFC: Hop-by-Hop Pause and Headroom
  3. 3.9.6
    Traffic Marking and Queue Design
  4. 3.9.7
    Enabling RoCE Lossless on the Switch
  5. 3.9.9
    Switching from Manual Configuration to DCBX
10. 3.10
  RoCE Connectivity and Baseline Performance
  Part 3 begins: a TCP baseline, then perftest for RoCE latency, bandwidth, and CPU cost.
  1. 3.10.1
    TCP Baseline: Two-Node qperf
  2. 3.10.3
    Send/Recv Latency and Bandwidth
  3. 3.10.5
    RDMA Write Latency and Bandwidth
  4. 3.10.7
    CPU Cost of TCP vs. RoCE
11. 3.11
  DCB Validation: PFC, ECN/CNP, and Lossy Drops
  Create controlled congestion and use counter deltas to validate PFC, ECN/CNP, and lossy queues.
  1. 3.11.3
    Creating Controlled Congestion
  2. 3.11.5
    Running RoCE and Plain UDP Together
  3. 3.11.6
    Computing Deltas and Checking Results
  4. 3.11.7
    Interpreting the Results
12. 3.12
  ECN Tuning: Keep Throughput, Cut Small-Message Latency
  Change one parameter per round to tune ECN marking, then return to baseline to confirm.
  1. 3.12.2
    One Parameter per Round
  2. 3.12.4
    Marking Earlier with ECN
  3. 3.12.5
    Stronger Feedback in a Smaller Backlog Window
  4. 3.12.7
    Back to Baseline to Confirm the Gain
13. 3.13
  ETS Tuning: More Bandwidth for Critical Traffic
  Start from 50:50, adjust ETS shares, verify, and derive a field tuning order.
  1. 3.13.2
    Measuring 50:50 First
  2. 3.13.3
    Adjusting Only ETS in Favor of TC3
  3. 3.13.4
    Restoring 50:50 and Rechecking
  4. 3.13.6
    A Field Tuning Order
14. 3.14
  Installing Network Operator and Verifying RoCE Resources
  Use Network Operator to hand networks and RDMA devices to pods, verified with three pods.
  1. 3.14.2
    How Network Operator Hands RDMA to Pods
  2. 3.14.3
    Installing the Components
  3. 3.14.5
    Requesting Networks and Devices in YAML
  4. 3.14.6
    Out-of-Band Exchange and GID Selection
  5. 3.14.7
    Checking Devices and Communication Results
04
1. 4.1
  GPU Training Traffic and Network Requirements
  Derive bandwidth, latency, and stability needs from DP/TP/PP communication scope.
2. 4.2
  Designing High-Performance RoCE Networks
  Design trade-offs in Clos topology, capacity, redundancy, rail optimization, and load balancing.
  1. 4.2.1
    Two-Tier Clos and Predictable Paths
  2. 4.2.3
    Redundancy and Resilience
  3. 4.2.4
    Port Capacity for 1:1 Non-Blocking Design
  4. 4.2.5
    Rail Optimization: Aligning Cabling with Communication
  5. 4.2.6
    Load Balancing and the Spectrum-X Platform
3. 4.4
  Lab Overview
  Lab topology, device roles, and the order of manual configuration.
4. 4.5
  Leaf Local Baseline
  Baseline configuration on each of the three leaves.
  1. 4.5.1
    Leaf01
  2. 4.5.2
    Leaf02
  3. 4.5.3
    Leaf03
5. 4.6
  Spine Underlay Foundation
  Complete spine interfaces and the underlay foundation.
6. 4.7
  Configuring MLAG and VRR
  MLAG and VRR shared gateway on Leaf01/Leaf02.
  1. 4.7.1
    Configuring Leaf01
  2. 4.7.2
    Configuring Leaf02
7. 4.8
  Configuring Server Access Ports
  Push server-side configuration and check interface state and bonding.
  1. 4.8.1
    Server-Side Approach
  2. 4.8.2
    Pushing Server Configuration
  3. 4.8.4
    Inspecting Bonding Details
8. 4.9
  Configuring Underlay BGP
  Underlay BGP on both leaves and spines.
  1. 4.9.1
    Leaf Side
  2. 4.9.2
    Spine Side
9. 4.10
  Overlaying EVPN and VXLAN
  Add EVPN/VXLAN on top of the underlay and verify.
  1. 4.10.1
    Leaf Side
  2. 4.10.2
    Spine Side
  3. 4.10.3
    Verification
10. 4.12
  Ansible/NVUE Automation
  Follow deploy.yaml, variable files, and templates to see automated configuration and verification.
  1. 4.12.1
    From hosts to Leaf01: deploy.yaml
  2. 4.12.2
    Leaf01 Data: leaf01.yml
  3. 4.12.3
    How the Leaf Template Calls Modules: leaf_main.j2
  4. 4.12.5
    Verifying with show and ping
11. 4.13
  NetQ in Practice
  Install NetQ, discover switches, view topology, BGP, and EVPN, and run fabric-wide validations.
  1. 4.13.1
    Components and Data Flow
  2. 4.13.3
    Installing the Platform
  3. 4.13.7
    Discovering Four Switches and Installing NetQ
  4. 4.13.11
    Fabric-Wide BGP Sessions
  5. 4.13.13
    Creating Validations and Reading Results
05
1. 5.1
  SONiC Overview
  SONiC architecture and positioning, with companion topics on ONIE and image builds.
2. 5.2
  SONiC LAB2 CLI Configuration
  Configure Leaf, Spine, MC-LAG, BGP, and EVPN/VXLAN via CLI, checking each step.
  1. 5.2.2
    Configuring and Checking the Leaves
  2. 5.2.3
    Configuring and Checking the Spine
  3. 5.2.4
    MC-LAG and Shared Gateway
  4. 5.2.6
    Underlay BGP and Routes
  5. 5.2.7
    EVPN, VXLAN, and VTEPs
3. 5.3
  Ansible Automation
  Reuse the Chapter 04 approach to configure SONiC LAB2 automatically and verify.
  1. 5.3.1
    From hosts to Leaf01: deploy.yaml
  2. 5.3.2
    Leaf01 Data: Leaf01.yml
  3. 5.3.3
    How the Two Templates Call Modules
  4. 5.3.5
    Verifying with show and ping
06
1. 6.1
  What InfiniBand Is and Why It Exists
  Origins of IB and its relevance in the AI era.
  1. 6.1.1
    InfiniBand in the AI Era
2. 6.2
  Components and Terminology
  Treat the subnet as a neighborhood to learn HCAs, switches, the SM, and more.
  1. 6.2.1
    Analogy Reference Table
3. 6.3
  Physical Links: Lanes, Gates, and Speed Limits
  Link width, connector form factors, and speed generations.
  1. 6.3.1
    Link Width: 1X, 2X, 4X
  2. 6.3.2
    Form Factors: QSFP and OSFP
  3. 6.3.3
    Speed Generations: EDR, HDR, NDR, XDR
4. 6.4
  Addresses and Packets: House Numbers and Letters
  GUID, LID, GID, and PKey, plus a real IB packet dissected in Wireshark.
  1. 6.4.1
    GUID and LID
  2. 6.4.2
    GID: The Full Address with Subnet Prefix
  3. 6.4.3
    Why Forwarding Uses LIDs
  4. 6.4.4
    PKey and Partitions
  5. 6.4.5
    A Real IB Packet in Wireshark
5. 6.5
  Subnet Bring-Up: From Power-On to Traffic
  The SM discovers topology, assigns LIDs, programs forwarding tables, and ports go Active.
  1. 6.5.3
    SM Topology Discovery
  2. 6.5.4
    LID Assignment
  3. 6.5.5
    Computing and Programming LFTs
  4. 6.5.6
    Ports Become Active
6. 6.6
  Adding a New Node
  What happens when a node joins a running subnet.
7. 6.7
  SM Operations and Lossless Delivery
  Impact of SM failure, credit-based flow control, SLs, and VLs.
  1. 6.7.1
    What Happens When the SM Fails
  2. 6.7.2
    Credit Flow Control, SL, and VL
07
1. 7.1
  Lab Platform Overview
  IB lab topology, switch and host roles.
2. 7.2
  Switch Basics
  Initialize the switch and check version, hardware, SM, and IB ports.
  1. 7.2.1
    Wiping and Initializing via Console
  2. 7.2.2
    Version and Hardware Inventory
  3. 7.2.4
    Confirming the SM and IB Ports
3. 7.3
  Host Basics
  Prepare hardware and software; check worker IB ports and HCA identity, rate, and MTU.
  1. 7.3.1
    Hardware and Software Preparation
  2. 7.3.2
    Checking IB Ports on Three Workers
  3. 7.3.3
    HCA Identity, Rate, and MTU
4. 7.4
  Subnet Discovery: Registry and Neighbors
  Reconstruct the fabric topology with discovery tools.
  1. 7.4.1
    Command Summary: Where the Data Comes From
5. 7.5
  Addresses, the SM, and Forwarding Tables
  Map hosts to GUID/LID, identify the master SM, and read GIDs and LFTs.
  1. 7.5.1
    Mapping Hosts to GUID and LID
  2. 7.5.2
    The Current Master SM
  3. 7.5.3
    Reading GIDs and LFTs
6. 7.6
  Connectivity and Paths: 1 Hop vs. 3 Hops
  Verify same-leaf and cross-leaf paths with ibping and path tracing.
  1. 7.6.1
    IB Management Probe: ibping
  2. 7.6.2
    Path Tracing: 1 Hop Same-Leaf, 3 Hops Cross-Leaf
7. 7.7
  Baseline Performance: TCP vs. IB
  Latency, bandwidth, and CPU use measured the same way as Chapter 3.
  1. 7.7.1
    TCP Baseline: Two-Node qperf
  2. 7.7.3
    Send/Recv Latency and Bandwidth
  3. 7.7.5
    RDMA Write Latency and Bandwidth
  4. 7.7.7
    CPU Utilization Comparison
08
1. 8.1
  UFM Positioning and Role
  The role of UFM in IB fabric management.
2. 8.2
  UFM Docker Standalone Deployment
  Check prerequisites, run the official installer, retire the old SM, and start UFM.
  1. 8.2.1
    Architecture and Prerequisites
  2. 8.2.2
    Host RDMA Check
  3. 8.2.4
    Running the Official Installer
  4. 8.2.5
    Retiring the Old SM and Starting UFM
3. 8.3
  Startup Verification and SM Takeover
  Verify container services, licensing, and the actual master; enable web access.
  1. 8.3.1
    Containers and Services
  2. 8.3.2
    Licensing and the Actual Master
  3. 8.3.3
    Web Access and Persistent Settings
4. 8.4
  UFM GUI, PKey, and Unified Operations
  Tour the GUI pages and run a Full/Limited PKey partition lab.
  1. 8.4.1
    Dashboard and Events & Alarms
  2. 8.4.3
    Network Map, Devices, and Ports
  3. 8.4.9
    Telemetry Analysis
  4. 8.4.11
    PKey: Logical Isolation in IB
  5. 8.4.14
    Full vs. Limited Membership
5. 8.5
  SHARP Principles and UFM Integration
  In-network aggregation principles, device requirements, and current lab status.
  1. 8.5.1
    What SHARP Is
  2. 8.5.4
    The SHARP Aggregation Tree
  3. 8.5.5
    Hardware Requirements for SHARP
  4. 8.5.7
    SHARP Status in This Lab
6. 8.6
  OpenSM: A Standalone Subnet Manager
  OpenSM origins, features, and its master/standby relationship with the spine.
  1. 8.6.1
    Origins and Main Features
  2. 8.6.2
    Master/Standby with the Spine
7. 8.7
  Replacing UFM with OpenSM via Docker Compose
  Deploy OpenSM to take over and restore the spine as standby SM.
  1. 8.7.1
    Source, Images, and Lab Directory
  2. 8.7.2
    Configuring OpenSM and Taking Over
  3. 8.7.3
    Restoring the Spine Standby SM
8. 8.8
  OpenSM Topology, Routing, and Partitions
  Read back topology and routing and run a CLI partition lab.
  1. 8.8.1
    Version, Topology, and Routing Readback
  2. 8.8.2
    Two Limited, One Full
  3. 8.8.3
    Removing the Business Partition
9. 8.9
  IB Monitor: Building Your Own Monitor
  Collect the whole fabric from one IB card and visualize with Prometheus and Grafana.
  1. 8.9.1
    Collecting the Whole Fabric from One Card
  2. 8.9.3
    Browsing Grafana Dashboards
  3. 8.9.4
    Checking Prometheus Data
10. 8.10
  Observing Master Changes
  Stop and restore OpenSM to watch master SM failover.
  1. 8.10.1
    Stopping OpenSM and Watching the Spine Take Over
  2. 8.10.2
    Restoring OpenSM
09
1. 9.1
  Meet a GPU Server
  Where GPUs, NICs, PCIe, and NVLink sit inside one server.
2. 9.2
  Splitting the Data: DP
  Data parallelism: model replicas, gradient sync, and waiting.
3. 9.3
  Splitting a Step: TP
  Tensor parallelism: matrix splitting and intra-layer collectives.
4. 9.4
  Pipelining the Model: PP
  Pipeline parallelism: stages, activation passing, and bubbles.
5. 9.5
  Placing the Work on Machines
  Combine DP/TP/PP and see how placement changes cross-node traffic.
  1. 9.5.1
    Adding DP: Placing Two Full Replicas
  2. 9.5.2
    Which Traffic Crosses Nodes After Placement
6. 9.6
  Communication Operations as Needed
  The roles of AllReduce and Send/Recv in training.
  1. 9.6.1
    AllReduce: Everyone Gets the Full Sum
  2. 9.6.2
    Send/Recv: Activations Forward, Gradients Backward
7. 9.7
  How NCCL Communicates
  Division of labor among framework, NCCL, and network, and basic Ring/Tree trade-offs.
8. 9.8
  Reading Real Topologies
  Read topology output, compare ideal vs. constrained GPU/NIC layouts, and verify GPUDirect RDMA.
  1. 9.8.1
    Reading the Output in Three Layers
  2. 9.8.2
    Root Complexes vs. PCIe Switches
  3. 9.8.3
    Ideal Case: Eight GPUs Aligned with Eight NICs
  4. 9.8.4
    Constrained Case: Shared Uplinks
  5. 9.8.5
    GPUDirect RDMA: Can the NIC Reach GPU Memory
9. 9.9
  What Changes for Inference
  How KV cache reuse and PD disaggregation affect the network.
  1. 9.9.1
    Cache Reuse: Compute Once for Shared Context
  2. 9.9.2
    PD Disaggregation: Prefill Reads, Decode Answers
10
1. 10.1
  GPU and Monitoring Readiness
  Deliver GPUs with GPU Operator, wire up DCGM monitoring, and read key metrics.
  1. 10.1.1
    GPU Operator: Handing GPUs to Kubernetes
  2. 10.1.2
    One Script for the GPU Stack
  3. 10.1.3
    DCGM Metrics into Prometheus and Grafana
  4. 10.1.4
    Reading Five Key Metrics
2. 10.2
  Launching a Three-Node IB Setup via DNS
  Three compute pods and a launcher discover each other via dynamic addressing and DNS.
  1. 10.2.1
    Three Servers, Three Pods, One Launcher
  2. 10.2.2
    Dynamic Addressing and DNS
  3. 10.2.3
    Checking Members and IB Devices
3. 10.3
  Running the First AllReduce
  Validate the NCCL path with small messages and rerun by name after rebuilding pods.
  1. 10.3.1
    What We Measure Without a Model
  2. 10.3.2
    Small-Message Validation with Defaults
  3. 10.3.3
    Rerunning by Name After Rebuilding Pods
4. 10.4
  How NCCL Chooses a Plan
  From topology and GPU-to-NIC paths to the plan NCCL actually picks.
  1. 10.4.1
    Topology and GPU-to-NIC Paths
  2. 10.4.2
    Planning Channels and Estimating Time
  3. 10.4.3
    Reading What This AllReduce Chose
5. 10.5
  Auto vs. Ring/Tree
  Run Auto, Ring, and Tree and plot by message size to explain the differences.
  1. 10.5.1
    Auto First, Then Ring and Tree
  2. 10.5.2
    Plotting by Size to Find Differences
  3. 10.5.3
    Explaining Differences by Protocol
6. 10.6
  Network Capability and QP Parameters
  Compare line rate, two-node RDMA, and three-node NCCL across QP counts.
  1. 10.6.1
    Line Rate, Two-Node RDMA, and Three-Node NCCL
  2. 10.6.2
    Comparing QP 1, 2, 4, 8
  3. 10.6.3
    Keeping What Works, Restoring Baseline
7. 10.7
  Three Typical Failures, End to End
  Work through three real failures from symptom to evidence to fix.
  1. 10.7.1
    Launcher Starts Before Workers Are Ready
  2. 10.7.2
    Training Succeeds but Data Uses Sockets
  3. 10.7.3
    Link Up but IB Port Not Ready
11
1. 11.1
  DPU and SmartNIC Basics
  Where a DPU runs and its modes; preparing the target and SNAP backend.
  1. 11.1.1
    Preparing the Target and SNAP Backend
  2. 11.1.2
    Finding BlueField from the Host
2. 11.2
  Host, DPU, and Switch Data Paths
  How data flows among the host, DPU Arm cores, and switch.
3. 11.3
  Basic Configuration and Network Verification
  Query interfaces, namespaces, Arm, and the remote backend.
  1. 11.3.1
    Interfaces and Namespaces
  2. 11.3.2
    Arm and Remote Backend
4. 11.4
  NVMe-oF and DPU Offload Demo
  Use before/after counters to show storage I/O offloaded to the DPU.
  1. 11.4.1
    How Storage Offload Happens
  2. 11.4.2
    Reading Counters First
  3. 11.4.3
    Reading the Dedicated Namespace from the Host
  4. 11.4.4
    Reconciling After the Run
5. 11.5
  Network Hardware Offload on One DPU
  Check OVS offload settings, send traffic, and confirm hardware-forwarded flows.
  1. 11.5.2
    OVS Topology and Offload Settings
  2. 11.5.3
    Enabling the Host-Side Lab Interface
  3. 11.5.4
    Sending Traffic and Querying Hardware Flows
12
1. 12.1
  Preparing Shared Storage and the Training Environment
  Shared model disk and NFS, training images, model readability, and three-node IB checks.
  1. 12.1.1
    Shared Model Disk and NFS
  2. 12.1.2
    Checking the Training Environment and Images
  3. 12.1.4
    Confirming Model and Sample Access
  4. 12.1.5
    Validating Three-Node IB First
2. 12.2
  Kubeflow Distributed Training Demo
  Use Kubeflow to run small-scale QLoRA training and merge a deployable model.
  1. 12.2.1
    Kubeflow Orchestrates Training
  2. 12.2.2
    Small-Scale QLoRA Training
  3. 12.2.3
    Merging Adapters into a Full Model
  4. 12.2.4
    Comparing with Archived Full Training
3. 12.3
  vLLM Inference Deployment and Calls
  Two single-GPU replicas, an HTTPS API entry, and chat/streaming calls.
  1. 12.3.1
    Two Independent Single-GPU Replicas
  2. 12.3.2
    A Trusted HTTPS Entry for the API
  3. 12.3.3
    Model Listing, Chat, and Streaming
  4. 12.3.4
    Checking Against Archived Q&A Results
4. 12.4
  The Value of the AI Network in Hindsight
  Compare training, storage, and serving paths; release jobs and keep the model service.
  1. 12.4.1
    Comparing Training, Storage, and Serving Paths
  2. 12.4.2
    Releasing Jobs, Keeping the Service

COURSE RECORDINGS

Course Video

Watch the public preview here or open it on the original video platform.

BILIBILI01

PUBLIC PREVIEW

Jiaozhu VIP NVIDIA Supercomputing Networking AIN2027 Preview: Course Notes Now in HTML with GIF Animations

Watch on the Original Platform

NVIDIA AI Supercomputing Networking Comprehensive Practice (AIN2027)

Part 0 K8S Environment: Kubernetes and Kubeflow Platform Foundation

Networking, Hostnames, and SSH

Host and VIP Planning

RoCE Physical Cabling

Hostnames, /etc/hosts, and Passwordless SSH

Network Checks and Port Planning

Master DNS

DNS Record Planning

Installing dnsmasq on the Master

Pointing Nodes to Master DNS and Verifying

OS, containerd, and Kubernetes Tools

Disabling Swap, Kernel Modules, and sysctl

Installing containerd

Installing Kubernetes Tools and Pinning node-ip

Node Baseline Check

Harbor and Image Trust

Checking Harbor Service and Images

Distributing the Harbor CA to Four Nodes

Configuring containerd and Verifying Pulls

Course Image Inventory

Installing Harbor from Scratch

Initializing the Control Plane

Pre-Initialization Checks

Control-Plane Images

Initializing and Verifying the Control Plane

Common Fixes

Joining the Three Workers

Join Prerequisites

Joining and Registering Each Worker

Calico, MetalLB, and Ingress

Installing Calico and Helm

Installing MetalLB

Installing ingress-nginx

Demo App and Windows Client Access

Kubeflow and CPU Notebooks

StorageClass, kustomize, and inotify Preparation

Installing the Kubeflow Community Distribution

Dashboard and Notebook Validation

Production HTTPS Entry

Common Issues

Platform Acceptance and Snapshot

DNS, Management Network, and SSH

Kubernetes, Harbor, and Ingress

Basic RoCE Reachability

Pre-Snapshot Checks for UFM and NVMe-oF Server

Reboot Recovery and Stop Point

Chapter 01 RDMA Fundamentals

Why AI Training Needs High-Speed Networks

Training Advances Step by Step

One Slow GPU Makes the Others Wait

Overhead of Traditional Paths and the Value of RDMA

The Two Mainstream RDMA Approaches

How RDMA Changes the Data Path

Sockets vs. RDMA

Three Key RDMA Mechanisms

Messaging and Remote Memory Access

Send/Receive: The Receiver Must Sign for It

RDMA Write: Placing Data Directly in the Remote Locker

RDMA Read: Fetching Data Directly from the Remote Locker

How an RDMA Write Completes

A Write and Read in Ten Steps

Mapping the Terms to the Flow

Connection Setup and Memory Exchange

RDMA Transports: InfiniBand and RoCE

From Host Memory to GPU Memory

The Path GPUDirect RDMA Shortens

Chapter 02 Introduction to the RoCE Protocol

What RoCE Is and Why It Exists

RoCE in the AI/GPU Era

How InfiniBand and RoCE Relate

From InfiniBand to RoCEv1 and RoCEv2

RoCEv1

RoCEv2

Why the Course Uses RoCEv2

Reading RoCEv2 Packets

Full Encapsulation of an RC Write

Inspecting Frame 16 in Wireshark

Management vs. RoCE Data Traffic in Captures

Capturing on Two Ports and Merging