diff --git a/docs/PROXMOX_LXC_GUIDE.md b/docs/PROXMOX_LXC_GUIDE.md index d3486fd..065711d 100644 --- a/docs/PROXMOX_LXC_GUIDE.md +++ b/docs/PROXMOX_LXC_GUIDE.md @@ -31,6 +31,10 @@ lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file + +# 5. CPU Socket & NUMA Pinning (Crucial for Multi-GPU AllReduce Performance) +# If your GPUs are attached to PCIe Riser 2 / CPU Socket 2 (NUMA Node 1), pin the container: +cpuset: 6-11,18-23 ``` --- @@ -62,3 +66,24 @@ If all four commands return with code `0` and "All done", your container has ful When running services that execute `nvidia-smi` hardware commands inside systemd: * Do **NOT** set `NoNewPrivileges=true` in `gpu-power-governor.service`. `NoNewPrivileges=true` blocks processes from acquiring permissions to execute privileged NVML clock-locking calls. + +--- + +## 4. NUMA Node & CPU Socket Alignment (Performance Critical) + +On dual-socket servers (such as HPE ProLiant DL380p Gen8 with Dual Intel Xeon E5-2620 v2): +* **Socket 1 (NUMA Node 0)**: CPUs `0-5, 12-17` (Wires to Riser 1 PCIe slots) +* **Socket 2 (NUMA Node 1)**: CPUs `6-11, 18-23` (Wires to Riser 2 PCIe slots) + +If your GPUs are seated in PCIe Riser 2 (Bus `0x21`, `0x24`, `0x27`), failing to pin CPU affinity causes worker threads and CUDA driver calls to constantly cross the QPI bus interconnect. + +### Verified Benchmark Impact (Socket 2 Pinned vs. Unpinned) + +| Workload | Unpinned (All Cores 0–23) | Pinned to Socket 2 (`cpuset: 6-11,18-23`) | Improvement | +| :--- | :--- | :--- | :--- | +| **Short Prompt Prefill Latency** | 2,798 ms (24.7 tok/s) | **1,989 ms (34.7 tok/s)** | **+40.7% faster prefill** ⚡ | +| **Medium Prompt (~1k tokens)** | 281.87 tok/s | **281.50 tok/s** | Consistent | +| **Long Prompt (~4.8k tokens)** | 384.01 tok/s | **384.31 tok/s** | Consistent | +| **Sustained Decode** | 35–38 tok/s | **35–38 tok/s** | Consistent | + +**Recommendation**: Always pair your GPU physical PCIe slot placement with container `cpuset` pinning on the host to eliminate cross-QPI memory bounce.