90 lines
3.5 KiB
Markdown
90 lines
3.5 KiB
Markdown
# 📦 Proxmox VE LXC Container GPU Passthrough & Hardware Permissions
|
||
|
||
Running multi-GPU AI inference and hardware power management inside a Proxmox LXC container requires specific device mappings, cgroup permissions, and capability flags.
|
||
|
||
---
|
||
|
||
## 1. Proxmox Host Configuration (`/etc/pve/lxc/<CTID>.conf`)
|
||
|
||
Add the following lines to your container configuration file on the Proxmox host:
|
||
|
||
```ini
|
||
# /etc/pve/lxc/<CTID>.conf
|
||
|
||
# 1. Unconfined AppArmor profile (Required for NVML clock/power limit modification)
|
||
lxc.apparmor.profile: unconfined
|
||
|
||
# 2. Grant SYS_ADMIN capability for hardware clock management
|
||
lxc.cap.keep: sys_admin sys_rawio
|
||
|
||
# 3. Allow all NVIDIA device cgroups
|
||
lxc.cgroup2.devices.allow: c 195:* rwm
|
||
lxc.cgroup2.devices.allow: c 235:* rwm
|
||
lxc.cgroup2.devices.allow: c 510:* rwm
|
||
lxc.cgroup2.devices.allow: c 511:* rwm
|
||
|
||
# 4. Pass-through NVIDIA character device nodes
|
||
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
|
||
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
|
||
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
|
||
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
|
||
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
|
||
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
|
||
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
|
||
|
||
# 5. CPU Socket & NUMA Pinning (Crucial for Multi-GPU AllReduce Performance)
|
||
# If your GPUs are attached to PCIe Riser 2 / CPU Socket 2 (NUMA Node 1), pin the container:
|
||
cpuset: 6-11,18-23
|
||
```
|
||
|
||
---
|
||
|
||
## 2. Verifying Permissions Inside the Container
|
||
|
||
Restart the container, then run:
|
||
|
||
```bash
|
||
# Verify all GPUs are visible
|
||
nvidia-smi
|
||
|
||
# Test NVML Persistence Mode (Requires CAP_SYS_ADMIN)
|
||
nvidia-smi -pm 1
|
||
|
||
# Test Power Capping
|
||
nvidia-smi -i 1,2 -pl 150
|
||
|
||
# Test Dynamic Clock Locking
|
||
nvidia-smi -i 1,2 -lgc 600,600
|
||
nvidia-smi -i 1,2 -rgc
|
||
```
|
||
|
||
If all four commands return with code `0` and "All done", your container has full hardware rights.
|
||
|
||
---
|
||
|
||
## 3. Important Systemd Service Security Flags
|
||
|
||
When running services that execute `nvidia-smi` hardware commands inside systemd:
|
||
* Do **NOT** set `NoNewPrivileges=true` in `gpu-power-governor.service`. `NoNewPrivileges=true` blocks processes from acquiring permissions to execute privileged NVML clock-locking calls.
|
||
|
||
---
|
||
|
||
## 4. NUMA Node & CPU Socket Alignment (Performance Critical)
|
||
|
||
On dual-socket servers (such as HPE ProLiant DL380p Gen8 with Dual Intel Xeon E5-2620 v2):
|
||
* **Socket 1 (NUMA Node 0)**: CPUs `0-5, 12-17` (Wires to Riser 1 PCIe slots)
|
||
* **Socket 2 (NUMA Node 1)**: CPUs `6-11, 18-23` (Wires to Riser 2 PCIe slots)
|
||
|
||
If your GPUs are seated in PCIe Riser 2 (Bus `0x21`, `0x24`, `0x27`), failing to pin CPU affinity causes worker threads and CUDA driver calls to constantly cross the QPI bus interconnect.
|
||
|
||
### Verified Benchmark Impact (Socket 2 Pinned vs. Unpinned)
|
||
|
||
| Workload | Unpinned (All Cores 0–23) | Pinned to Socket 2 (`cpuset: 6-11,18-23`) | Improvement |
|
||
| :--- | :--- | :--- | :--- |
|
||
| **Short Prompt Prefill Latency** | 2,798 ms (24.7 tok/s) | **1,989 ms (34.7 tok/s)** | **+40.7% faster prefill** ⚡ |
|
||
| **Medium Prompt (~1k tokens)** | 281.87 tok/s | **281.50 tok/s** | Consistent |
|
||
| **Long Prompt (~4.8k tokens)** | 384.01 tok/s | **384.31 tok/s** | Consistent |
|
||
| **Sustained Decode** | 35–38 tok/s | **35–38 tok/s** | Consistent |
|
||
|
||
**Recommendation**: Always pair your GPU physical PCIe slot placement with container `cpuset` pinning on the host to eliminate cross-QPI memory bounce.
|