📦 Proxmox VE LXC Container GPU Passthrough & Hardware Permissions

Running multi-GPU AI inference and hardware power management inside a Proxmox LXC container requires specific device mappings, cgroup permissions, and capability flags.

1. Proxmox Host Configuration (`/etc/pve/lxc/<CTID>.conf`)

Add the following lines to your container configuration file on the Proxmox host:

# /etc/pve/lxc/<CTID>.conf

# 1. Unconfined AppArmor profile (Required for NVML clock/power limit modification)
lxc.apparmor.profile: unconfined

# 2. Grant SYS_ADMIN capability for hardware clock management
lxc.cap.keep: sys_admin sys_rawio

# 3. Allow all NVIDIA device cgroups
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 235:* rwm
lxc.cgroup2.devices.allow: c 510:* rwm
lxc.cgroup2.devices.allow: c 511:* rwm

# 4. Pass-through NVIDIA character device nodes
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file

# 5. CPU Socket & NUMA Pinning (Crucial for Multi-GPU AllReduce Performance)
# If your GPUs are attached to PCIe Riser 2 / CPU Socket 2 (NUMA Node 1), pin the container:
cpuset: 6-11,18-23

2. Verifying Permissions Inside the Container

Restart the container, then run:

# Verify all GPUs are visible
nvidia-smi

# Test NVML Persistence Mode (Requires CAP_SYS_ADMIN)
nvidia-smi -pm 1

# Test Power Capping
nvidia-smi -i 1,2 -pl 150

# Test Dynamic Clock Locking
nvidia-smi -i 1,2 -lgc 600,600
nvidia-smi -i 1,2 -rgc

If all four commands return with code 0 and "All done", your container has full hardware rights.

3. Important Systemd Service Security Flags

When running services that execute nvidia-smi hardware commands inside systemd:

Do NOT set NoNewPrivileges=true in gpu-power-governor.service. NoNewPrivileges=true blocks processes from acquiring permissions to execute privileged NVML clock-locking calls.

4. NUMA Node & CPU Socket Alignment (Performance Critical)

On dual-socket servers (such as HPE ProLiant DL380p Gen8 with Dual Intel Xeon E5-2620 v2):

Socket 1 (NUMA Node 0): CPUs 0-5, 12-17 (Wires to Riser 1 PCIe slots)
Socket 2 (NUMA Node 1): CPUs 6-11, 18-23 (Wires to Riser 2 PCIe slots)

If your GPUs are seated in PCIe Riser 2 (Bus 0x21, 0x24, 0x27), failing to pin CPU affinity causes worker threads and CUDA driver calls to constantly cross the QPI bus interconnect.

Verified Benchmark Impact (Socket 2 Pinned vs. Unpinned)

Workload	Unpinned (All Cores 0–23)	Pinned to Socket 2 (`cpuset: 6-11,18-23`)	Improvement
Short Prompt Prefill Latency	2,798 ms (24.7 tok/s)	1,989 ms (34.7 tok/s)	+40.7% faster prefill ⚡
Medium Prompt (~1k tokens)	281.87 tok/s	281.50 tok/s	Consistent
Long Prompt (~4.8k tokens)	384.01 tok/s	384.31 tok/s	Consistent
Sustained Decode	35–38 tok/s	35–38 tok/s	Consistent

Recommendation: Always pair your GPU physical PCIe slot placement with container cpuset pinning on the host to eliminate cross-QPI memory bounce.

3.5 KiB Raw Permalink Blame History Unescape Escape