wmantly/turing-multi-gpu-llm-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Hardware: Add MSI CMP 50HX VBIOS ROM and document the 100% Video Engine Pinning bug and cross-flash idle power drop

Browse Source
This commit is contained in:
wmantly
2026-09-01 19:24:00 +00:00
parent 5f261c191f
commit 2a21eae435
3 changed files with 18 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/HARDWARE_LEARNINGS.md
+16 -13
View File
@@ -6,21 +6,24 @@ This document details the practical hardware behaviors, quirks, and engineering
## 1. NVIDIA CMP 50HX Mining GPUs: VBIOS & Power States
### Why CMP 50HX Mining Cards Idle at ~75W–85W Stock
* **No Display / Headless Architecture**: CMP 50HX mining cards lack display outputs and display engines.
* **Missing Deep P8 VBIOS Tables**: Standard GeForce VBIOSes drop to P8 state (405 MHz GDDR6 / ~10W) when no display is active. Mining VBIOSes lock the cards in P0 state (1,890 MHz core / 7,000 MHz GDDR6) by default.
### Why CMP 50HX Mining Cards Idle at ~75W–85W Stock (The 100% Video Engine Pinning Bug)
* **No Display / Headless Architecture**: CMP 50HX mining cards lack physical display outputs and raster display engines.
* **The 100% Video Engine Pinning Bug**: The `.17` (and newer `.1A`) CMP mining VBIOSes have a firmware bug where the GPU's hardware **Video Engine (NVDEC)** registers as 100% utilized at all times. This prevents the GPU power controller from automatically dropping into its lowest P8 sleep state, trapping stock idle power at roughly **75W–85W per card**.
* **The Fix**:
* **On Windows**: Use *NVIDIA Inspector's Multi-Display Power Saver*. Add AI backends (e.g., `ollama_server.exe` or `python.exe`) to the high-performance threshold list to force the cards into true ~5W–8W idle states when inactive.
* **On Linux**: Use our custom dynamic power governor ([`power-governor.py`](../scripts/power-governor.py)) with `nvidia-smi` script wrappers to manually clamp idle clocks to **300 MHz** (`-lgc 300,300`) and apply a **150W power limit** (`-pl 150`).
### OEM Reference VBIOS vs. MSI VBIOS
Comparing the hardware profiles of two CMP 50HX cards on the exact same rig:
* **GPU 2 (MSI Board `0x1462`, VBIOS `90.02.60.00.17`)**:
* Idles at **`~31.8W`** when core is locked to 300–600 MHz.
* Minimum fan speed: **`25%`**.
### OEM Reference VBIOS vs. MSI VBIOS (`90.02.60.00.17`)
Comparing the hardware profiles and the cross-flashing outcome:
* **GPU 2 (Original MSI Board `0x1462`, VBIOS `90.02.60.00.17`)**:
* Idles at **`~31.8W – 33.7W`** when core is locked to 300 MHz.
* Minimum fan speed floor: **`25%`**.
* Aggressive P3 voltage gating tables.
* **GPU 1 (NVIDIA Reference OEM `0x10DE`, VBIOS `90.02.60.00.01`)**:
* Idles at **`~61.2W`** when core is locked to 300–600 MHz.
* Minimum fan speed: **`40%`** (locked in VBIOS).
* Higher static VRM voltage rail leakage.
* **Actionable Solution**: Cross-flash GPU 2's MSI VBIOS (`90.02.60.00.17`) onto GPU 1 using `nvflash -6` to cut GPU 1's idle draw by ~30W and match fan curves.
* **GPU 1 (Original NVIDIA Reference OEM `0x10DE`, VBIOS `90.02.60.00.01`)**:
* Idled at **`~61.2W – 61.8W`** stock (locked 40% fan curve floor, higher static VRM leakage).
* **Cross-Flashing Result**:
* Cross-flashed GPU 1 with the MSI VBIOS ROM (`vbios/msi_cmp50hx_90.02.60.00.17.rom`) using `nvflash -6`.
* **Verified Outcome**: GPU 1 idle power immediately dropped from **`61.8W` down to `31.89W`** (a **~30W direct reduction!**), and fan curve synchronized to 25% minimum floor across both cards.
---