# 🔬 Hardware Architecture & Key Technical Learnings

This document details the practical hardware behaviors, quirks, and engineering solutions discovered while building and optimizing this 3-GPU Turing inference rig on an **HPE ProLiant DL380p Gen8** server.

---

## 1. NVIDIA CMP 50HX Mining GPUs: VBIOS & Power States

### Why CMP 50HX Mining Cards Idle at ~75W–85W Stock (The 100% Video Engine Pinning Bug)
* **No Display / Headless Architecture**: CMP 50HX mining cards lack physical display outputs and raster display engines.
* **The 100% Video Engine Pinning Bug**: The `.17` (and newer `.1A`) CMP mining VBIOSes have a firmware bug where the GPU's hardware **Video Engine (NVDEC)** registers as 100% utilized at all times. This prevents the GPU power controller from automatically dropping into its lowest P8 sleep state, trapping stock idle power at roughly **75W–85W per card**.
* **The Fix**:
  * **On Windows**: Use *NVIDIA Inspector's Multi-Display Power Saver*. Add AI backends (e.g., `ollama_server.exe` or `python.exe`) to the high-performance threshold list to force the cards into true ~5W–8W idle states when inactive.
  * **On Linux**: Use our custom dynamic power governor ([`power-governor.py`](../scripts/power-governor.py)) with `nvidia-smi` script wrappers to manually clamp idle clocks to **300 MHz** (`-lgc 300,300`) and apply a **150W power limit** (`-pl 150`).

### OEM Reference VBIOS vs. MSI VBIOS (`90.02.60.00.17`)
Comparing the hardware profiles and the cross-flashing outcome:
* **GPU 2 (Original MSI Board `0x1462`, VBIOS `90.02.60.00.17`)**:
  * Idles at **`~31.8W – 33.7W`** when core is locked to 300 MHz.
  * Minimum fan speed floor: **`25%`**.
  * Aggressive P3 voltage gating tables.
* **GPU 1 (Original NVIDIA Reference OEM `0x10DE`, VBIOS `90.02.60.00.01`)**:
  * Idled at **`~61.2W – 61.8W`** stock (locked 40% fan curve floor, higher static VRM leakage).
* **Cross-Flashing Result**:
  * Cross-flashed GPU 1 with the MSI VBIOS ROM (`vbios/msi_cmp50hx_90.02.60.00.17.rom`) using `nvflash -6`.
  * **Verified Outcome**: GPU 1 idle power immediately dropped from **`61.8W` down to `31.89W`** (a **~30W direct reduction!**), and fan curve synchronized to 25% minimum floor across both cards.

### 📊 Cluster Power Optimization Benchmark Table

| Optimization State | Per-Card Draw (CMP 50HX) | RTX 2060 Draw | 3x Cluster Idle Total | Idle VRAM Temp |
| :--- | :---: | :---: | :---: | :---: |
| **Stock (Bugged VBIOS Idle)** | 75W – 85W | 11.5W | **~165W – 185W** *(240W before flash)* | ~55°C – 62°C |
| **Linux Governor (P3 Clamped @ 300MHz)** | **`31.8W – 33.7W`** | **`10.9W`** | **`~76.5W`** ⚡ *(−108W wall savings)* | **27°C – 39°C** |
| **Ideal P-State Sleep (P8 Deep Sleep)** | ~5W – 15W | ~10W | ~25W – 40W | ~25°C – 30°C |

---

## 2. Power Cap vs Inference Throughput

* **100W Power Cap (`-pl 100`)**: Capping CMP 50HX cards to 100W throttles GPU 1's core clock down to **1,305 MHz** during tensor-parallel splits, dropping generation speed from **18.0 tok/s $\rightarrow$ ~14.0 tok/s**.
* **150W Power Cap (`-pl 150`)**: Provides sufficient headroom for full **1,800–1,900 MHz boost clocks**, sustaining **18.03 tok/s** while protecting against unnecessary 225W power spikes and heat.

---

## 3. Automated Dynamic Power Governor ([`power-governor.py`](../scripts/power-governor.py))

* **The Problem with GPU Utilization Polling**: On Turing GPUs during batch=1 token generation, `utilization.gpu` fluctuates between 0% and 5% between token steps. Relying on GPU utilization causes false idle triggers.
* **The Solution**: Direct slot polling against `llama-server` (`http://127.0.0.1:8080/slots` `is_processing: True`).
* **Governor Behavior**:
  * When `is_processing: True`: Instantly sets unconstrained clocks (`nvidia-smi -rgc`) $\rightarrow$ **18.03 tok/s**.
  * When idle for > 15 seconds: Sets low-power 300–600 MHz core clocks $\rightarrow$ **drops power to ~31W/card**.

---

## 4. HPE ProLiant DL380p Gen8 Platform Learnings

### Sandy Bridge-EP (v1) vs. Ivy Bridge-EP (v2) PCIe 3.0 Jitter
* **The Sandy Bridge-EP v1 Bug**: Intel's first-generation PCIe 3.0 controller on `Xeon E5-2600 v1` (2012) suffered from transmitter signal margin attenuation (*Intel Errata BD78/BD105*). Over riser cables, the 8.0 GHz eye diagram degrades, forcing links down to Gen 1 (2.5 GT/s).
* **The Ivy Bridge-EP v2 Fix**: `Xeon E5-2600 v2` (22nm Tri-Gate) completely redesigned the PCIe 3.0 PHY with CTLE equalization, reliably locking Gen 3 speeds across risers.

### NUMA Socket Pinning
* Dual-socket Xeon servers have two distinct NUMA nodes.
* If all GPUs are plugged into **Primary Riser 1**, they are physically connected to **CPU Socket 1 (NUMA Node 1)**.
* Binding `llama-server` to NUMA Node 1 (`CPUAffinity` & `NUMAPolicy=bind`) eliminates all host-to-device memory traffic over the cross-socket Intel QPI bus.

### External PSU & Common Grounding
* When GPUs receive PCIe data signals from the server motherboard but 12V power from an external PSU:
* Ensure the **server chassis** and **external PSU casing** share a solid common ground to eliminate high-frequency ground loop noise on PCIe differential clock lines (`REFCLK`).

---

## 6. Low-Latency NCCL Ring Buffer Tuning (2x Prefill Speedup)

* **The Problem**: Default NCCL ring allocations (4MB–8MB) introduce high PCIe bus synchronization latency on multi-GPU AllReduce barriers on Turing architecture (`sm_75`).
* **The Solution**: 
  ```bash
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export NCCL_BUFFSIZE=2097152       # 2MB ring buffer (aligned to Turing L2 cache lines)
  export NCCL_NET_GDR_LEVEL=0         # Pure local PCIe bus routing
  export NCCL_P2P_DISABLE=0          # Direct P2P transfers
  export NCCL_ALGO=RING              # Direct 1-stage ring AllReduce
  export NCCL_PROTO=SIMPLE           # Eliminates LL128 packet framing overhead over PCIe
  ```
* **Measured Benchmark**:
  * Baseline Prefill Speed: **`72.0 tok/s`**
  * Tuned NCCL Prefill Speed: **`140.3 – 179.4 tok/s`** (**+149% prompt processing throughput!**)

---

## 7. Speculative Decoding & Multimodal (`mmproj`) Interaction

* **The Conflict**: In `llama.cpp`, loading a secondary GGUF draft model (`-md`) simultaneously with a multimodal vision projector (`--mmproj`) causes an internal context initialization conflict inside `libmtmd.so`.
* **The Resolution**:
  * **Multimodal Mode**: Use N-gram context-lookup speculation (`--spec-type ngram-simple --spec-ngram-simple-size-n 24 --spec-ngram-simple-size-m 4`), which runs entirely in primary context with 0 additional VRAM and full vision support.
  * **Pure Text Mode**: Remove `--mmproj` to enable secondary neural draft models (`-md /opt/models/gguf/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf`).

---

## 8. 20GB VRAM Modding Feasibility (CMP 50HX TU102)

* **Architecture**: CMP 50HX uses the **TU102 PCB layout (320-bit bus, 10 memory pads)**.
* **Hardware Swap**: Desolder 10x 1GB (8Gbit) GDDR6 BGA-180 chips and replace with 10x 2GB (16Gbit) chips (e.g., Samsung `K4ZAF325BM-HC14` or Micron `D9ZCL`). Adjust memory strapping resistor dividers.
* **Driver Patching for 20GB (WPR2 Fix)**: See [`docs/20GB_MOD_GUIDE.md`](20GB_MOD_GUIDE.md) for details on patching the NVIDIA open kernel driver to bypass the `0x8d` FWSEC error.
* **Capacity Scaling**:
  * 2x Modded Cards = **40 GB Total VRAM** (runs full 70B/72B models on 2 slots).
  * 4x Modded Cards = **80 GB Total VRAM** (enterprise A100-tier capacity for under $1,000).

---

## 9. Private NVAPI `NvAPI_GPU_SetForcePstate` (`0x025BFB10`)

* **The Discovery**: While `nvidia-smi` cannot force P8 state on headless Linux, the private NVAPI entry point `NvAPI_GPU_SetForcePstate` (`0x025BFB10`) inside `libnvidia-api.so.1` can explicitly force the GPU into **P8 state (645 MHz core / 405 MHz memory)**.
* **Implementation**: We packaged [`tools/cmp-pstate.py`](../tools/cmp-pstate.py) to directly invoke this entry point per PCI Bus ID.
* **Restoration**: Passing `--pstate 16` returns power management control to unconstrained P0/P3 states before active inference.

---

## 10. Automated PCIe Link Retraining Service (`cmp50hx-gen2`)

* **The Boot Lock**: CMP 50HX cards feature an internal PCIe speed lock register that takes a short period after boot to become accessible.
* **The Solution**: A systemd boot service triggers a link retrain on the upstream PCIe port after boot, ensuring the bus reliably trains at full speed without manual re-initialization.