Initial commit: Complete deployment scripts, power governor, systemd units, and architecture documentation for Turing multi-GPU LLM rig

docs/HARDWARE_LEARNINGS.md

@@ -0,0 +1,70 @@
+# 🔬 Hardware Architecture & Key Technical Learnings
+
+This document details the practical hardware behaviors, quirks, and engineering solutions discovered while building and optimizing this 3-GPU Turing inference rig on an **HPE ProLiant DL380p Gen8** server.
+
+---
+
+## 1. NVIDIA CMP 50HX Mining GPUs: VBIOS & Power States
+
+### Why CMP 50HX Mining Cards Idle at ~75W–85W Stock
+* **No Display / Headless Architecture**: CMP 50HX mining cards lack display outputs and display engines.
+* **Missing Deep P8 VBIOS Tables**: Standard GeForce VBIOSes drop to P8 state (405 MHz GDDR6 / ~10W) when no display is active. Mining VBIOSes lock the cards in P0 state (1,890 MHz core / 7,000 MHz GDDR6) by default.
+
+### OEM Reference VBIOS vs. MSI VBIOS
+Comparing the hardware profiles of two CMP 50HX cards on the exact same rig:
+* **GPU 2 (MSI Board `0x1462`, VBIOS `90.02.60.00.17`)**:
+  * Idles at **`~31.8W`** when core is locked to 300–600 MHz.
+  * Minimum fan speed: **`25%`**.
+  * Aggressive P3 voltage gating tables.
+* **GPU 1 (NVIDIA Reference OEM `0x10DE`, VBIOS `90.02.60.00.01`)**:
+  * Idles at **`~61.2W`** when core is locked to 300–600 MHz.
+  * Minimum fan speed: **`40%`** (locked in VBIOS).
+  * Higher static VRM voltage rail leakage.
+* **Actionable Solution**: Cross-flash GPU 2's MSI VBIOS (`90.02.60.00.17`) onto GPU 1 using `nvflash -6` to cut GPU 1's idle draw by ~30W and match fan curves.
+
+---
+
+## 2. Power Cap vs Inference Throughput
+
+* **100W Power Cap (`-pl 100`)**: Capping CMP 50HX cards to 100W throttles GPU 1's core clock down to **1,305 MHz** during tensor-parallel splits, dropping generation speed from **18.0 tok/s $\rightarrow$ ~14.0 tok/s**.
+* **150W Power Cap (`-pl 150`)**: Provides sufficient headroom for full **1,800–1,900 MHz boost clocks**, sustaining **18.03 tok/s** while protecting against unnecessary 225W power spikes and heat.
+
+---
+
+## 3. Automated Dynamic Power Governor ([`power-governor.py`](../scripts/power-governor.py))
+
+* **The Problem with GPU Utilization Polling**: On Turing GPUs during batch=1 token generation, `utilization.gpu` fluctuates between 0% and 5% between token steps. Relying on GPU utilization causes false idle triggers.
+* **The Solution**: Direct slot polling against `llama-server` (`http://127.0.0.1:8080/slots` `is_processing: True`).
+* **Governor Behavior**:
+  * When `is_processing: True`: Instantly sets unconstrained clocks (`nvidia-smi -rgc`) $\rightarrow$ **18.03 tok/s**.
+  * When idle for > 15 seconds: Sets low-power 300–600 MHz core clocks $\rightarrow$ **drops power to ~31W/card**.
+
+---
+
+## 4. HPE ProLiant DL380p Gen8 Platform Learnings
+
+### Sandy Bridge-EP (v1) vs. Ivy Bridge-EP (v2) PCIe 3.0 Jitter
+* **The Sandy Bridge-EP v1 Bug**: Intel's first-generation PCIe 3.0 controller on `Xeon E5-2600 v1` (2012) suffered from transmitter signal margin attenuation (*Intel Errata BD78/BD105*). Over riser cables, the 8.0 GHz eye diagram degrades, forcing links down to Gen 1 (2.5 GT/s).
+* **The Ivy Bridge-EP v2 Fix**: `Xeon E5-2600 v2` (22nm Tri-Gate) completely redesigned the PCIe 3.0 PHY with CTLE equalization, reliably locking Gen 3 speeds across risers.
+
+### NUMA Socket Pinning
+* Dual-socket Xeon servers have two distinct NUMA nodes.
+* If all GPUs are plugged into **Primary Riser 1**, they are physically connected to **CPU Socket 1 (NUMA Node 1)**.
+* Binding `llama-server` to NUMA Node 1 (`CPUAffinity` & `NUMAPolicy=bind`) eliminates all host-to-device memory traffic over the cross-socket Intel QPI bus.
+
+### External PSU & Common Grounding
+* When GPUs receive PCIe data signals from the server motherboard but 12V power from an external PSU:
+* Ensure the **server chassis** and **external PSU casing** share a solid common ground to eliminate high-frequency ground loop noise on PCIe differential clock lines (`REFCLK`).
+
+---
+
+## 5. 20GB VRAM Modding Feasibility (CMP 50HX TU102)
+
+* **Architecture**: CMP 50HX uses the **TU102 PCB layout (320-bit bus, 10 memory pads)**.
+* **Memory Swap**:
+  * Desolder 10x 1GB (8Gbit) GDDR6 BGA-180 chips (e.g., Samsung `K4Z80325BC-HC14`).
+  * Solder 10x 2GB (16Gbit) GDDR6 BGA-180 chips (e.g., Samsung `K4ZAF325BM-HC14` or Micron `D9ZCL`).
+  * Modify memory strapping resistor dividers to signal 16Gbit density to the TU102 memory controller.
+* **Payoff**:
+  * 2x Modded Cards = **40 GB Total VRAM** (capable of running full 70B/72B models like `Llama-3.3-70B` or `Qwen2.5-72B` on just 2 dedicated x16 slots).
+  * 4x Modded Cards = **80 GB Total VRAM** (enterprise A100-tier capacity for under $1,000).