Hardware: Add 20GB modding guide, cmp-pstate NVAPI tool, and document private NvAPI_GPU_SetForcePstate and PCIe retrain mechanisms

docs/20GB_MOD_GUIDE.md

@@ -0,0 +1,48 @@
+# 🛠️ CMP 50HX 20GB VRAM Mod & Driver Patching Guide
+
+This guide covers how to physically mod and software-unlock the **NVIDIA CMP 50HX** from **10 GB to 20 GB VRAM** (20,480 MiB per card), based on the validated research from [`xrip/cmp50hx-unlock`](https://github.com/xrip/cmp50hx-unlock).
+
+---
+
+## 1. Hardware Modification: Swapping GDDR6 BGA Chips
+
+### Architecture & Memory Topology
+* The CMP 50HX is built on the **TU102 PCB** with a **320-bit memory bus (10 memory pads)**.
+* **Stock Configuration**: 10x 1GB (8Gbit) 14Gbps GDDR6 BGA-180 chips (e.g., Samsung `K4Z80325BC-HC14`).
+* **20GB Configuration**: 10x 2GB (16Gbit) 14Gbps/16Gbps GDDR6 BGA-180 chips:
+  * **Samsung**: `K4ZAF325BM-HC14` or `K4ZAF325BM-HC16`
+  * **Micron**: `D9ZCL` (MT61K512M32KPA-14) / `D9WCW`
+  * **Hynix**: `H56G32AO8DX-004`
+
+### Hardware Strapping Resistors
+* After desoldering the old 8Gbit ICs and soldering the 16Gbit ICs:
+* Locate the **memory density strapping resistor dividers** on the rear of the TU102 PCB near the memory controller.
+* Move the strapping pull-up/pull-down resistor to configure the memory controller for **16Gbit per channel**.
+
+---
+
+## 2. The Software Barrier: NVIDIA Booter Error `0x8d`
+
+### Why Stock NVIDIA Drivers Fail on 20GB CMP 50HX
+* In stock closed drivers, the firmware security module (FWSEC) uses hardcoded WPR2 bounds calculated for a 10GB address space.
+* On a 20GB card, stock FWSEC creates:
+  ```text
+  WPR high:low = 04ffee00:04ffe000
+  span         = 00000e00
+  ```
+* Because the hardcoded 10GB bounds do not match the 20GB span, NVIDIA Booter aborts with error `0x8d`, GSP-RM fails initialization, and `nvidia-smi` detects no devices.
+
+### The Fix: Dynamic WPR2 Handling (`xrip/cmp50hx-unlock`)
+* The patched NVIDIA Open Kernel Module (`610.43.03`) reads the FWSEC WPR2 range dynamically for each GPU's PCI BDF (Bus/Device/Function).
+* It accepts span `0xe00`, saves the low/high bounds per BDF, and passes GSP readiness checks.
+* **Mixed Cards Supported**: You can run 10GB and 20GB CMP 50HX cards in the same system simultaneously because WPR2 bounds are indexed per PCI BDF.
+
+---
+
+## 3. Multi-Card VRAM Scaling for LLM Inference
+
+| Cluster Setup | Total VRAM | Model Capability |
+| :--- | :---: | :--- |
+| **2x 20GB CMP 50HX** | **40 GB VRAM** | Runs `Llama-3.3-70B` or `Qwen2.5-72B` (Q4_K_M) across 2 slots! |
+| **3x 20GB CMP 50HX** | **60 GB VRAM** | Runs `Qwen2.5-72B` (Q6_K / Q8_0) with full 64K+ context |
+| **4x 20GB CMP 50HX** | **80 GB VRAM** | Enterprise A100-class capacity (runs 70B FP8 / Q8_0 at 128K context) |

docs/HARDWARE_LEARNINGS.md

@@ -99,10 +99,23 @@ Comparing the hardware profiles and the cross-flashing outcome:
 ## 8. 20GB VRAM Modding Feasibility (CMP 50HX TU102)
 
 * **Architecture**: CMP 50HX uses the **TU102 PCB layout (320-bit bus, 10 memory pads)**.
-* **Memory Swap**:
-  * Desolder 10x 1GB (8Gbit) GDDR6 BGA-180 chips (e.g., Samsung `K4Z80325BC-HC14`).
-  * Solder 10x 2GB (16Gbit) GDDR6 BGA-180 chips (e.g., Samsung `K4ZAF325BM-HC14` or Micron `D9ZCL`).
-  * Modify memory strapping resistor dividers to signal 16Gbit density to the TU102 memory controller.
-* **Payoff**:
-  * 2x Modded Cards = **40 GB Total VRAM** (capable of running full 70B/72B models like `Llama-3.3-70B` or `Qwen2.5-72B` on just 2 dedicated x16 slots).
+* **Hardware Swap**: Desolder 10x 1GB (8Gbit) GDDR6 BGA-180 chips and replace with 10x 2GB (16Gbit) chips (e.g., Samsung `K4ZAF325BM-HC14` or Micron `D9ZCL`). Adjust memory strapping resistor dividers.
+* **Driver Patching for 20GB (WPR2 Fix)**: See [`docs/20GB_MOD_GUIDE.md`](20GB_MOD_GUIDE.md) for details on patching the NVIDIA open kernel driver to bypass the `0x8d` FWSEC error.
+* **Capacity Scaling**:
+  * 2x Modded Cards = **40 GB Total VRAM** (runs full 70B/72B models on 2 slots).
   * 4x Modded Cards = **80 GB Total VRAM** (enterprise A100-tier capacity for under $1,000).
+
+---
+
+## 9. Private NVAPI `NvAPI_GPU_SetForcePstate` (`0x025BFB10`)
+
+* **The Discovery**: While `nvidia-smi` cannot force P8 state on headless Linux, the private NVAPI entry point `NvAPI_GPU_SetForcePstate` (`0x025BFB10`) inside `libnvidia-api.so.1` can explicitly force the GPU into **P8 state (645 MHz core / 405 MHz memory)**.
+* **Implementation**: We packaged [`tools/cmp-pstate.py`](../tools/cmp-pstate.py) to directly invoke this entry point per PCI Bus ID.
+* **Restoration**: Passing `--pstate 16` returns power management control to unconstrained P0/P3 states before active inference.
+
+---
+
+## 10. Automated PCIe Link Retraining Service (`cmp50hx-gen2`)
+
+* **The Boot Lock**: CMP 50HX cards feature an internal PCIe speed lock register that takes a short period after boot to become accessible.
+* **The Solution**: A systemd boot service triggers a link retrain on the upstream PCIe port after boot, ensuring the bus reliably trains at full speed without manual re-initialization.