Hardware: Add 20GB modding guide, cmp-pstate NVAPI tool, and document private NvAPI_GPU_SetForcePstate and PCIe retrain mechanisms

docs/20GB_MOD_GUIDE.md

@@ -0,0 +1,48 @@
+# 🛠️ CMP 50HX 20GB VRAM Mod & Driver Patching Guide
+
+This guide covers how to physically mod and software-unlock the **NVIDIA CMP 50HX** from **10 GB to 20 GB VRAM** (20,480 MiB per card), based on the validated research from [`xrip/cmp50hx-unlock`](https://github.com/xrip/cmp50hx-unlock).
+
+---
+
+## 1. Hardware Modification: Swapping GDDR6 BGA Chips
+
+### Architecture & Memory Topology
+* The CMP 50HX is built on the **TU102 PCB** with a **320-bit memory bus (10 memory pads)**.
+* **Stock Configuration**: 10x 1GB (8Gbit) 14Gbps GDDR6 BGA-180 chips (e.g., Samsung `K4Z80325BC-HC14`).
+* **20GB Configuration**: 10x 2GB (16Gbit) 14Gbps/16Gbps GDDR6 BGA-180 chips:
+  * **Samsung**: `K4ZAF325BM-HC14` or `K4ZAF325BM-HC16`
+  * **Micron**: `D9ZCL` (MT61K512M32KPA-14) / `D9WCW`
+  * **Hynix**: `H56G32AO8DX-004`
+
+### Hardware Strapping Resistors
+* After desoldering the old 8Gbit ICs and soldering the 16Gbit ICs:
+* Locate the **memory density strapping resistor dividers** on the rear of the TU102 PCB near the memory controller.
+* Move the strapping pull-up/pull-down resistor to configure the memory controller for **16Gbit per channel**.
+
+---
+
+## 2. The Software Barrier: NVIDIA Booter Error `0x8d`
+
+### Why Stock NVIDIA Drivers Fail on 20GB CMP 50HX
+* In stock closed drivers, the firmware security module (FWSEC) uses hardcoded WPR2 bounds calculated for a 10GB address space.
+* On a 20GB card, stock FWSEC creates:
+  ```text
+  WPR high:low = 04ffee00:04ffe000
+  span         = 00000e00
+  ```
+* Because the hardcoded 10GB bounds do not match the 20GB span, NVIDIA Booter aborts with error `0x8d`, GSP-RM fails initialization, and `nvidia-smi` detects no devices.
+
+### The Fix: Dynamic WPR2 Handling (`xrip/cmp50hx-unlock`)
+* The patched NVIDIA Open Kernel Module (`610.43.03`) reads the FWSEC WPR2 range dynamically for each GPU's PCI BDF (Bus/Device/Function).
+* It accepts span `0xe00`, saves the low/high bounds per BDF, and passes GSP readiness checks.
+* **Mixed Cards Supported**: You can run 10GB and 20GB CMP 50HX cards in the same system simultaneously because WPR2 bounds are indexed per PCI BDF.
+
+---
+
+## 3. Multi-Card VRAM Scaling for LLM Inference
+
+| Cluster Setup | Total VRAM | Model Capability |
+| :--- | :---: | :--- |
+| **2x 20GB CMP 50HX** | **40 GB VRAM** | Runs `Llama-3.3-70B` or `Qwen2.5-72B` (Q4_K_M) across 2 slots! |
+| **3x 20GB CMP 50HX** | **60 GB VRAM** | Runs `Qwen2.5-72B` (Q6_K / Q8_0) with full 64K+ context |
+| **4x 20GB CMP 50HX** | **80 GB VRAM** | Enterprise A100-class capacity (runs 70B FP8 / Q8_0 at 128K context) |

docs/HARDWARE_LEARNINGS.md

@@ -99,10 +99,23 @@ Comparing the hardware profiles and the cross-flashing outcome:
 ## 8. 20GB VRAM Modding Feasibility (CMP 50HX TU102)
 
 * **Architecture**: CMP 50HX uses the **TU102 PCB layout (320-bit bus, 10 memory pads)**.
-* **Memory Swap**:
+* **Hardware Swap**: Desolder 10x 1GB (8Gbit) GDDR6 BGA-180 chips and replace with 10x 2GB (16Gbit) chips (e.g., Samsung `K4ZAF325BM-HC14` or Micron `D9ZCL`). Adjust memory strapping resistor dividers.
-  * Desolder 10x 1GB (8Gbit) GDDR6 BGA-180 chips (e.g., Samsung `K4Z80325BC-HC14`).
+* **Driver Patching for 20GB (WPR2 Fix)**: See [`docs/20GB_MOD_GUIDE.md`](20GB_MOD_GUIDE.md) for details on patching the NVIDIA open kernel driver to bypass the `0x8d` FWSEC error.
-  * Solder 10x 2GB (16Gbit) GDDR6 BGA-180 chips (e.g., Samsung `K4ZAF325BM-HC14` or Micron `D9ZCL`).
+* **Capacity Scaling**:
-  * Modify memory strapping resistor dividers to signal 16Gbit density to the TU102 memory controller.
+  * 2x Modded Cards = **40 GB Total VRAM** (runs full 70B/72B models on 2 slots).
-* **Payoff**:
-  * 2x Modded Cards = **40 GB Total VRAM** (capable of running full 70B/72B models like `Llama-3.3-70B` or `Qwen2.5-72B` on just 2 dedicated x16 slots).
   * 4x Modded Cards = **80 GB Total VRAM** (enterprise A100-tier capacity for under $1,000).
+
+---
+
+## 9. Private NVAPI `NvAPI_GPU_SetForcePstate` (`0x025BFB10`)
+
+* **The Discovery**: While `nvidia-smi` cannot force P8 state on headless Linux, the private NVAPI entry point `NvAPI_GPU_SetForcePstate` (`0x025BFB10`) inside `libnvidia-api.so.1` can explicitly force the GPU into **P8 state (645 MHz core / 405 MHz memory)**.
+* **Implementation**: We packaged [`tools/cmp-pstate.py`](../tools/cmp-pstate.py) to directly invoke this entry point per PCI Bus ID.
+* **Restoration**: Passing `--pstate 16` returns power management control to unconstrained P0/P3 states before active inference.
+
+---
+
+## 10. Automated PCIe Link Retraining Service (`cmp50hx-gen2`)
+
+* **The Boot Lock**: CMP 50HX cards feature an internal PCIe speed lock register that takes a short period after boot to become accessible.
+* **The Solution**: A systemd boot service triggers a link retrain on the upstream PCIe port after boot, ensuring the bus reliably trains at full speed without manual re-initialization.

tools/cmp-pstate.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Force one NVIDIA GPU into a P-state through the private NVAPI entry point."""
+
+import argparse
+import ctypes
+import re
+import sys
+
+
+NVAPI_OK = 0
+NVAPI_MAX_PHYSICAL_GPUS = 64
+NVAPI_ENUM_PHYSICAL_GPUS = 0xE5AC921F
+NVAPI_GPU_GET_BUS_ID = 0x1BE0B8E5
+NVAPI_GPU_SET_FORCE_PSTATE = 0x025BFB10
+NVAPI_INITIALIZE = 0x0150E828
+NVAPI_UNLOAD = 0xD22BDD7E
+
+
+def status_text(status):
+    return "0x%08x" % (ctypes.c_uint32(status).value,)
+
+
+def bus_number(bus_id):
+    match = re.search(r"(?:^|:)([0-9a-fA-F]{2}):[0-9a-fA-F]{2}\.[0-7]$",
+                      bus_id.strip())
+    if not match:
+        raise ValueError("invalid PCI bus ID: %s" % bus_id)
+    return int(match.group(1), 16)
+
+
+def query_function(query, function_id, restype, *argtypes):
+    address = query(ctypes.c_uint32(function_id))
+    if not address:
+        raise RuntimeError("NVAPI function 0x%08x is unavailable" % function_id)
+    return ctypes.CFUNCTYPE(restype, *argtypes)(address)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--bus-id", required=True,
+                        help="Linux PCI bus ID, for example 00000000:01:00.0")
+    parser.add_argument("--pstate", required=True, type=int,
+                        help="P-state number: 0..15, or 16 for automatic/high")
+    args = parser.parse_args()
+    if not 0 <= args.pstate <= 16:
+        parser.error("--pstate must be between 0 and 16")
+
+    try:
+        wanted_bus = bus_number(args.bus_id)
+        api = ctypes.CDLL("libnvidia-api.so.1")
+        query = api.nvapi_QueryInterface
+        query.argtypes = [ctypes.c_uint32]
+        query.restype = ctypes.c_void_p
+
+        initialize = query_function(query, NVAPI_INITIALIZE, ctypes.c_int32)
+        unload = query_function(query, NVAPI_UNLOAD, ctypes.c_int32)
+        enum_gpus = query_function(
+            query, NVAPI_ENUM_PHYSICAL_GPUS, ctypes.c_int32,
+            ctypes.POINTER(ctypes.c_void_p), ctypes.POINTER(ctypes.c_uint32))
+        get_bus_id = query_function(
+            query, NVAPI_GPU_GET_BUS_ID, ctypes.c_int32,
+            ctypes.c_void_p, ctypes.POINTER(ctypes.c_uint32))
+        set_force_pstate = query_function(
+            query, NVAPI_GPU_SET_FORCE_PSTATE, ctypes.c_int32,
+            ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint32)
+    except (OSError, RuntimeError, ValueError) as exc:
+        print("cmp-pstate: %s" % exc, file=sys.stderr)
+        return 1
+
+    status = initialize()
+    if status != NVAPI_OK:
+        print("cmp-pstate: NvAPI_Initialize failed (%s)" % status_text(status),
+              file=sys.stderr)
+        return 1
+
+    try:
+        handles = (ctypes.c_void_p * NVAPI_MAX_PHYSICAL_GPUS)()
+        count = ctypes.c_uint32()
+        status = enum_gpus(handles, ctypes.byref(count))
+        if status != NVAPI_OK:
+            print("cmp-pstate: GPU enumeration failed (%s)" %
+                  status_text(status), file=sys.stderr)
+            return 1
+
+        handle = None
+        for index in range(min(count.value, NVAPI_MAX_PHYSICAL_GPUS)):
+            bus = ctypes.c_uint32()
+            status = get_bus_id(handles[index], ctypes.byref(bus))
+            if status == NVAPI_OK and bus.value == wanted_bus:
+                handle = handles[index]
+                break
+        if handle is None:
+            print("cmp-pstate: NVAPI GPU for PCI bus %02x was not found" %
+                  wanted_bus, file=sys.stderr)
+            return 1
+
+        status = set_force_pstate(handle, args.pstate, 0)
+        if status != NVAPI_OK:
+            print("cmp-pstate: P%d request failed (%s)" %
+                  (args.pstate, status_text(status)), file=sys.stderr)
+            return 1
+        return 0
+    finally:
+        unload()
+
+
+if __name__ == "__main__":
+    sys.exit(main())