llama.cpp-xrip/docs/backend/snapdragon/developer.md

Hexagon backend developer details

Backend libraries

The Hexagon backend consists of two parts:

• libggml-hexagon This is the regular CPU-side GGML backend library, either shared or statically linked.

• libggml-htp-vNN This is the NPU-side (HTP stands for Hexagon Tensor Processor) shared library that contains the Op dispatcher and kernels. The correct library is selected automatically at runtime based on the HW version.

Here is an example of the build artifacts:

~/src/llama.cpp$ ls -l pkg-adb/llama.cpp/lib/libggml*
pkg-adb/llama.cpp/lib/libggml-base.so
pkg-adb/llama.cpp/lib/libggml-cpu.so
pkg-adb/llama.cpp/lib/libggml-hexagon.so      <<< CPU library
pkg-adb/llama.cpp/lib/libggml-htp-v73.so      <<< HTP op/kernels for Hexagon v73
pkg-adb/llama.cpp/lib/libggml-htp-v75.so
pkg-adb/llama.cpp/lib/libggml-htp-v79.so
pkg-adb/llama.cpp/lib/libggml-htp-v81.so

Memory buffers

The Hexagon NPU backend takes advantage of Snapdragon unified memory where all DDR buffers are accessible by CPU, GPU, and NPU. The NPU has dedicated tightly-coupled memory called VTCM (Vector Tightly-Coupled Memory). VTCM is used for intermediate data (such as dynamically quantized activations) and streaming buffers (chunks of weight and activation tensors fetched via DMA).

Large model handling

Hexagon NPU sessions have a 32-bit virtual address space window of around 3.5GB. In llama.cpp/GGML, each Hexagon session is mapped to a single GGML backend device (e.g., HTP0:0, HTP0:1, etc. when using GGML_HEXAGON_DEVICES, or HTP0, HTP1 in legacy mode).

To support running models larger than 3.5GB on a single device, the Hexagon backend dynamically maps and unmaps buffers:

• Buffers are allocated in shared DDR (RPCMEM) and mapped through FastRPC file descriptors. Non-pinned buffers use delayed mappings (FASTRPC_MAP_FD_DELAYED or FASTRPC_MAP_FD_DELAYED_EXTENDED).
• Pinned buffers (such as KV cache and active compute buffers) remain mapped throughout execution.
• Inactive weight buffers are dynamically mapped into the NPU session during batch buffer preparation (prep_op_bufs() in htp/main.c) and unmapped via htp_iface_munmap() when no longer needed by the active batch.
• This dynamic sliding window allows a single NPU session to execute models that exceed the 3.5GB window.

Alternatively, users can partition and split the model across multiple virtual sessions or physical NPUs using layer-splitting, tensor-splitting, or row-splitting modes. For user-facing execution modes and examples, see the Snapdragon user guide.

Op and Kernel Development Guidelines

Writing high-performance operators for Hexagon requires following specific guidelines.

DDR -> DMA -> VTCM Execution Pipeline

• Strongly prefer the DDR -> DMA -> VTCM -> compute (HVX/HMX) -> VTCM -> DMA -> DDR data flow.

• Direct HVX reads/writes from/to DDR are less efficient and should only be used as a fallback.

• Use dma_addr_t only for DMA base and final addresses. Form a final address by adding a 32-bit byte offset to a dma_addr_t tensor base address. This permits a 64-bit mapped base address on newer platforms while retaining 32-bit relative addressing.

• The DMA queue is a strict FIFO where operations must be pushed and popped in strict order.

• Follow the pipelined multi-buffering sequence properly (typically 2x to 16x buffering) so every push has a corresponding pop:

  1. In the prologue, push initial DDR -> VTCM transfers to prime the pipeline.
  2. In the loop body, wait for buffer N via DMA pop, launch HVX/HMX compute on buffer N, push VTCM -> DDR writeback of result N, and push DDR -> VTCM prefetch of buffer N+2.
  3. In the epilogue, pop all remaining in-flight transfers to drain the pipeline.

• Because every push must be matched by a pop, dma_queue_flush() is not required when the pipeline sequence is followed properly. Flushing is only used in rare exceptions where a batch of operations is pushed without individual pops.

• Use the DMA queue interface from dma-queue.h (dma_queue_push(), dma_queue_pop(), and dma_queue_flush()). See cumsum-ops.c and act-ops.c for reference implementations.

Avoid Scalar Reads and Writes to VTCM

• Access VTCM data using DMA transfers or HVX/HMX vector instructions rather than scalar reads and writes.

Avoid Scalar Division in Inner Loops

• Hexagon cores do not have hardware division instructions.
• For recurring divisions across iterations or threads, use fastdiv from hex-fastdiv.h with precomputed divisors (such as octx->ctx->mdev.count_div or octx->n_threads_div).
• Do not call init_fastdiv_values() for single-use divisions; use standard compiler division (/) instead.

Host-Side Precomputation via kernel_params

• Precompute tensor shapes, strides, scale conversions, tiling layouts, and validation checks on the host CPU during graph preparation in ggml-hexagon.cpp.
• Pack precomputed parameters into the operator's fixed kernel_params structure in htp_op_node (such as htp_mm_kernel_params, htp_unary_kernel_params, htp_fa_kernel_params, htp_get_rows_kernel_params).
• The NPU executes directly using octx->kernel_params without redundant runtime metadata extraction or validation.
• Strict Host-Kernel Alignment:
  • Verify that parameters calculated by the host CPU are strictly honored by the NPU kernel.
  • Ensure the kernel does not ignore host-computed fields (for example, falling back to octx->n_threads instead of using kparams->n_threads, or ignoring precomputed tasks_per_thread and chunk counts).
  • Both human developers and coding agents must audit both sides of the interface: ensure fields populated in kernel_params in ggml-hexagon.cpp are actively and consistently utilized by the corresponding operator entry point and worker threads in htp/*-ops.c.

Tracing Instrumentation

• All kernels must include trace events for performance profiling and timeline visualization in Perfetto (hex-profile.h).
• Surround compute sections with htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info) and htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info).
• Use specific event types for major phases:
  • HTP_TRACE_EVT_HVX_COMP: Vector compute execution.
  • HTP_TRACE_EVT_DMA: DMA transfer wait or poll cycles.
  • HTP_TRACE_EVT_FENCE: Multi-device fence barrier synchronization.
  • HTP_TRACE_EVT_L2FLUSH: L2 cache cleaning operations.
• Pass meaningful progress metrics (such as row index, chunk index, or token index) in the 16-bit info parameter.

Work Queue and Threading

• Distribute parallel work across NPU worker threads using the thread pool work queue:

  work_queue_run(ctx->work_queue, worker_func, &op_ctx, n_threads);
  

• Keep worker functions independent and re-entrant. Worker threads should only operate on their designated chunk of rows or elements.

Avoid Redundant Defensive NULL Checks

• Do not add defensive NULL checks or assertions for internal framework pointers or required graph operands and outputs. Internal pointers include ctx, octx, local context structs like *ctx, kparams, and worker callback data.
• These pointers are architectural invariants during kernel execution and host-side graph preparation.
• Do not turn an invariant violation into an unsupported operation or missed fusion. Checks such as if (!octx || !octx->ctx) clutter the code, obscure intent, and hide upstream errors.
• Distinction: octx->src[N] pointers can be NULL by design and must be checked when optional. Examples include attention masks, optional bias or weights in fused kernels, and frequency factors.

supports_op Independence from Buffer and Repack State

• supports_op (and op-specific checkers such as supported_mul_mat(), supported_get_rows()) runs during GGML graph scheduling before buffers are allocated and before tensor data is populated.
• Never check buffer usage or repacking state in supports_op:
  • t->buffer may be NULL or uninitialized.
  • t->data may be NULL.
  • buffer->usage is unknown at scheduling time.
  • is_tiled or extra->flags & GGML_HEXAGON_TENSOR_TILED is not set yet.
• Operator support decisions must depend strictly on static tensor properties: operand types (t->type), shapes (t->ne), strides (t->nb), and operation parameters.
• Repacking (converting weights to tiled layout) is an internal optimization applied during set_tensor() or rt-repack. Non-weight tensors or test harness tensors with the same type remain in flat layout.
• Therefore, any supported type (such as BF16, F16, quantized types) must be supported both when repacked into tiles and when stored flat in memory.

Multiline Macro Formatting

• Keep trailing backslashes in multiline #define macros cleanly aligned to a consistent column.

• Avoid trailing whitespace after macro backslashes.

• Use scripts/snapdragon/ggml-hexagon-align-macros.py to inspect, diff, or automatically align macro definitions across Hexagon kernel sources:

  # Check for misaligned macros
  python3 scripts/snapdragon/ggml-hexagon-align-macros.py ggml/src/ggml-hexagon/htp/
  
  # Fix misaligned macros in-place
  python3 scripts/snapdragon/ggml-hexagon-align-macros.py --fix ggml/src/ggml-hexagon/htp/
  

Binary Inspection and Spill Analysis

Use scripts/snapdragon/ggml-hexagon-inspect.py to audit Hexagon binaries for register spills, unexpected float promotions, or disassembly:

• Always verify that compute kernels have zero in-loop vector spills (--spills --strict) and no float promotions (--promotions).
• Avoid excessive loop unrolling (#pragma unroll), which increases register pressure and causes spills.

# Check for vector and scalar register spills
python3 scripts/snapdragon/ggml-hexagon-inspect.py --spills --strict --func "^compute_"

# Check for float promotions
python3 scripts/snapdragon/ggml-hexagon-inspect.py --promotions --func "^compute_"

# Disassemble with annotated loops and spill markers
python3 scripts/snapdragon/ggml-hexagon-inspect.py --disasm compute_same_shape_div_f32

# Resolve crash addresses to function symbols and lines
python3 scripts/snapdragon/ggml-hexagon-inspect.py --addr2line 0x51a30 0x5ba54

Multi-Device Partitioning (mdev)

Multi-device (mdev) mode enables row-level tensor parallel execution across multiple physical NPU cores or virtual NPU sessions.

128-Byte Cache Line Alignment

• Shared tensor buffers reside in DDR (RPCMEM) with a 128-byte cache line granularity (HEX_L2_LINE_SIZE = 128 bytes, HTP_TENSOR_MDEV_LINE_SIZE).
• Rule: Multi-device work partitions that write directly to DDR through HVX/L2 must align destination write regions to 128-byte cache line boundaries so distinct devices never share or overwrite the same cache line.
• DMA writes to DDR are not subject to this cache-line ownership rule. They may use smaller non-overlapping destination ranges when the operator only writes through DMA.

Partitioning Helpers in htp-tensor.h

Common partitioning logic is factored into reusable inline helpers in htp-tensor.h:

1. htp_tensor_mdev_rows_per_chunk: Determines the minimum number of rows per chunk so that the chunk byte size is a multiple of 128 bytes:

   rows_per_chunk = 128 / hex_gcd_u32(row_size, 128)
   

   If the active row and outer strides are already multiples of 128 bytes, rows_per_chunk = 1. Returns false if the tensor cannot be safely row-partitioned (such as unaligned base pointer, permuted layout, or non-128-byte aligned outer strides).

2. htp_tensor_mdev_partition: Calculates the per-device work range struct htp_tensor_mdev_range { uint32_t start; uint32_t count; } given total_units, units_per_chunk, mdev_idx, mdev_count, and the precomputed mdev_count_div. Handles chunk distribution across devices, assigns remainder units to the last device, and automatically triggers single-device fallback when partitioning is unsafe.

Row-Partitioned Operators

For row-wise operators that write directly to DDR (such as activations in act-ops.c, binary ops in binary-ops.c, and unary ops in unary-ops.c):

const uint32_t total_rows   = ne01 * ne02 * ne03;
const size_t   dst_row_size = dst->ne[0] * elem_size;

uint32_t row_start = 0;
uint32_t nrows     = total_rows;

if (octx->ctx->mdev.count > 1) {
    uint32_t rows_per_chunk = 0;
    htp_tensor_mdev_rows_per_chunk(dst, elem_size, (uint32_t) dst_row_size, &rows_per_chunk);
    const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(
        total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
    row_start = range.start;
    nrows     = range.count;
}

if (nrows == 0) {
    return HTP_STATUS_OK;
}

Element-Partitioned Operators

For flat element-wise operations that write directly to DDR:

• Partition total linear elements N = ne0 * ne1 * ne2 * ne3 in 128-byte cache line chunks (elems_per_line = (elem_size == 4) ? 32 : 64).
• Requires strict 1D contiguity: htp_tensor_is_contiguous(dst, elem_size) and 128-byte aligned destination pointer htp_tensor_mdev_data_aligned(dst).
• If contiguous and aligned, pass elems_per_line to htp_tensor_mdev_partition; otherwise pass 0 to trigger Device 0 fallback.

Single-Device Fallback (Device 0)

• Fallback to Device 0 (mdev.idx == 0) when partitioning would cause cache line tearing or there are too few aligned chunks.
• Triggers:
  1. Destination tensor cannot be safely partitioned (rows_per_chunk == 0 or non-contiguous/unaligned buffer).
  2. Total aligned chunks < mdev_count.
• Device 0 processes the entire tensor [0, total_units).
• Devices 1 ... N-1 receive count = 0 and return HTP_STATUS_OK immediately.

Flatten Outer Dimensions Globally

• Never partition solely on ne01 (dimension 1).
• Partitioning only on ne01 repeats the device boundary across every 2D slice (ne02, ne03). If each 2D slice is small, false sharing occurs repeatedly throughout the tensor.
• Always flatten outer dimensions globally: total_rows = ne01 * ne02 * ne03 and partition once across the combined row space.

Stateless Starting Coordinates

• Do not use incremental state variables across slices that assume the thread or device starts at index 0.
• Precompute starting multidimensional coordinates at r = row_start (or e = elem_start) once using fastdiv.
• In inner loops, step base pointers directly (ptr += stride) or reset/wrap coordinates explicitly (if (++i01 == ne01) { ... }).

Clean Range Encapsulation

• Initialize single-device default ranges at declaration:

  uint32_t row_start = 0;
  uint32_t nrows     = total_rows;
  

• Encapsulate all multi-device logic inside if (octx->ctx->mdev.count > 1). If the block is omitted or compiled out, the operator runs standard single-device execution untouched.

• Do not propagate mdev_ prefixes to worker functions or context structs. Worker threads are device-agnostic and should only receive standard range parameters (ctx.row_start, ctx.nrows).

• In worker threads, calculate row intervals using standard arithmetic:

  const uint32_t ir0 = ctx->row_start + dr * ith;
  const uint32_t ir1 = MIN(ir0 + dr, ctx->row_start + ctx->nrows);
  

  In single-device mode (row_start == 0), this naturally simplifies to dr * ith and MIN(ir0 + dr, ctx->nrows) with zero overhead.

Multi-Device Synchronization

Multi-device execution synchronizes worker sessions across devices using explicit barriers and tensor cache flushing.

Synchronization Fence Protocol

Multi-device execution synchronizes worker sessions through atomic fence slots and barriers defined in htp-fence.h:

[NPU Session 0]                              [NPU Session 1]
       |                                            |
  (Input Prep)                                 (Input Prep)
       |                                            |
  Pre-Op Barrier ----------------------------- Pre-Op Barrier
  (htp_mdev_group_barrier)                     (htp_mdev_group_barrier)
       |                                            |
  Kernel Execution                             Kernel Execution
  (Output Slice 0)                             (Output Slice 1)
       |                                            |
  Tensor Cache Flush                           Tensor Cache Flush
  (htp_dcache_flush_dirty_tensors)             (htp_dcache_flush_dirty_tensors)
       |                                            |
  Post-Op/Batch Barrier ---------------------- Post-Op/Batch Barrier
  (htp_mdev_group_barrier)                     (htp_mdev_group_barrier)
       |                                            |
  Return Response to Host                      Return Response to Host

Atomic Fence Slots and Cache Invalidation

• Fence synchronization operates on dedicated RPCMEM shared memory mapped across all participating sessions (ctx->mdev.fence_base).

• Each device owns a dedicated 128-byte cache-line aligned fence slot:

  atomic_uint * my_fence = htp_mdev_fence_slot(fence_base, mdev_idx);
  

• Writing to fence (htp_fence_write): Stores seq and status, issues a syncht thread synchronization barrier, and flushes/invalidates the line using Q6_dccleaninva_A(fence).

• Reading from peer fence (htp_fence_read): Executes Q6_dccleaninva_A(fence) and syncht before reading atomic values to ensure fresh data from DDR.

Deterministic Monotonic Sequence Numbers

• Barrier fences use monotonically increasing sequence numbers:

  const uint32_t seq = ++ctx->mdev.fence_seq;
  

• Comparing sequence numbers with signed arithmetic (int32_t)(peer_seq - seq) >= 0 prevents race conditions or misaligned barrier arrivals across iterations.

• If any peer reports an error status (peer_status > HTP_STATUS_OK), the barrier propagates the error and unblocks immediately.

Tensor Cache Flush and Pipeline Completion

• In the kernel, ensure all pushed DMA operations have been popped in strict FIFO order to drain the queue.
• Use htp_dcache_flush_dirty_tensors() to flush specific dirty tensors back to DDR:
  • htp_dcache_flush_dirty_tensors() flushes modified tensor address ranges, or the full D-cache when their total size exceeds the flush threshold, ensuring peer devices and the host CPU observe consistent data in DDR.
• Never signal completion before all DMA transfers are drained and dirty tensor flushes have completed.