* hex-build: bump toolchain, enable v85
* hex-build: bump toolchain to v0.8 in the CI and build scripts
* hex-build: add a little python script for swiv needed on some iot devices
* hex-build: fix flake8 warnings
* hex-arch: update version cap to v85
* hex-inspect: flexible TOOLS_ROOT
* hex-build: update to snake_case naming
* hex-build: update leftover old toolchain refs
* hex-target: update build and run scripts to support ubuntu with adb transport
* hexagon: move the converting CPY in blocks of rows per DMA descriptor so short rows are not DMA-latency bound
* hex-cpy: cleanup cpy cursor handling
* hex-cpy-concat: introduce threaded dma/hvx gather-row helper
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
* hex-cpy: remove leftover rejections for extended buffers
* hex-cpy: relax the gate for fast-path
* hex-cpy: remove scalar divs and vtcm access and use explicit gather_sync
* hex-cpy: add proper support for mdev work splitting to fastpath
* hex-cpy: optimize dense gather-row tasks
* hex-cpy: unroll gather-rows loops
* hex-cpy: add fastpath for f16
* hex-cpy-concat: support for colapsing 3/4D dims
* hex-cpy: double-buffer support
* hex-cpy: factor out gather-rows into more of an op by itself
* hex-cpy-concat: dedup kernel-params
* hex-cpy-concat: step-down nr to fit instead of hard fail
* hex-gather: replace div with a mask
* hex-gather-rows: cleanup dma pipeline and remove the stalls
* hex-gather: specialize compute funcs
* hex-gather: reduce scalar reg.spills and func calls
* hex-gather: do not recompute rows-per-thread on the npu
* hex-build: add support for building/installing just hexagon
* hex-cpy-concat: remove redundant barrier in concat and move cpy barrier under fence
* hex-gather: annotations for dma-out
* hex-mdev: make all key vtcm_acquire sync state atomic (fixes random hangs in mdev setups)
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Ruslan Karymov <67752306+karusrus@users.noreply.github.com>
* port setup-build.ps1 to setup_sdk.py, to facilitate installation of Hexagon and OpenCL SDKs on Windows
* rename setup_sdk.py -> setup-sdk.py
* flake8 fix: print() -> logger.info()
---------
Co-authored-by: Kristopher Urquhart <kurquhar@qti.qualcom.com>
* hexagon: use non-host bufs by default and make the backend fully async
* hex-hb: remove optional hostbuf support and fix async copy
* hex-unary: relax supported unary check
* hex-bufs: use same get_alignment for host bufs
* snapdragon: bump android_platform to 34
* hex-rows: super hacky get/set rows for q8_0
* hex-get-rows: fix q8_0
* hex-get-rows: supprot for f16 and cleanup for q8_0
* hex-get-rows: generic macros and specialized thread funcs
* hex-get-rows: add DMA pipeline, vtcm_layout and kernel params
* hex-set-rows: fix q8_0 support, add dma and tracing
* hex-tests: override nmse threshold for HTP of Q8_0 quants
* hex-fa: add support for Q8_0 with inplace dequantizers
* hex-get-rows: simplify type dispatch
* hex-rows: simplify GET/SET_ROWS DMA pipeline
* hex-async: add events, set/get-tensor-async and rest of the async api support
* hex-repack: use slice instead of expert in repack functions
* hex-cpy: update event/async-cpy logging
* hex-set-rows: optimize smaller tensors
* hex-geglu: fix perf regression with larger tensors
* hex-get-rows: add missing header
* hex-set-rows: add missing header
* hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default
* hexagon: do not reject ops with non-heaxon buffers
* hex-get-rows: apply >=32 restriction only for q8_0
* hex-res: bump vtcm acquire timeout to 10 seconds
* hex-bufs: add support for cloning buffers between sessions to speed up tensor copies
* hex-async: rework event recording and batch flushing and integrate with meta backend
* hex-bufs: improved handling of repacked tensors
* hex-repack: handle get_tensor_2d offsets
* hex-dev: add support for devices with multiple NPUs
* hex-sync: add support for sync tokens to synchronize npu devices for async splits
* hex-mmap: cleanup mmap calls and add a retry for robustness
* hex-sync: add failsafe if sync wait gets stuck
* hex-sync: use sync_seq to check for completed events
* hex-sync: rotate tokens for extra robustness
* hex-devs: add supprot for legacy device names for now
* hex-bufs: add support for auto-cloning buffers from diff sessions
* hex-fusion: simplify and optimize htp-opnode fusion handling
* hex-sync: override opnode name so that it shows up in the profiles
* hex-trace: update scripts to handle multiple devices
* hex-sync: bump the size of the opbatch queue and number of sync tokens
* hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma
* hex-sync: add graph-flush threshold to avoid single op batches
* hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops
* hex-bufs: introduce tensor->extra and shadow_bufs for repacking
* hex-l2: flush tiny tensors inline
* hex-sync: use explicit l2flush for sync tokens
* hex-extra: track weight flags via tensor extra
* hex-fence: rename sync to fence
* hex-repack: proper handling of set-tensor-2d in the shadow_buf
* hex-trace: remove obsolete opstage mask that we used for profiling
* hex-env: remove obsolete use_hmx variable
* hexagon: new unified run.py and build.py and updated docs
* snapdragon: update run script to auto-escapt test-backend-op -p argument
* hex-scripts: fix trailing spaces
* hex-scripts: fix flake8 warnings
* snapdragon: cleanup dst lib/bin dirs before copying new build
* hex-ops: add support for allreduce
* hex-ar: improved allreduce with dma pipeline
* hex-ar: align macros
* hex-ar: consistent use of fence_seq
* hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback
* hex-ar: add proper synchronize handling for ALLREDUCE
* hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold
* hex-ar: bump block size to improve dma efficiency
* hex-ar: fused ALLREDUCE+ADD
* hex-ar: cleaner fence buffer management
* hex-ar: futher allreduce tweaking to remove race conditions
* hex-ar: add simple solver and remove non-dma kernels
* hex-ar: add row-broadcast to fuse with bias ADD
* hex-fence: pass seq numbers via op_params
* hex-ar: allow for both entry/exit seq for completing entry wait
* hex-ar: align macros
* hex-ar: do not refetch broadcast row
* hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things
* hex-fusion: fix incorrect MUL_MAT reordering
* hex-mm: make fused 2x and 3x matmuls more generic
* hex-fusion: move tensor fusion tagging to graph_compute
* hexagon: make sure to copy tensor->extra by value
* hex-get-rows: fix offset calc with row-chunking
* hex-repack: get_tensor_2d fixes for non-zero offsets
* snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default
* hex-devices: use legacy device nameing by default to ease the transition
* hex-devices: hardcode CDSP domain IDs for current devices for now
* hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values
* hex-optrace: more robust handling of the fence events
Max Krasnyansky
kurquharandKristopher Urquhart