* common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client
common_http_client() was ignoring standard proxy env vars, causing
downloads via -hf and similar flags to fail silently in environments
behind an HTTP proxy (corporate networks, air-gapped infra, etc.).
Checks HTTPS_PROXY/https_proxy for https:// URLs and HTTP_PROXY/http_proxy
for http:// URLs, in that order. If a proxy URL is set and parseable,
passes it to httplib's set_proxy() and set_proxy_basic_auth() as
appropriate. Malformed proxy URLs are silently ignored so existing
callers aren't affected.
Fixes#21694
* common: drop misleading comment about generic proxy fallback
The comment referenced a 'generic one' that doesn't exist in the code — both
HTTP_PROXY and HTTPS_PROXY are scheme-specific. Reviewer noted the mismatch.
Addresses review feedback from @ServeurpersoCom.
* common : warn when the proxy URL from the environment is malformed
* common : honor NO_PROXY when selecting the http proxy
* ci : retrigger checks (test-thread-safety flake)
* common : bypass the proxy for loopback destinations
llama cli talks to its spawned server on 127.0.0.1 through common_http_client, so with HTTP_PROXY set and no NO_PROXY the health polling went to the corporate proxy, which rejects loopback targets, and the wait loop never exits ("Loading model..." hangs forever).
A remote proxy can never reach the client machine's own loopback, so these destinations are now always contacted directly, before NO_PROXY is consulted: localhost, *.localhost, 127.0.0.0/8, ::1 and 0.0.0.0. Similar to Go's net/http. Proxy handling for downloads and remote --server-base endpoints is unchanged.
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
* docs: move proxy tip from README to server docs
per review: the top level README stays lean; the proxy howto now lives with the server docs under Usage
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
---------
Signed-off-by: texasich <101962694+texasich@users.noreply.github.com>
Co-authored-by: texasich <texasich@users.noreply.github.com>
Co-authored-by: texasich <texasich@gmail.com>
Co-authored-by: marecabo <23156476+marecabo@users.noreply.github.com>
Use the instance's is_mistral_format flag when DeepseekV2Model reloads
configuration, allowing Mistral MoE checkpoints to read params.json.
Fixes#29899
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* llama-eval : add sandboxed execution for code suites
Adds a lightweight replacement for the Docker containers usually used to run
generated code, so a benchmark can be executed without a daemon, an image or
root.
A cached venv is provisioned per suite and reused; candidate programs run in a
throwaway working directory as their own process group, under the strongest
isolation the host offers:
bwrap read-only root, tmpfs /tmp and $HOME, no network
unshare no network, via --map-current-user
rlimit cpu/memory/file-size/process caps only
Candidate code must not run as uid 0, which is why "unshare -r" is a
last-resort tier that warns: root bypasses file permission bits, so a test
asserting that a write to a chmod 0444 file fails would instead see it succeed.
Two details worth noting for the bwrap tier. Only paths hidden by the $HOME
tmpfs are bound back, because bwrap cannot mount onto a symlink and
version-manager alias directories usually are one. And the venv's base
interpreter tree is resolved by walking the symlink chain, since a
version-managed interpreter is commonly reached through an unversioned alias
that startup resolves through.
Suites can declare provisioning steps, which run once at build time while the
network is still up, so test time never needs it.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : add HumanEval and ClassEval code suites
Adds two execution-graded suites alongside the existing text-graded ones. The
model's reply is parsed for a python code block, combined with the task's own
unit tests, and run in the sandbox.
--dataset humaneval 164 function-level problems, scored pass@k
--dataset classeval 100 class-level tasks, scored at class level (feeding
pass@k) and method level, matching upstream
pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the
dataset size gives that many samples per problem, aggregated with the unbiased
estimator from the Codex paper.
Program construction and result interpretation are dataset-owned hooks, because
the two suites differ: one HumanEval problem is a function judged by a single
assert, whereas one ClassEval task is a class judged by many independent test
methods, and reducing that to pass/fail would throw away partial credit. Suites
also declare their own package set, provisioning steps, interpreter and timeout.
The package set is derived from the dataset's own imports rather than a
requirements file. HumanEval turns out to need no third-party packages at all;
ClassEval needs thirteen, two of which must be declared because no import
mentions them -- lxml is selected by string in ClassEval_44, and scipy is a
gensim requirement.
Notes on ClassEval specifically:
* gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two
pure-Python APIs it uses are provided by a small shim instead.
* The three nltk corpora it needs are pre-seeded at venv build time. These are
the modern names -- the dataset asks for the since-renamed punkt and
averaged_perceptron_tagger.
* It defaults to a patched fork, ilintar/ClassEval, because three upstream
tests cannot be passed by anything: a hard-coded 2024 date checked against
datetime.now(), an assertion on the dataset author's own hostname, and an
exact float comparison. Reference solutions go from 96/100 to 100/100 classes
and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the
model is asked to write is unchanged; --dataset-source selects the original.
--exec-seed (default 0) seeds the RNG before each test, so suites whose tests
draw random inputs score reproducibly. ClassEval_58 fails its own reference
solution about 30% of the time without it.
The simulator gains both suites, so the harness can be exercised without a
model.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : fix silent isolation downgrade, generalise the sandbox runner
The isolation probe ran the interpreter without binding it back, so whenever
this tool is itself started from a venv under $HOME -- the usual case, since it
needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the
sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in
the output looked wrong when that happened, and the existing suites have been
running that way. The probe now uses the same interpreter and the same binds the
suite will actually get.
Two additions the agentic suite needs, neither of which changes the existing
suites:
* run_argv() runs an arbitrary command in a caller-owned directory that survives
the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take
exactly the same path as before. A multi-step agent needs its edits to persist
across many invocations, and a suite for a language other than Python needs to
drive a linter and a test runner rather than a single source string.
* RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved
address space rather than resident memory, which is the wrong instrument for a
runtime that reserves a large region and commits little of it: node's
TypeScript type-stripping goes through WebAssembly, reserves gigabytes per
instance, and aborts at 4 GiB with an out-of-memory error that reads like a
fault in the code under test.
HumanEval stays at 164/164 and ClassEval at 100/100.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : add SACB, a multi-turn agentic coding suite
`--dataset agentic` is a different shape from the other code suites. Rather than
answering a single prompt, the model is given a small but real repository, a
narrow tool set and a bug report, drives its own multi-turn conversation, and is
graded by running a test suite it was never allowed to see.
The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three
hand-written repositories in Python and TypeScript, hosted at ilintar/SACB.
Grading needs no container: a cached venv gives ruff and mypy for Python, and
node's own test runner plus tsc for TypeScript, so the whole dependency set is
four packages.
Three constraints are deliberate:
* No shell, and no way to run the tests. The only feedback channel is `lint`. A
syntax or type error can be driven out mechanically; a logic error has to be
reasoned about. A harness that lets an agent iterate against the test suite
measures something closer to search than to understanding.
* Two edit tools, one line-addressed and one content-addressed. Which one a
model reaches for, and whether it keeps line numbers straight after its own
earlier edit, is itself a signal.
* The tests never touch the working tree. They are held outside it and copied
into a throwaway copy only once the agent has stopped, so they can be neither
read nor edited. Every tool path is resolved and confinement-checked against
the repository root, so a symlink planted inside the tree cannot reach out of
it.
A task counts as resolved when every test that was failing passes and nothing
that already passed broke. Both sets are derived by running the tests against
the defective tree and against the reference fix, never written by hand, so a
task whose defect no test exercises cannot enter the corpus. Many tasks carry
several independent defects, so the fraction of target tests passed is reported
alongside, separating "fixed two of the three faults" from "changed nothing".
An episode that arrives with no tool call and no `finish` is nudged rather than
ended. Models routinely narrate their analysis mid-task, and treating that as
completion ends the episode with no edits and scores a reasoning failure that
never happened; about one episode in four exits that way.
Nothing here is installed unless this suite is selected, and then only for the
language being run: agentic_eval is imported lazily and each toolchain is keyed
per language.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : make the agentic hooks typed rather than discovered
`ty` flagged nine unresolved attributes, of which the seven in main() predate
the agentic suite: `EvalState.dataset` is Optional and was dereferenced without
narrowing.
* `loaded_dataset` turns the Optional into a value in one place, raising rather
than handing back None, so callers do not each have to narrow it.
* `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were
reached through getattr with a default at three call sites, which no checker
can see through; `run()` now raises a clear error naming the suite instead of
failing as a missing attribute.
* The sandbox setup is one branch on `is_agentic` rather than two tests of
`grader_type`, which also drops a duplicated condition.
No behaviour change: HumanEval stays at 164/164, and `ty check` on
examples/llama-eval is clean.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* feat: API key
* fix reasoning and response length for agentic runs, add option to save entire transcripts
* llama-eval : fix ty unresolved-attribute on loaded_dataset.ae
Assisted-by: AI coding agent
* llama-eval : trim comments and remove non-ASCII from README
Assisted-by: AI coding agent
* gguf-py : add numpy Q1_0 and Q2_0 de/quantization support
* cont: added q1_0 / q2_0 to the test list removing the small test
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
---------
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
* CUDA: HIP: rms_norm: load float4 at a time and store x in registers
* Add template paramter comments
* use ggml_cuda_is_aligned for src
* nits
* also use ggml_cuda_is_aligned for src in fused variants
* fix nits
* tests : update the Metal fusion baseline
Re-record the MTL fusion baseline with the current build: 276 rows over 131
dummy models.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : fail test-fusion when a baseline row stops matching
The report was built from the counters the run produced, and the baseline was
only ever used as a lookup table, so a fusion that stopped matching produced no
row and was never compared. qwen4exp and mellum lost their fusions this way
while the Fusion / metal job stayed green.
Report a baseline row that the run does not reproduce as a regression. Scope
the pass to the arches that were actually run so --model keeps working.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : never write inline comments unless the user asks
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : keep the mellum dummy model softmax-gated
mellum reads expert_gating_func from the GGUF since #30578, while the fixture
writes sigmoid for every MoE arch, so the Mellum dummy model stopped building
the softmax MoE routing fusion. Write softmax for mellum, matching the arch
default and the GGUFs that conversion/mellum.py produces.
Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : use emoji in the AI stats disclosure
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : list the disclosure model after the hardware
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : use monochrome symbols in the AI stats disclosure
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : add muse-spark to the Metal fusion baseline
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round
Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors
* hex-get-rows: vectorize raw k-quant ukernels
* hex-binary: refactor dma-loop thread functions (no need for macros, etc)
* hex-quant: move/cleanup flat and tiled dequant ukernels
* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights
* hex-get-rows: remove redundant cast
* hex-repack: proper filtering for BF16 that were leaking through during the tests
* hex-repack: streamling tiling rules (broken MM for now)
* hex-repack: fix mmid after tiling updates
* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars
* hex-mm: fix valid_rows calc in various kernels
* hex-mm: remove unused hvx_mm_nx_2d_flat
* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16
* hex-mm: another pass at hvx-tiled f16 matmul
* hex-get-rows: optimize small src1
* hex-get-rows: one more pass at recovering get-rows perf regression
* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py
* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul
* hex-mm: fold tiled_f16 checks into the general flow
* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op
* hex-quant: proper tail handling in new dequant ukernels
* hex-get-rows: remove unused code and cleanup support checks
* hex-mm: cleanup ne1 padding/tiling checks
* hex-mm: cleanup fusion checks for tiled tensors
* hex-repack: add a little can_repack helper to reduce duplication
* hex-repack: never repack host buffers
* hex-repack: factor out repack functions into separate .h and .cpp
* hex-tensor: cleanup tensor ifaces for tiled/padded sizes
* hex-repack: update k-quant tiling comments and cleanup #defines
* hex-dcache: factor out l2/dcache dirty tracking and flushing
* hex-tensor: update docs and remove #defs duplication
* hex-tensor: remove unused/incorrect get_data interface
* hex-get-rows: cleanup vtcm layout build
* tests: disable huge MM tests
* hex-dcache: add missing header
The scheduler uploads each view of a host input on its own, so the per-layer reshape of new_pool_idxs (and of sel_mask in glm5-next) was copied once per layer.
* enable OPENSSL=ON as default in build script, update it in CI and dockerfile
* change to LLAMA_BUILD_BORINGSSL on windows
* refactor
* rm Download UI build
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model: add Muse Spark Open (text, vision and audio)
Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.
Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.
The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Muse Spark: drop the redundant forward expand
* Muse Spark: drop the redundant forward expand and tighten the comments
* Update src/llama-quant.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-graph.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update conversion/muse_spark.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* pi : expand AI disclosure with hardware and stats
Prefer PI_HARDWARE for the hardware field, spell out the disclosure block
template (model link, harness, hardware, stats) and source [STATS] from
~/.pi/stats.sh. Record the hardware in the Assisted-by commit tag.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* pi : drop square brackets from placeholders
Brackets inside a markdown link text break the link on GitHub. Describe the
disclosure bullets in words instead of embedding a bracketed template.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Small helper around the GitHub Actions API: lists the workflow runs that are not
finished yet (in_progress, queued, pending, waiting, requested), grouped by status.
Supports selecting runs by status, by branch (server-side, to reduce API traffic)
or by a regex on the branch name, and can cancel the selected runs via --kill.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
The first evaluated token attends to the positions above
pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the
whole window, but the reuse threshold asked for two more cells.
A saved SWA state keeps only the cells inside its window, so a
state read back from a slot file or from the RAM prompt cache was
always rejected and the slot rolled back to a checkpoint.
The SWA threshold now matches the window, while a state without SWA
keeps the exact bound. The test harness gains override_kv, used to
give the SWA test model a window smaller than the test context.
[no release]
The slot file held the target context and the context checkpoints,
but never the draft context, so a restored slot drafted with an
empty or stale draft KV until the cache refilled.
The file now ends with the live draft state, the full draft context
and the speculative state, in the same appendix style as the
checkpoints. A draft state that does not fit the draft context is
dropped and the draft starts empty, while older files and older
servers keep working.
[no release]
Slot selection takes free slots by id or by last use only, so child
tasks often land on a slot holding a parent state while empty slots
are idle, and overwrite it without saving it to the prompt cache.
Free slots are now ordered empty first, then least recently used, for
both the LRU fallback and the child slots, and a child task saves the
state it overwrites. With 4 grouped decision requests in flight and
12 states in rotation, prompt cache reads go from 5 to 36.
[no release]
The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG
said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's
`|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10
turns. Reference the shared constant, as the adjacent MCP timeout does.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* Add Support to maion-coder arch
* iAdd Support to support Draft models
* Address comments
* Remove maion context shift function
---------
Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
tojson wrote nan, inf and -inf for these floats, as values and as dict
keys; json.dumps writes NaN, Infinity and -Infinity. The string and safe
filters of a float keep nan and inf, as Python's str() does, so they no
longer use the tojson function. float_to_json is renamed float_repr, as
it now gives str(); float_json gives the json.dumps text.
Also say in a comment why a last digit 9 is not bumped at a power of 2.
* vulkan : fix concurrent first device init
* tests : load models in parallel in test-thread-safety
Loading them serially never exercised concurrent first backend init, so the crash reported in #29764 was invisible to this test.
* vulkan : create fence before retaining device in buffer context
* replace oneAPI package by DLE+oneDNN package
* support to install oneDNN in windows, in release and docker
* for internal test
* remove test code
* Update intel.Dockerfile
aiqubits
Piotr Wilkin (ilintar)
Jhen-Jie HongandMax Krasnyansky
Mikhail Podvitskii
Aaron Teo
Michael Taylor
Sh1m13
uvos
hayder
FINAL-Bench
Trivalik
Neo Zhang
Titaniumtown
Georgi Gerganov
Ruben Ortlam
Karthik VijayanandKarthik Vijayan
Aman Gupta
Gaurav Garg
Frosty40
Xuan-Son Nguyen
Nigel JonesandGabe Goodhart
Matt Rutkowski
Said Taghadouini
Kartik Gulia
Aleksander Grygier
William Tan
Yuri Khrustalev
losewayy
lhezandHongqiang Wang
Adrien Gallouët
Simone IacominoandSigbjørn Skjæret