master
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Piotr Wilkin (ilintar)
|
f3363593f4 |
Add more benchmarks to llama-eval (#26174)
* llama-eval : add sandboxed execution for code suites Adds a lightweight replacement for the Docker containers usually used to run generated code, so a benchmark can be executed without a daemon, an image or root. A cached venv is provisioned per suite and reused; candidate programs run in a throwaway working directory as their own process group, under the strongest isolation the host offers: bwrap read-only root, tmpfs /tmp and $HOME, no network unshare no network, via --map-current-user rlimit cpu/memory/file-size/process caps only Candidate code must not run as uid 0, which is why "unshare -r" is a last-resort tier that warns: root bypasses file permission bits, so a test asserting that a write to a chmod 0444 file fails would instead see it succeed. Two details worth noting for the bwrap tier. Only paths hidden by the $HOME tmpfs are bound back, because bwrap cannot mount onto a symlink and version-manager alias directories usually are one. And the venv's base interpreter tree is resolved by walking the symlink chain, since a version-managed interpreter is commonly reached through an unversioned alias that startup resolves through. Suites can declare provisioning steps, which run once at build time while the network is still up, so test time never needs it. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * llama-eval : add HumanEval and ClassEval code suites Adds two execution-graded suites alongside the existing text-graded ones. The model's reply is parsed for a python code block, combined with the task's own unit tests, and run in the sandbox. --dataset humaneval 164 function-level problems, scored pass@k --dataset classeval 100 class-level tasks, scored at class level (feeding pass@k) and method level, matching upstream pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the dataset size gives that many samples per problem, aggregated with the unbiased estimator from the Codex paper. Program construction and result interpretation are dataset-owned hooks, because the two suites differ: one HumanEval problem is a function judged by a single assert, whereas one ClassEval task is a class judged by many independent test methods, and reducing that to pass/fail would throw away partial credit. Suites also declare their own package set, provisioning steps, interpreter and timeout. The package set is derived from the dataset's own imports rather than a requirements file. HumanEval turns out to need no third-party packages at all; ClassEval needs thirteen, two of which must be declared because no import mentions them -- lxml is selected by string in ClassEval_44, and scipy is a gensim requirement. Notes on ClassEval specifically: * gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two pure-Python APIs it uses are provided by a small shim instead. * The three nltk corpora it needs are pre-seeded at venv build time. These are the modern names -- the dataset asks for the since-renamed punkt and averaged_perceptron_tagger. * It defaults to a patched fork, ilintar/ClassEval, because three upstream tests cannot be passed by anything: a hard-coded 2024 date checked against datetime.now(), an assertion on the dataset author's own hostname, and an exact float comparison. Reference solutions go from 96/100 to 100/100 classes and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the model is asked to write is unchanged; --dataset-source selects the original. --exec-seed (default 0) seeds the RNG before each test, so suites whose tests draw random inputs score reproducibly. ClassEval_58 fails its own reference solution about 30% of the time without it. The simulator gains both suites, so the harness can be exercised without a model. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * llama-eval : fix silent isolation downgrade, generalise the sandbox runner The isolation probe ran the interpreter without binding it back, so whenever this tool is itself started from a venv under $HOME -- the usual case, since it needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in the output looked wrong when that happened, and the existing suites have been running that way. The probe now uses the same interpreter and the same binds the suite will actually get. Two additions the agentic suite needs, neither of which changes the existing suites: * run_argv() runs an arbitrary command in a caller-owned directory that survives the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take exactly the same path as before. A multi-step agent needs its edits to persist across many invocations, and a suite for a language other than Python needs to drive a linter and a test runner rather than a single source string. * RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved address space rather than resident memory, which is the wrong instrument for a runtime that reserves a large region and commits little of it: node's TypeScript type-stripping goes through WebAssembly, reserves gigabytes per instance, and aborts at 4 GiB with an out-of-memory error that reads like a fault in the code under test. HumanEval stays at 164/164 and ClassEval at 100/100. Assisted-by: Claude Opus 5 <noreply@anthropic.com> * llama-eval : add SACB, a multi-turn agentic coding suite `--dataset agentic` is a different shape from the other code suites. Rather than answering a single prompt, the model is given a small but real repository, a narrow tool set and a bug report, drives its own multi-turn conversation, and is graded by running a test suite it was never allowed to see. The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three hand-written repositories in Python and TypeScript, hosted at ilintar/SACB. Grading needs no container: a cached venv gives ruff and mypy for Python, and node's own test runner plus tsc for TypeScript, so the whole dependency set is four packages. Three constraints are deliberate: * No shell, and no way to run the tests. The only feedback channel is `lint`. A syntax or type error can be driven out mechanically; a logic error has to be reasoned about. A harness that lets an agent iterate against the test suite measures something closer to search than to understanding. * Two edit tools, one line-addressed and one content-addressed. Which one a model reaches for, and whether it keeps line numbers straight after its own earlier edit, is itself a signal. * The tests never touch the working tree. They are held outside it and copied into a throwaway copy only once the agent has stopped, so they can be neither read nor edited. Every tool path is resolved and confinement-checked against the repository root, so a symlink planted inside the tree cannot reach out of it. A task counts as resolved when every test that was failing passes and nothing that already passed broke. Both sets are derived by running the tests against the defective tree and against the reference fix, never written by hand, so a task whose defect no test exercises cannot enter the corpus. Many tasks carry several independent defects, so the fraction of target tests passed is reported alongside, separating "fixed two of the three faults" from "changed nothing". An episode that arrives with no tool call and no `finish` is nudged rather than ended. Models routinely narrate their analysis mid-task, and treating that as completion ends the episode with no edits and scores a reasoning failure that never happened; about one episode in four exits that way. Nothing here is installed unless this suite is selected, and then only for the language being run: agentic_eval is imported lazily and each toolchain is keyed per language. Assisted-by: Claude Opus 5 <noreply@anthropic.com> * llama-eval : make the agentic hooks typed rather than discovered `ty` flagged nine unresolved attributes, of which the seven in main() predate the agentic suite: `EvalState.dataset` is Optional and was dereferenced without narrowing. * `loaded_dataset` turns the Optional into a value in one place, raising rather than handing back None, so callers do not each have to narrow it. * `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were reached through getattr with a default at three call sites, which no checker can see through; `run()` now raises a clear error naming the suite instead of failing as a missing attribute. * The sandbox setup is one branch on `is_agentic` rather than two tests of `grader_type`, which also drops a duplicated condition. No behaviour change: HumanEval stays at 164/164, and `ty check` on examples/llama-eval is clean. Assisted-by: Claude Opus 5 <noreply@anthropic.com> * feat: API key * fix reasoning and response length for agentic runs, add option to save entire transcripts * llama-eval : fix ty unresolved-attribute on loaded_dataset.ae Assisted-by: AI coding agent * llama-eval : trim comments and remove non-ASCII from README Assisted-by: AI coding agent |
||
 Georgi Gerganov
|
da46e59cbf |
llama-eval : fix crash when answer is None in HTML dump (#25435)
dict.get("key", default) returns None (not default) when the key
exists but its value is explicitly None. This caused an AttributeError
in _escape_html() when a task errored before grading and answer was
set to None.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
|
||
|
Georgi Gerganov
|
d2e179a477 |
llama-eval : add per-task summary stats (#23151)
* llama-eval : add per-problem summary table to HTML reports
- Add chunk_idx and problem_idx to TaskState and saved case dicts
- Group completed cases by problem_idx in dump_html()
- Render per-problem summary table before individual task table
- Columns: Problem (zero-padded), Runs, Correct (n/r),
Tokens (min/avg/max), T/s (min/avg/max), Gen s (min/avg/max)
- Sorted by problem index, monospace font, right-aligned numbers
- Colspan headers for grouped stats, auto width
- Simulator: add /v1/models endpoint, timings in response,
template-aware question matching, --dataset arg (aime/aime2025)
Assisted-by: llama.cpp:local pi
* llama-eval : add tabs for Detailed and Summary tables, apply monospace font globally
- Wrap Detailed and Summary tables in switchable tabs (Detailed active by default)
- Remove summary-section wrapper, use tab labels instead
- Apply monospace font to all tables and the top bar
Assisted-by: llama.cpp:local pi
* llama-eval : redesign top bar as CSS grid label/value pairs
- Replace flat span list with 4-column grid layout (2 pairs per row)
- Labels in muted color (#888), values in dark (#222)
- Bold dataset name and model name
- Removed media query, always uses 4 columns
Assisted-by: llama.cpp:local pi
* llama-eval : use realistic token counts and throughput in simulator
- comp_tokens: [30, 80] → [10000, 60000]
- tps_gen: derived → uniform [90.0, 110.0]
- t_gen_ms: now computed from tokens/tps
Assisted-by: llama.cpp:local pi
* llama-eval : color Answer column green/red based on correctness
Use the same .correct/.incorrect CSS classes on the Answer column
to make correct answers green and incorrect answers red.
Assisted-by: llama.cpp:local pi
* llama-eval : fix pyright errors from max(..., key=len) type inference
Use key=lambda x: len(x) instead of key=len so the type checker
infers the return type as str instead of Sized, fixing:
- unresolved-attribute: Object of type Sized has no attribute lower
- not-subscriptable: Cannot subscript object of type Sized
Assisted-by: llama.cpp:local pi
|
||
|
Georgi Gerganov
|
d5dc2e0a02 |
llama-eval : add AIME 2026 dataset support (#23058)
Add Aime2026Dataset class loading from MathArena/aime_2026 on HuggingFace. 30 problems (two sets of 15), single config/split. Usage: --dataset aime2026 Assisted-by: llama.cpp:local pi |
||
 Sigbjørn Skjæret
|
bcfe63fc53 | llama-eval : enable type check (#22988) | ||
 Georgi Gerganovandgatbontonpc
|
fde69a3607 |
examples : add llama-eval (#21152)
* working llama-eval mc and math suite * multi source llama-eval * Add readme * add checkpointing * examples: add llama-server simulator for testing eval scripts Add a standalone Python script that simulates a llama-server HTTP endpoint for testing the eval script. The simulator: - Implements /v1/chat/completions endpoint with OpenAI-compatible format - Loads AIME dataset from HuggingFace with local caching - Uses Levenshtein distance for intelligent question matching - Supports configurable success rate for correct/wrong answer generation - Provides debug logging for troubleshooting Also includes test scripts and documentation for testing and understanding the simulator functionality. * examples: refactor test-simulator.sh for better readability Extract repeating question string into TEST_QUESTION variable and create make_request() helper function to reduce code duplication. Add proper error handling for error responses. * docs: update llama-eval-discussion.md with session work summary Add summary of llama-server-simulator implementation work including features, testing results, technical decisions, and refactoring. * examples: add simplified llama-eval-new.py for AIME evaluation - Create new simplified evaluation script focused only on AIME - Implement EvalState and Processor dataclasses for structured state management - Add real-time feedback showing correct/incorrect status per case - Abstract grading interface for external grader support - Use structured JSON output for eval state - Apply HuggingFace dataset caching to avoid repeated downloads - Remove Levenshtein matching - eval script only sends requests and validates answers * docs: remove README.md from llama-eval * examples: implement flexible grader system for answer validation - Add Grader class supporting regex and CLI-based grading - Implement built-in regex patterns for AIME, GSM8K, MMLU, HellaSwag, ARC, WinoGrande - Add CLI grader interface: python script.py --answer <pred> --expected <gold> - Add HF telemetry disable to avoid warnings - Support exact match requirement for regex patterns - Add 30-second timeout for CLI grader - Handle both boxed and plain text formats for AIME answers * examples: use HF_HUB_OFFLINE to avoid HF Hub warnings * examples: remove HF_HUB_OFFLINE to allow dataset download * examples: use cached dataset path to avoid HF Hub requests * examples: use cached dataset path in simulator to avoid HF Hub requests * docs: update llama-eval-discussion.md with session work summary * examples: add threading support and model parameter to llama-eval-new.py - Add ThreadPoolExecutor for parallel request processing controlled by --threads - Add --model argument to specify model name in request data - Refactor process() to use thread-safe _process_single_case() method - Update progress tracking to work with concurrent execution * docs: update llama-eval-discussion.md with threading and model parameter updates - Add threading support implementation details - Document ThreadPoolExecutor usage and thread safety - Add model parameter implementation details - Include testing results for both features * examples: add task summary table to llama-eval-new.py * eval : print progress * eval : add prompts * test : fix path * sim : fix answer matching * eval : support multiple dataset runs * minor * improve grader * docs * remove old files * datasets : add gsm8k * add gpqa + sampling + docs * rename * grader : improve example answers * cont * datasets : add aime2025 * grader : update prompt * grade : improve regex + logs * datasets : fix aime2025 * cleanup * add AGENTS.md * ignore errors * resume eval * cleanup * fix counts * simplify * fix prompts * add html * store full response * add tokens * resoning and error handling * refactor * track total time * remove junk * eval : unify "judge" terminology to "grader" Replace all occurrences of "judge" with "grader" for consistency across the codebase (CLI args, Grader class fields, help text). Assisted-by: llama.cpp:local pi * eval : add Wilson score confidence interval to results Compute 95% CI on-the-fly from completed cases. Displayed in terminal output, HTML report, and JSON state. * llama-eval : add per-task generation speed from server timings Extract predicted_per_second from the server timings response and store it as tps_gen per task. Display in console progress, print_all_tasks, and HTML report. Assisted-by: llama.cpp:local pi * llama-eval : add per-task generation time from server timings Extract predicted_ms from the server timings response and store it as t_gen_ms per task. Display in seconds with one decimal digit in console progress, print_all_tasks, and HTML report. Assisted-by: llama.cpp:local pi * llama-eval : rename display, escaped, and count variables to use prefix convention - _display suffix → display_ prefix (answer, tokens, tps, t_gen) - _escaped suffix → escaped_ prefix (response, prompt, reasoning) - _count suffix → n_ prefix (correct, incorrect, pending) Assisted-by: llama.cpp:local pi * llama-eval : support multiple evaluation endpoints with dynamic task distribution - Add ServerConfig dataclass (url, threads, name) - Accept comma-separated --server, --threads, --server-name CLI args - Dynamic shared-queue task distribution across servers (fast servers do more work) - One ThreadPoolExecutor per server, workers pull from shared Queue - Track which server processed each task (server_name in results) - Thread-safe EvalState with threading.Lock for concurrent mutations - Server column in HTML report and console output - Backward compatible: single server works as before Assisted-by: llama.cpp:local pi * llama-server-simulator : replace Flask with stdlib http.server - Use HTTPServer + BaseHTTPRequestHandler instead of Flask - RequestHandler handles POST /v1/chat/completions - Server runs in daemon thread with clean Ctrl+C shutdown - Remove flask and unused asdict imports Assisted-by: llama.cpp:local pi * llama-eval : update README with PR link and quick-start examples Assisted-by: llama.cpp:local pi * llama-eval : track model name in eval state and verify on resume - Store model_name in EvalState and JSON output - Display model in HTML summary table - Verify --model matches stored model when resuming Assisted-by: llama.cpp:local pi * llama-server-simulator : fix comment - Dice coefficient, not Levenshtein Assisted-by: llama.cpp:local pi * llama-eval : require --grader-model or --model when using --grader-type llm Assisted-by: llama.cpp:local pi * llama-eval : protect dump() with lock for thread safety Assisted-by: llama.cpp:local pi * llama-eval : compact HTML report output - Replace verbose summary table with single inline bar - Shorten status text: '✓'/'✗'/'–'/'!' instead of full words - Flatten CSS: remove box-shadows, border-radius, reduce padding - Use system-ui font, 13px table, 12px details - Conditional reasoning section (only shown when present) - Single toggle JS function instead of two - Shorter column headers Assisted-by: llama.cpp:local pi * llama-eval : check server connectivity on startup - Hit /v1/models for each server before evaluation - Exit with error if any server is unreachable - Print comma-separated model IDs per server in startup output - Sequential checks, no retries, no timeout override Assisted-by: llama.cpp:local pi * llama-eval : use server1/server2 instead of gpu1/gpu2 in README Assisted-by: llama.cpp:local pi --------- Co-authored-by: gatbontonpc <gatbontonpc@gmail.com> |