Add more benchmarks to llama-eval (#26174)
* llama-eval : add sandboxed execution for code suites Adds a lightweight replacement for the Docker containers usually used to run generated code, so a benchmark can be executed without a daemon, an image or root. A cached venv is provisioned per suite and reused; candidate programs run in a throwaway working directory as their own process group, under the strongest isolation the host offers: bwrap read-only root, tmpfs /tmp and $HOME, no network unshare no network, via --map-current-user rlimit cpu/memory/file-size/process caps only Candidate code must not run as uid 0, which is why "unshare -r" is a last-resort tier that warns: root bypasses file permission bits, so a test asserting that a write to a chmod 0444 file fails would instead see it succeed. Two details worth noting for the bwrap tier. Only paths hidden by the $HOME tmpfs are bound back, because bwrap cannot mount onto a symlink and version-manager alias directories usually are one. And the venv's base interpreter tree is resolved by walking the symlink chain, since a version-managed interpreter is commonly reached through an unversioned alias that startup resolves through. Suites can declare provisioning steps, which run once at build time while the network is still up, so test time never needs it. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * llama-eval : add HumanEval and ClassEval code suites Adds two execution-graded suites alongside the existing text-graded ones. The model's reply is parsed for a python code block, combined with the task's own unit tests, and run in the sandbox. --dataset humaneval 164 function-level problems, scored pass@k --dataset classeval 100 class-level tasks, scored at class level (feeding pass@k) and method level, matching upstream pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the dataset size gives that many samples per problem, aggregated with the unbiased estimator from the Codex paper. Program construction and result interpretation are dataset-owned hooks, because the two suites differ: one HumanEval problem is a function judged by a single assert, whereas one ClassEval task is a class judged by many independent test methods, and reducing that to pass/fail would throw away partial credit. Suites also declare their own package set, provisioning steps, interpreter and timeout. The package set is derived from the dataset's own imports rather than a requirements file. HumanEval turns out to need no third-party packages at all; ClassEval needs thirteen, two of which must be declared because no import mentions them -- lxml is selected by string in ClassEval_44, and scipy is a gensim requirement. Notes on ClassEval specifically: * gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two pure-Python APIs it uses are provided by a small shim instead. * The three nltk corpora it needs are pre-seeded at venv build time. These are the modern names -- the dataset asks for the since-renamed punkt and averaged_perceptron_tagger. * It defaults to a patched fork, ilintar/ClassEval, because three upstream tests cannot be passed by anything: a hard-coded 2024 date checked against datetime.now(), an assertion on the dataset author's own hostname, and an exact float comparison. Reference solutions go from 96/100 to 100/100 classes and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the model is asked to write is unchanged; --dataset-source selects the original. --exec-seed (default 0) seeds the RNG before each test, so suites whose tests draw random inputs score reproducibly. ClassEval_58 fails its own reference solution about 30% of the time without it. The simulator gains both suites, so the harness can be exercised without a model. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com> * llama-eval : fix silent isolation downgrade, generalise the sandbox runner The isolation probe ran the interpreter without binding it back, so whenever this tool is itself started from a venv under $HOME -- the usual case, since it needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in the output looked wrong when that happened, and the existing suites have been running that way. The probe now uses the same interpreter and the same binds the suite will actually get. Two additions the agentic suite needs, neither of which changes the existing suites: * run_argv() runs an arbitrary command in a caller-owned directory that survives the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take exactly the same path as before. A multi-step agent needs its edits to persist across many invocations, and a suite for a language other than Python needs to drive a linter and a test runner rather than a single source string. * RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved address space rather than resident memory, which is the wrong instrument for a runtime that reserves a large region and commits little of it: node's TypeScript type-stripping goes through WebAssembly, reserves gigabytes per instance, and aborts at 4 GiB with an out-of-memory error that reads like a fault in the code under test. HumanEval stays at 164/164 and ClassEval at 100/100. Assisted-by: Claude Opus 5 <noreply@anthropic.com> * llama-eval : add SACB, a multi-turn agentic coding suite `--dataset agentic` is a different shape from the other code suites. Rather than answering a single prompt, the model is given a small but real repository, a narrow tool set and a bug report, drives its own multi-turn conversation, and is graded by running a test suite it was never allowed to see. The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three hand-written repositories in Python and TypeScript, hosted at ilintar/SACB. Grading needs no container: a cached venv gives ruff and mypy for Python, and node's own test runner plus tsc for TypeScript, so the whole dependency set is four packages. Three constraints are deliberate: * No shell, and no way to run the tests. The only feedback channel is `lint`. A syntax or type error can be driven out mechanically; a logic error has to be reasoned about. A harness that lets an agent iterate against the test suite measures something closer to search than to understanding. * Two edit tools, one line-addressed and one content-addressed. Which one a model reaches for, and whether it keeps line numbers straight after its own earlier edit, is itself a signal. * The tests never touch the working tree. They are held outside it and copied into a throwaway copy only once the agent has stopped, so they can be neither read nor edited. Every tool path is resolved and confinement-checked against the repository root, so a symlink planted inside the tree cannot reach out of it. A task counts as resolved when every test that was failing passes and nothing that already passed broke. Both sets are derived by running the tests against the defective tree and against the reference fix, never written by hand, so a task whose defect no test exercises cannot enter the corpus. Many tasks carry several independent defects, so the fraction of target tests passed is reported alongside, separating "fixed two of the three faults" from "changed nothing". An episode that arrives with no tool call and no `finish` is nudged rather than ended. Models routinely narrate their analysis mid-task, and treating that as completion ends the episode with no edits and scores a reasoning failure that never happened; about one episode in four exits that way. Nothing here is installed unless this suite is selected, and then only for the language being run: agentic_eval is imported lazily and each toolchain is keyed per language. Assisted-by: Claude Opus 5 <noreply@anthropic.com> * llama-eval : make the agentic hooks typed rather than discovered `ty` flagged nine unresolved attributes, of which the seven in main() predate the agentic suite: `EvalState.dataset` is Optional and was dereferenced without narrowing. * `loaded_dataset` turns the Optional into a value in one place, raising rather than handing back None, so callers do not each have to narrow it. * `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were reached through getattr with a default at three call sites, which no checker can see through; `run()` now raises a clear error naming the suite instead of failing as a missing attribute. * The sandbox setup is one branch on `is_agentic` rather than two tests of `grader_type`, which also drops a duplicated condition. No behaviour change: HumanEval stays at 164/164, and `ty check` on examples/llama-eval is clean. Assisted-by: Claude Opus 5 <noreply@anthropic.com> * feat: API key * fix reasoning and response length for agentic runs, add option to save entire transcripts * llama-eval : fix ty unresolved-attribute on loaded_dataset.ae Assisted-by: AI coding agent * llama-eval : trim comments and remove non-ASCII from README Assisted-by: AI coding agent
This commit is contained in:
Piotr Wilkin (ilintar) authored and 
GitHub committed
1 parent
ceefd16daa
commit
f3363593f4
5 files changed
+2993
-40
No files matched your search
@@ -24,3 +24,267 @@ python3 llama-eval.py \
|
||||
--dataset aime2025 --n_cases 240 \
|
||||
--grader-type regex
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
A server started with `--api-key` (or any OpenAI-compatible endpoint behind a
|
||||
token) needs one here too. `--api-key` is sent as `Authorization: Bearer` on
|
||||
every eval request; it falls back to `$LLAMA_API_KEY`, which keeps the secret
|
||||
out of your shell history and off the process list.
|
||||
|
||||
```bash
|
||||
# One key for every server
|
||||
python3 llama-eval.py --server http://localhost:8033 --api-key sk-... ...
|
||||
|
||||
# From the environment
|
||||
export LLAMA_API_KEY=sk-...
|
||||
python3 llama-eval.py --server http://localhost:8033 ...
|
||||
|
||||
# One key per server, in --server order
|
||||
python3 llama-eval.py \
|
||||
--server http://server1:8033,http://server2:8033 \
|
||||
--api-key sk-one,sk-two --threads 16,16 ...
|
||||
```
|
||||
|
||||
The LLM grader authenticates too. It reuses the key of the server it runs on,
|
||||
so `--grader-type llm` without `--grader-server` needs nothing extra; a grader
|
||||
on a separate endpoint takes `--grader-api-key`.
|
||||
|
||||
Endpoints on an opencode host additionally get an `x-opencode-session` header,
|
||||
generated fresh for every task. An agentic episode sends the same id on all of
|
||||
its turns, so opencode sees one session per task instead of one per request;
|
||||
other servers never see the header.
|
||||
|
||||
## Agentic suite
|
||||
|
||||
`--dataset agentic` is a different shape from the other code suites. It runs
|
||||
**SACB** (Simple Agent Coding Benchmark), hosted at
|
||||
[`ilintar/SACB`](https://huggingface.co/datasets/ilintar/SACB) -- 60 tasks over
|
||||
three hand-written repositories, in Python and TypeScript. Rather than
|
||||
answering one prompt, the model is given a small but real repository, a
|
||||
deliberately narrow tool set, and a bug report, and drives its own multi-turn
|
||||
conversation until it declares itself finished. It is then graded by running a
|
||||
test suite it was never allowed to see.
|
||||
|
||||
```bash
|
||||
python3 llama-eval.py \
|
||||
--server http://localhost:8033 --model my-model \
|
||||
--dataset agentic --threads 1 --temperature 0
|
||||
```
|
||||
|
||||
### The tools
|
||||
|
||||
| tool | purpose |
|
||||
|---|---|
|
||||
| `list_files` | enumerate the tree |
|
||||
| `read_file` | read, with 1-based line numbers and optional line-range narrowing |
|
||||
| `search` | regular-expression search across the repository |
|
||||
| `edit_lines` | replace an inclusive line range |
|
||||
| `edit_replace` | replace an exact snippet, which must be unique |
|
||||
| `write_file` | write a whole file |
|
||||
| `lint` | run the language's linter and type checker |
|
||||
| `finish` | declare the change complete |
|
||||
|
||||
Three constraints are deliberate:
|
||||
|
||||
**There is no shell and no way to run the tests.** The only feedback channel is
|
||||
`lint`. A syntax or type error can be driven out mechanically; a logic error has
|
||||
to be reasoned about. A harness that lets an agent iterate against the test
|
||||
suite measures something closer to search than to understanding.
|
||||
|
||||
**There are two edit tools**, one line-addressed and one content-addressed.
|
||||
Which one a model reaches for, and whether it keeps line numbers straight after
|
||||
its own earlier edit, is itself a signal.
|
||||
|
||||
**The tests never touch the working tree.** They are held outside it and copied
|
||||
into a throwaway copy only after the agent has stopped, so they can be neither
|
||||
read nor edited. Every tool path is resolved and confinement-checked against the
|
||||
repository root, so a symlink planted inside the tree cannot reach outside it.
|
||||
|
||||
### Scoring
|
||||
|
||||
A task is **resolved** when every test that was failing now passes *and* nothing
|
||||
that was already passing broke. Both sets are derived by running the tests twice
|
||||
when the corpus is built -- once against the defective tree and once against the
|
||||
reference fix -- never hand-written, so a task whose defect no test exercises is
|
||||
rejected rather than shipped.
|
||||
|
||||
Because many tasks contain several independent defects, the fraction of target
|
||||
tests passed is reported alongside, which separates "fixed two of the three
|
||||
faults" from "changed nothing". Process metrics are reported too: turns, tool
|
||||
calls, rejected calls, edits, and the peak conversation size.
|
||||
|
||||
### Dependencies
|
||||
|
||||
Nothing here is installed unless you select this suite, and then only for the
|
||||
language you actually run. `--agentic-lang python` provisions ruff and mypy;
|
||||
`--agentic-lang typescript` provisions the TypeScript compiler and uses node's
|
||||
own test runner. Neither touches the other, and no other suite touches either.
|
||||
|
||||
### Useful flags
|
||||
|
||||
| flag | meaning |
|
||||
|---|---|
|
||||
| `--agentic-lang` | restrict to one language, and provision only its toolchain |
|
||||
| `--agentic-max-turns` | give up after this many assistant turns (default 50) |
|
||||
| `--agentic-max-tokens` | cap generation per turn (default: no cap); bounds a model that falls into a repetition loop, but also cuts reasoning off mid-thought. Truncated turns are counted as `truncated_turns` |
|
||||
| `--agentic-context-limit` | abandon an episode once its context exceeds this many tokens |
|
||||
| `--agentic-transcripts DIR` | save each episode's full message history to `DIR/<task_id>.json`, plus a `DIR/<task_id>.html` showing every reasoning block, reply, tool call and tool result; the main report links to it |
|
||||
|
||||
|
||||
## Code suites (HumanEval, ClassEval)
|
||||
|
||||
`--dataset humaneval` runs the 164 OpenAI HumanEval problems; `--dataset
|
||||
classeval` runs the 100 ClassEval class-generation tasks. Unlike the other
|
||||
suites these are not graded by matching text: the model's reply is parsed for a
|
||||
```` ```python ```` block, that code is combined with the task's own unit tests,
|
||||
and the result is **executed**.
|
||||
|
||||
ClassEval asks for a whole class rather than one function, so it is scored the
|
||||
way upstream does - **class level** (every one of the task's tests passes, which
|
||||
is what feeds pass@k) and **method level** (what fraction of all 2196 test
|
||||
methods passed). The second number separates "generated nothing usable" from
|
||||
"got most of the class right", and both are printed at the end of a run.
|
||||
|
||||
Give it more room to generate than HumanEval - a class with ten methods does not
|
||||
fit in a short budget:
|
||||
|
||||
```bash
|
||||
python3 llama-eval.py \
|
||||
--server http://localhost:8033 --model my-model \
|
||||
--dataset classeval --threads 8 --n_predict 4096 --temperature 0
|
||||
```
|
||||
|
||||
```bash
|
||||
# pass@1, greedy
|
||||
python3 llama-eval.py \
|
||||
--server http://localhost:8033 --model my-model \
|
||||
--dataset humaneval --threads 8 --n_predict 1024 --temperature 0
|
||||
|
||||
# pass@10 -- each extra multiple of 164 is another sample per problem
|
||||
python3 llama-eval.py \
|
||||
--server http://localhost:8033 --model my-model \
|
||||
--dataset humaneval --n_cases 1640 --temperature 0.8 --top-p 0.95
|
||||
```
|
||||
|
||||
`--grader-type exec` is the default for code suites, so neither it nor
|
||||
`--dataset-source` normally needs to be passed.
|
||||
|
||||
### Execution sandbox - no Docker required
|
||||
|
||||
Generated code is untrusted, but these suites need far less than a container to
|
||||
run safely. **The package set is derived from the dataset's own imports at
|
||||
startup**, not hardcoded and not taken from the upstream `requirements.txt` -
|
||||
which for ClassEval lists `torch`, `openai` and `transformers` that its *tests*
|
||||
never touch. So each suite installs exactly what it needs:
|
||||
|
||||
| suite | third-party packages |
|
||||
|---|---|
|
||||
| HumanEval | **none** - every problem imports only stdlib (`typing, math, random, copy, string, collections, hashlib, re`) |
|
||||
| ClassEval | 13: `beautifulsoup4, gensim, lxml, netifaces, nltk, numpy, openpyxl, pandas, pillow, pypdf2, python-docx, reportlab, scipy` |
|
||||
|
||||
Two of ClassEval's are *declared* rather than derived, because no import
|
||||
statement mentions them: `lxml` is selected by string
|
||||
(`BeautifulSoup(html, 'lxml')`, ClassEval_44 - without it that task silently
|
||||
drops from 23/23 to 7/23), and `scipy` is a hard gensim requirement. Note also
|
||||
that `PyPDF2` must **not** be modernised to `pypdf`: ClassEval_69's test imports
|
||||
`PdfFileReader`, a name pypdf removed.
|
||||
|
||||
The venv is cached under `~/.cache/llama-eval/` (override with
|
||||
`LLAMA_EVAL_CACHE`) and rebuilt only when the package set, interpreter version
|
||||
or provisioning steps change; force one with `--rebuild-sandbox`. `uv` is used
|
||||
when available, which makes even the ClassEval venv build in seconds.
|
||||
|
||||
**Interpreter selection.** A suite can prefer a different Python than the one
|
||||
running the tool, and declares it: ClassEval asks for 3.13, because gensim
|
||||
publishes no wheel past cp313 and its generated C no longer compiles on 3.14
|
||||
(it still uses `PyLongObject.ob_digit`). That interpreter is found automatically,
|
||||
including via `uv python`; `--sandbox-python` overrides it.
|
||||
|
||||
3.14 is fully supported anyway. ClassEval uses exactly two gensim APIs
|
||||
(`utils.decode_htmlentities`, `matutils.unitvec`), both pure Python upstream, so
|
||||
on 3.14+ the suite drops gensim from the install set and provisions a small shim
|
||||
instead. Both paths score **100/100**.
|
||||
|
||||
**Offline provisioning.** A suite that would otherwise fetch data at test time
|
||||
declares a provisioning step that runs once at venv build time, while the
|
||||
network is still up. ClassEval uses this to pre-seed exactly three nltk corpora
|
||||
(`punkt_tab`, `averaged_perceptron_tagger_eng`, `wordnet`, ~33 MB), so the tests
|
||||
themselves run with no network at all. Those are the *current* names - the
|
||||
dataset asks for `punkt` and `averaged_perceptron_tagger`, which nltk has since
|
||||
renamed, and seeding the legacy names too is not harmless: with a network
|
||||
available nltk considers them satisfied and re-fetches the modern ones anyway.
|
||||
The corpora are written *inside* the venv, not `~/nltk_data`, because `$HOME` is
|
||||
a tmpfs under the strongest isolation tier.
|
||||
|
||||
Each candidate program runs in a throwaway directory as its own process group,
|
||||
under the strongest isolation the host offers:
|
||||
|
||||
| tier | requires | isolates |
|
||||
|---|---|---|
|
||||
| `bwrap` | bubblewrap installed | read-only root, tmpfs `/tmp` and `$HOME`, **no network** |
|
||||
| `unshare` | `unshare --map-current-user` | **no network** |
|
||||
| `unshare-root` | any `unshare -r` | no network, but **runs as uid 0** - see below |
|
||||
| `rlimit` | always available | cpu/memory/file-size/process caps only |
|
||||
|
||||
**Candidate code must not run as root**, which is why `unshare -r` is a
|
||||
last-resort tier that prints a warning. Root bypasses file permission bits, so a
|
||||
test asserting that writing to a `chmod 0444` file *fails* instead sees it
|
||||
succeed - ClassEval_50 scores 14/16 under uid 0 and 16/16 under uid 1000, with
|
||||
nothing in the output to suggest the sandbox caused it. Install bubblewrap.
|
||||
|
||||
All tiers additionally cap CPU time, address space (4 GiB), file size and core
|
||||
dumps, and kill the whole process group on timeout (`--exec-timeout`, default
|
||||
15 s - raise it for ClassEval, whose tests do real work). Writes to the
|
||||
throwaway working directory are allowed, which is all the file-handling tasks in
|
||||
ClassEval need.
|
||||
|
||||
Note that the weakest tier does not contain filesystem access. If you are
|
||||
scoring output from a model you do not trust, check the tier printed at startup.
|
||||
|
||||
### Reproducibility
|
||||
|
||||
Several tasks draw their test inputs from the *global, unseeded* RNG, so a
|
||||
verdict is not reproducible run to run - which adds noise exactly when you are
|
||||
comparing two models or two quantisations. `--exec-seed` (default `0`) seeds the
|
||||
RNG before each test; pass `--exec-seed -1` for the stock upstream behaviour.
|
||||
This does not change what the model is asked to compute.
|
||||
|
||||
It matters more than it sounds. HumanEval/38, /50 and /53 are affected, and
|
||||
ClassEval_58 (duplicate mine placement) fails about 30% of the time on its own
|
||||
*reference* solution - measured here at 21/30 passes unseeded versus **30/30
|
||||
seeded**.
|
||||
|
||||
### Dataset fidelity
|
||||
|
||||
The canonical dataset runs **unmodified on current Python** - all 164 reference
|
||||
solutions pass on CPython 3.14 with no patches, so no forked or re-hosted copy
|
||||
is needed. `--dataset-source` accepts a local `.jsonl` or a HuggingFace repo id
|
||||
if you do want to point at a modified copy.
|
||||
|
||||
HumanEval does contain a handful of long-standing defects where the *prompt*
|
||||
contradicts the graded test (notably HumanEval/47, /116 and /148). Those
|
||||
penalise a model that reads the docstring carefully, but correcting them makes
|
||||
the benchmark marginally easier and the resulting scores no longer comparable
|
||||
with published HumanEval numbers, so nothing is patched by default.
|
||||
|
||||
ClassEval is different: the upstream data has defects that cap the achievable
|
||||
score below 100% for reasons unrelated to the model, so `--dataset classeval`
|
||||
defaults to a **patched fork**,
|
||||
[`ilintar/ClassEval`](https://huggingface.co/datasets/ilintar/ClassEval).
|
||||
Six edits over five tasks lift the reference solutions from 96/100 classes and
|
||||
2182/2196 test methods to **100/100 and 2196/2196**. Three repair tests nothing
|
||||
could pass:
|
||||
|
||||
- **ClassEval_17** was a time bomb - a hard-coded 2024 date checked against
|
||||
`datetime.now()`, unpassable since January 2024.
|
||||
- **ClassEval_48** asserted `get_hostname('0.0.0.0') == 'LAPTOP-2CS86KUM'`, the
|
||||
dataset author's own machine name.
|
||||
- **ClassEval_31** compared floats with `assertEqual`, failing on a 4-ULP numpy
|
||||
difference.
|
||||
|
||||
The other three fix only the *reference* solutions (NumPy 2.0 removals in
|
||||
ClassEval_51, a 30%-flaky mine generator in ClassEval_58) and cannot affect a
|
||||
model's score. No `skeleton` was touched, so what the model is asked to write is
|
||||
byte-identical to upstream. Pass `--dataset-source FudanSELab/ClassEval` for the
|
||||
unpatched original.
|
||||
@@ -0,0 +1,918 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Multi-turn agentic coding eval: tools over a real working tree.
|
||||
|
||||
The existing code suites are single-shot -- one prompt, one reply, execute,
|
||||
grade. This one is not. The model is given a small but real repository and a
|
||||
deliberately narrow tool set, and has to find its way to the defect before it
|
||||
can fix it. That shape is what puts a run in the tens-of-thousands-of-tokens
|
||||
range without any artificial padding: the context grows because the model
|
||||
chose to read things.
|
||||
|
||||
Three deliberate constraints:
|
||||
|
||||
* No shell, and no way to run the tests. The only feedback channel is `lint`.
|
||||
A syntax or type error can be driven out mechanically; a logic error has to
|
||||
be reasoned about. Suites that let an agent iterate against the test suite
|
||||
measure something closer to search than to understanding.
|
||||
* Two edit tools, one line-addressed and one content-addressed. Which one a
|
||||
model reaches for, and whether it keeps line numbers straight after its own
|
||||
earlier edit, is itself a signal.
|
||||
* Tests never touch the working tree. They are held outside it and copied in
|
||||
after the agent has stopped, so they can be neither read nor edited.
|
||||
|
||||
Everything here is imported lazily by llama-eval.py, and every dependency it
|
||||
needs is scoped to the language actually under test, so none of it is
|
||||
installed for anyone running the other suites.
|
||||
"""
|
||||
|
||||
import html
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
MAX_READ_LINES = 400
|
||||
MAX_TOOL_CHARS = 12000
|
||||
MAX_SEARCH_HITS = 60
|
||||
|
||||
|
||||
@dataclass
|
||||
class LangSpec:
|
||||
"""Everything language-specific: how to lint, how to test, what to install.
|
||||
|
||||
Kept as data so adding a language is a corpus concern rather than a harness
|
||||
change, and so a run only ever provisions the toolchain for the language it
|
||||
is actually evaluating.
|
||||
"""
|
||||
name: str
|
||||
exts: Tuple[str, ...]
|
||||
packages: Tuple[str, ...] = ()
|
||||
provision: Tuple[str, ...] = ()
|
||||
env: Dict[str, str] = field(default_factory=dict)
|
||||
address_space_limit: Optional[int] = 4 * 1024 * 1024 * 1024
|
||||
|
||||
def source_files(self, root: Path) -> List[Path]:
|
||||
return sorted(p for p in root.rglob("*")
|
||||
if p.is_file() and p.suffix in self.exts)
|
||||
|
||||
|
||||
PYTHON = LangSpec(
|
||||
name="python",
|
||||
exts=(".py",),
|
||||
packages=("ruff==0.14.2", "mypy==1.18.2"),
|
||||
)
|
||||
|
||||
TYPESCRIPT = LangSpec(
|
||||
name="typescript",
|
||||
exts=(".ts",),
|
||||
address_space_limit=16 * 1024 * 1024 * 1024,
|
||||
provision=(
|
||||
"import os, subprocess, shutil, sys\n"
|
||||
"npm = shutil.which('npm')\n"
|
||||
"if not npm: sys.exit('npm not found on PATH; needed for the typescript suite')\n"
|
||||
"dest = os.path.join(os.environ['SANDBOX_VENV'], 'node')\n"
|
||||
"os.makedirs(dest, exist_ok=True)\n"
|
||||
"r = subprocess.run([npm, 'install', '--silent', '--no-audit', '--no-fund',\n"
|
||||
" '--prefix', dest, 'typescript@5.9.3'],\n"
|
||||
" capture_output=True, text=True)\n"
|
||||
"sys.exit(0 if r.returncode == 0 else (r.stderr or r.stdout)[-2000:])\n",
|
||||
),
|
||||
)
|
||||
|
||||
LANGS: Dict[str, LangSpec] = {"python": PYTHON, "typescript": TYPESCRIPT}
|
||||
|
||||
|
||||
class PathEscape(Exception):
|
||||
"""A tool was handed a path outside the repository."""
|
||||
|
||||
|
||||
class Workspace:
|
||||
"""A private, writable copy of one task's repository.
|
||||
|
||||
Every tool path is resolved through here. Confinement is checked on the
|
||||
fully-resolved path, so a symlink planted inside the tree cannot be used to
|
||||
read or write outside it.
|
||||
"""
|
||||
|
||||
def __init__(self, root: Path):
|
||||
self.root = root.resolve()
|
||||
self.escape_attempts = 0
|
||||
|
||||
def resolve(self, rel: str) -> Path:
|
||||
if rel is None or not str(rel).strip():
|
||||
raise PathEscape("no path given")
|
||||
raw = Path(str(rel))
|
||||
base = raw if raw.is_absolute() else (self.root / raw)
|
||||
candidate = base.resolve()
|
||||
if candidate != self.root and self.root not in candidate.parents:
|
||||
self.escape_attempts += 1
|
||||
raise PathEscape(f"path {rel!r} is outside the repository")
|
||||
return candidate
|
||||
|
||||
def rel(self, path: Path) -> str:
|
||||
return str(path.relative_to(self.root))
|
||||
|
||||
def files(self) -> List[Path]:
|
||||
return sorted(p for p in self.root.rglob("*")
|
||||
if p.is_file() and "__pycache__" not in p.parts)
|
||||
|
||||
def snapshot(self) -> Dict[str, str]:
|
||||
out = {}
|
||||
for p in self.files():
|
||||
try:
|
||||
out[self.rel(p)] = p.read_text()
|
||||
except (UnicodeDecodeError, OSError):
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def _clip(text: str, limit: int = MAX_TOOL_CHARS) -> str:
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[:limit] + f"\n... [truncated, {len(text) - limit} more characters]"
|
||||
|
||||
|
||||
def _numbered(lines: Sequence[str], start: int) -> str:
|
||||
width = len(str(start + len(lines) - 1))
|
||||
return "\n".join(f"{start + i:>{width}}\t{ln}" for i, ln in enumerate(lines))
|
||||
|
||||
|
||||
def tool_schemas() -> List[Dict[str, Any]]:
|
||||
"""OpenAI-format function schemas for the whole tool set."""
|
||||
def fn(name, desc, props, required=()):
|
||||
return {"type": "function", "function": {
|
||||
"name": name, "description": desc,
|
||||
"parameters": {"type": "object", "properties": props,
|
||||
"required": list(required)}}}
|
||||
|
||||
return [
|
||||
fn("list_files", "List files in the repository.",
|
||||
{"path": {"type": "string",
|
||||
"description": "Directory relative to the repository root. "
|
||||
"Omit for the whole tree."}}),
|
||||
fn("read_file",
|
||||
"Read a file. Output is prefixed with 1-based line numbers. Give "
|
||||
"start_line/end_line to read only part of a large file.",
|
||||
{"path": {"type": "string", "description": "File path relative to the repository root."},
|
||||
"start_line": {"type": "integer", "description": "First line to read, 1-based, inclusive."},
|
||||
"end_line": {"type": "integer", "description": "Last line to read, 1-based, inclusive."}},
|
||||
["path"]),
|
||||
fn("search",
|
||||
"Search the repository with a regular expression. Returns matching "
|
||||
"lines with their file and line number.",
|
||||
{"pattern": {"type": "string", "description": "Python regular expression."},
|
||||
"path": {"type": "string", "description": "Limit the search to this subdirectory."},
|
||||
"glob": {"type": "string", "description": "Limit to files matching this glob, e.g. '*.py'."}},
|
||||
["pattern"]),
|
||||
fn("edit_lines",
|
||||
"Replace an inclusive 1-based line range with new text. Line numbers "
|
||||
"refer to the file as it is now, so re-read it after any edit that "
|
||||
"changed the number of lines.",
|
||||
{"path": {"type": "string", "description": "File path relative to the repository root."},
|
||||
"start_line": {"type": "integer", "description": "First line to replace, 1-based, inclusive."},
|
||||
"end_line": {"type": "integer", "description": "Last line to replace, 1-based, inclusive."},
|
||||
"new_text": {"type": "string", "description": "Replacement text. Use an empty string to delete the range."}},
|
||||
["path", "start_line", "end_line", "new_text"]),
|
||||
fn("edit_replace",
|
||||
"Replace an exact snippet of text. The snippet must occur exactly "
|
||||
"once in the file unless replace_all is set.",
|
||||
{"path": {"type": "string", "description": "File path relative to the repository root."},
|
||||
"old_text": {"type": "string", "description": "Exact text to find, including indentation."},
|
||||
"new_text": {"type": "string", "description": "Replacement text."},
|
||||
"replace_all": {"type": "boolean", "description": "Replace every occurrence instead of requiring exactly one."}},
|
||||
["path", "old_text", "new_text"]),
|
||||
fn("write_file",
|
||||
"Write a whole file, creating or overwriting it. Prefer the edit "
|
||||
"tools for changes to existing files.",
|
||||
{"path": {"type": "string", "description": "File path relative to the repository root."},
|
||||
"content": {"type": "string", "description": "Full new file contents."}},
|
||||
["path", "content"]),
|
||||
fn("lint",
|
||||
"Run the language's linter and type checker over the repository and "
|
||||
"return the diagnostics. This is the only feedback available; the "
|
||||
"test suite cannot be run.",
|
||||
{"path": {"type": "string", "description": "Limit to this file or directory."}}),
|
||||
fn("finish",
|
||||
"Declare the task complete and end the session. This is the only "
|
||||
"way to end it; call this once the change is made.",
|
||||
{"summary": {"type": "string", "description": "Brief description of the change made."}}),
|
||||
]
|
||||
|
||||
|
||||
class ToolBox:
|
||||
"""Executes tool calls against one workspace. Returns plain text results.
|
||||
|
||||
Failures are reported as ordinary text rather than raised, because how a
|
||||
model recovers from a rejected edit is part of what is being measured.
|
||||
"""
|
||||
|
||||
def __init__(self, ws: Workspace, linter=None):
|
||||
self.ws = ws
|
||||
self.linter = linter
|
||||
self.calls = 0
|
||||
self.errors = 0
|
||||
self.edits = 0
|
||||
self.finished = False
|
||||
self.finish_summary: Optional[str] = None
|
||||
self.per_tool: Dict[str, int] = {}
|
||||
|
||||
def dispatch(self, name: str, args: Dict[str, Any]) -> str:
|
||||
self.calls += 1
|
||||
self.per_tool[name] = self.per_tool.get(name, 0) + 1
|
||||
handler = getattr(self, f"_t_{name}", None)
|
||||
if handler is None:
|
||||
result = f"Error: unknown tool {name!r}."
|
||||
else:
|
||||
try:
|
||||
result = handler(args)
|
||||
except PathEscape as e:
|
||||
result = f"Error: {e}"
|
||||
except KeyError as e:
|
||||
result = f"Error: missing required argument {e}."
|
||||
except Exception as e:
|
||||
result = f"Error: {type(e).__name__}: {e}"
|
||||
if result.startswith("Error:"):
|
||||
self.errors += 1
|
||||
return _clip(result)
|
||||
|
||||
def _t_list_files(self, a):
|
||||
base = self.ws.resolve(a.get("path") or ".")
|
||||
if not base.exists():
|
||||
return f"Error: {a.get('path')!r} does not exist."
|
||||
paths = [p for p in self.ws.files() if p == base or base in p.parents]
|
||||
if not paths:
|
||||
return "(no files)"
|
||||
return "\n".join(f"{self.ws.rel(p)}\t{p.stat().st_size} bytes" for p in paths)
|
||||
|
||||
def _t_read_file(self, a):
|
||||
path = self.ws.resolve(a["path"])
|
||||
if not path.is_file():
|
||||
return f"Error: {a['path']!r} is not a file."
|
||||
lines = path.read_text().splitlines()
|
||||
start = max(1, int(a.get("start_line") or 1))
|
||||
end = int(a.get("end_line") or len(lines))
|
||||
end = min(end, len(lines))
|
||||
if start > len(lines):
|
||||
return f"Error: {a['path']!r} has only {len(lines)} lines."
|
||||
window = lines[start - 1:end]
|
||||
note = ""
|
||||
if len(window) > MAX_READ_LINES:
|
||||
window = window[:MAX_READ_LINES]
|
||||
note = (f"\n... [showing {MAX_READ_LINES} of {end - start + 1} requested "
|
||||
f"lines; re-read with start_line/end_line for the rest]")
|
||||
header = f"{a['path']} ({len(lines)} lines total)\n"
|
||||
return header + _numbered(window, start) + note
|
||||
|
||||
def _t_search(self, a):
|
||||
try:
|
||||
rx = re.compile(a["pattern"])
|
||||
except re.error as e:
|
||||
return f"Error: bad regular expression: {e}"
|
||||
base = self.ws.resolve(a.get("path") or ".")
|
||||
glob = a.get("glob")
|
||||
hits, scanned = [], 0
|
||||
for p in self.ws.files():
|
||||
if not (p == base or base in p.parents):
|
||||
continue
|
||||
if glob and not p.match(glob):
|
||||
continue
|
||||
try:
|
||||
text = p.read_text()
|
||||
except (UnicodeDecodeError, OSError):
|
||||
continue
|
||||
scanned += 1
|
||||
for i, line in enumerate(text.splitlines(), 1):
|
||||
if rx.search(line):
|
||||
hits.append(f"{self.ws.rel(p)}:{i}:{line.strip()[:200]}")
|
||||
if len(hits) >= MAX_SEARCH_HITS:
|
||||
return ("\n".join(hits) +
|
||||
f"\n... [stopped at {MAX_SEARCH_HITS} matches; narrow the pattern]")
|
||||
if not hits:
|
||||
return f"No matches for {a['pattern']!r} in {scanned} file(s)."
|
||||
return "\n".join(hits)
|
||||
|
||||
def _t_edit_lines(self, a):
|
||||
path = self.ws.resolve(a["path"])
|
||||
if not path.is_file():
|
||||
return f"Error: {a['path']!r} is not a file."
|
||||
lines = path.read_text().splitlines()
|
||||
start, end = int(a["start_line"]), int(a["end_line"])
|
||||
if start < 1 or start > len(lines):
|
||||
return (f"Error: start_line {start} out of range; "
|
||||
f"{a['path']!r} has {len(lines)} lines.")
|
||||
if end < start or end > len(lines):
|
||||
return (f"Error: end_line {end} out of range; must be between "
|
||||
f"{start} and {len(lines)}.")
|
||||
new = str(a["new_text"]).splitlines()
|
||||
path.write_text("\n".join(lines[:start - 1] + new + lines[end:]) + "\n")
|
||||
self.edits += 1
|
||||
delta = len(new) - (end - start + 1)
|
||||
return (f"Replaced lines {start}-{end} of {a['path']} "
|
||||
f"({end - start + 1} -> {len(new)} lines, {delta:+d}). "
|
||||
f"Later line numbers have shifted." if delta else
|
||||
f"Replaced lines {start}-{end} of {a['path']}.")
|
||||
|
||||
def _t_edit_replace(self, a):
|
||||
path = self.ws.resolve(a["path"])
|
||||
if not path.is_file():
|
||||
return f"Error: {a['path']!r} is not a file."
|
||||
text = path.read_text()
|
||||
old, new = str(a["old_text"]), str(a["new_text"])
|
||||
if not old:
|
||||
return "Error: old_text must not be empty."
|
||||
n = text.count(old)
|
||||
if n == 0:
|
||||
return (f"Error: old_text not found in {a['path']}. It must match "
|
||||
f"exactly, including indentation.")
|
||||
if n > 1 and not a.get("replace_all"):
|
||||
return (f"Error: old_text occurs {n} times in {a['path']}. Include "
|
||||
f"more surrounding context to make it unique, or set "
|
||||
f"replace_all.")
|
||||
path.write_text(text.replace(old, new))
|
||||
self.edits += 1
|
||||
return f"Replaced {n} occurrence(s) in {a['path']}."
|
||||
|
||||
def _t_write_file(self, a):
|
||||
path = self.ws.resolve(a["path"])
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
existed = path.is_file()
|
||||
path.write_text(str(a["content"]))
|
||||
self.edits += 1
|
||||
return f"{'Overwrote' if existed else 'Created'} {a['path']}."
|
||||
|
||||
def _t_lint(self, a):
|
||||
if self.linter is None:
|
||||
return "Error: no linter is configured for this task."
|
||||
target = a.get("path")
|
||||
if target:
|
||||
self.ws.resolve(target)
|
||||
return self.linter(target)
|
||||
|
||||
def _t_finish(self, a):
|
||||
self.finished = True
|
||||
self.finish_summary = a.get("summary")
|
||||
return "Task marked complete."
|
||||
|
||||
|
||||
PY_TEST_RUNNER = r'''
|
||||
import json, os, random, sys, unittest
|
||||
sys.path.insert(0, os.getcwd())
|
||||
random.seed(0)
|
||||
|
||||
class Collect(unittest.TestResult):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.outcomes = {}
|
||||
def addSuccess(self, t): self.outcomes[t.id()] = "pass"
|
||||
def addFailure(self, t, e): self.outcomes[t.id()] = "fail"
|
||||
def addError(self, t, e): self.outcomes[t.id()] = "error"
|
||||
def addSkip(self, t, r): self.outcomes[t.id()] = "skip"
|
||||
def addExpectedFailure(self, t, e): self.outcomes[t.id()] = "pass"
|
||||
def addUnexpectedSuccess(self, t): self.outcomes[t.id()] = "fail"
|
||||
|
||||
def flatten(s):
|
||||
for t in s:
|
||||
if isinstance(t, unittest.TestSuite):
|
||||
yield from flatten(t)
|
||||
else:
|
||||
yield t
|
||||
|
||||
out = {"tests": {}}
|
||||
try:
|
||||
suite = unittest.TestLoader().discover(os.environ.get("TEST_DIR", "tests"),
|
||||
top_level_dir=".")
|
||||
res = Collect()
|
||||
for t in flatten(suite):
|
||||
tid = t.id()
|
||||
try:
|
||||
t.run(res)
|
||||
except Exception:
|
||||
pass
|
||||
res.outcomes.setdefault(tid, "error")
|
||||
out["tests"] = res.outcomes
|
||||
except Exception as e:
|
||||
out["error"] = f"{type(e).__name__}: {e}"
|
||||
|
||||
print("###RESULT###")
|
||||
print(json.dumps(out))
|
||||
'''
|
||||
|
||||
|
||||
def _node_roots() -> List[Path]:
|
||||
"""Directories that must stay visible for node to start under bwrap."""
|
||||
node = shutil.which("node")
|
||||
if not node:
|
||||
return []
|
||||
real = Path(node).resolve()
|
||||
return [Path(node).parent.parent, real.parent.parent]
|
||||
|
||||
|
||||
class Runner:
|
||||
"""Runs the linter and the hidden tests for one language, in the sandbox."""
|
||||
|
||||
def __init__(self, sandbox, lang: LangSpec, timeout: float = 90.0):
|
||||
self.sandbox = sandbox
|
||||
self.lang = lang
|
||||
self.timeout = timeout
|
||||
|
||||
def _binds(self) -> List[Path]:
|
||||
return _node_roots() if self.lang.name == "typescript" else []
|
||||
|
||||
def _env(self) -> Dict[str, str]:
|
||||
if self.lang.name != "typescript":
|
||||
return {}
|
||||
node = shutil.which("node")
|
||||
return {"PATH": f"{Path(node).parent}:/usr/bin:/bin"} if node else {}
|
||||
|
||||
def lint(self, workdir: Path, target: Optional[str] = None) -> str:
|
||||
if self.lang.name == "python":
|
||||
return self._lint_python(workdir, target)
|
||||
return self._lint_typescript(workdir, target)
|
||||
|
||||
def _lint_python(self, workdir: Path, target: Optional[str]) -> str:
|
||||
venv = self.sandbox.venv_dir
|
||||
tgt = target or "."
|
||||
chunks = []
|
||||
r = self.sandbox.run_argv(
|
||||
[str(venv / "bin" / "ruff"), "check", "--no-cache",
|
||||
"--output-format", "concise", tgt],
|
||||
workdir, timeout=self.timeout)
|
||||
text = (r.stdout or r.stderr).strip()
|
||||
if text and "All checks passed" not in text:
|
||||
chunks.append(text)
|
||||
r = self.sandbox.run_argv(
|
||||
[str(venv / "bin" / "mypy"), "--no-color-output", "--no-error-summary",
|
||||
"--cache-dir", "/dev/null", "--ignore-missing-imports", tgt],
|
||||
workdir, timeout=self.timeout)
|
||||
text = (r.stdout or r.stderr).strip()
|
||||
if text and not text.startswith("Success:"):
|
||||
chunks.append(text)
|
||||
return "\n".join(chunks) if chunks else "No problems found."
|
||||
|
||||
def _lint_typescript(self, workdir: Path, target: Optional[str]) -> str:
|
||||
tsc = self.sandbox.venv_dir / "node" / "node_modules" / "typescript" / "bin" / "tsc"
|
||||
files = [str(p.relative_to(workdir)) for p in sorted(workdir.rglob("*.ts"))
|
||||
if "node_modules" not in p.parts]
|
||||
if target:
|
||||
t = (workdir / target).resolve()
|
||||
files = [f for f in files if (workdir / f).resolve() == t
|
||||
or t in (workdir / f).resolve().parents]
|
||||
if not files:
|
||||
return "No TypeScript files to check."
|
||||
r = self.sandbox.run_argv(
|
||||
["node", str(tsc), "--noEmit", "--strict", "--target", "es2022",
|
||||
"--module", "esnext", "--moduleResolution", "bundler",
|
||||
"--allowImportingTsExtensions", *files],
|
||||
workdir, timeout=self.timeout, env=self._env(), ro_binds=self._binds())
|
||||
out = (r.stdout or "") + (r.stderr or "")
|
||||
return out.strip() or "No problems found."
|
||||
|
||||
def test(self, workdir: Path) -> Dict[str, Any]:
|
||||
if self.lang.name == "python":
|
||||
return self._test_python(workdir)
|
||||
return self._test_typescript(workdir)
|
||||
|
||||
def _test_python(self, workdir: Path) -> Dict[str, Any]:
|
||||
runner = workdir / "__run_tests.py"
|
||||
runner.write_text(PY_TEST_RUNNER)
|
||||
r = self.sandbox.run_argv(
|
||||
[str(self.sandbox.python), "-I", "-B", str(runner)],
|
||||
workdir, timeout=self.timeout, env={"TEST_DIR": "tests"})
|
||||
return self._parse_json_result(r)
|
||||
|
||||
def _parse_json_result(self, r) -> Dict[str, Any]:
|
||||
if r.status == "timeout":
|
||||
return {"tests": {}, "error": "timeout"}
|
||||
marker = "###RESULT###"
|
||||
if marker not in (r.stdout or ""):
|
||||
return {"tests": {},
|
||||
"error": f"no result: {(r.stderr or r.stdout or '')[-400:]}"}
|
||||
try:
|
||||
return json.loads(r.stdout.split(marker, 1)[1].strip())
|
||||
except Exception as e:
|
||||
return {"tests": {}, "error": f"unparseable result: {e}"}
|
||||
|
||||
def _test_typescript(self, workdir: Path) -> Dict[str, Any]:
|
||||
tests = sorted(p for p in (workdir / "tests").rglob("*.ts")) \
|
||||
if (workdir / "tests").is_dir() else []
|
||||
if not tests:
|
||||
return {"tests": {}, "error": "no test files"}
|
||||
r = self.sandbox.run_argv(
|
||||
["node", "--test", "--experimental-strip-types",
|
||||
"--test-reporter=tap",
|
||||
*[str(p.relative_to(workdir)) for p in tests]],
|
||||
workdir, timeout=self.timeout, env=self._env(), ro_binds=self._binds())
|
||||
return self._parse_tap(r)
|
||||
|
||||
@staticmethod
|
||||
def _parse_tap(r) -> Dict[str, Any]:
|
||||
if r.status == "timeout":
|
||||
return {"tests": {}, "error": "timeout"}
|
||||
text = (r.stdout or "") + "\n" + (r.stderr or "")
|
||||
outcomes: Dict[str, str] = {}
|
||||
for line in text.splitlines():
|
||||
m = re.match(r"\s*(not )?ok\s+\d+\s*-\s*(.+?)\s*$", line)
|
||||
if not m:
|
||||
continue
|
||||
name = m.group(2).strip()
|
||||
if name.endswith(" # SKIP") or " # SKIP" in name:
|
||||
outcomes[name.split(" # ")[0]] = "skip"
|
||||
continue
|
||||
verdict = "fail" if m.group(1) else "pass"
|
||||
if outcomes.get(name) != "fail":
|
||||
outcomes[name] = verdict
|
||||
if not outcomes:
|
||||
return {"tests": {},
|
||||
"error": f"no TAP output: {text.strip()[-400:]}"}
|
||||
return {"tests": outcomes}
|
||||
|
||||
|
||||
SYSTEM_PROMPT = """You are an expert {lang} engineer working in an existing repository.
|
||||
|
||||
Resolve the user's request by editing files with the tools provided.
|
||||
|
||||
How this session works:
|
||||
- You are working autonomously. There is no human watching; nobody will answer
|
||||
questions, approve a plan, or read progress reports.
|
||||
- Every reply you send must contain at least one tool call until the task is
|
||||
done. A reply that contains only text does nothing: no work happens and the
|
||||
turn is wasted. Do not stop to describe what you are about to do; do it.
|
||||
- The session ends only when you call `finish`. Call it once the change is
|
||||
made, and not before. Only the files as you leave them are evaluated.
|
||||
|
||||
Important constraints:
|
||||
- You cannot run the code, the tests, or any shell command. The only feedback
|
||||
available is the `lint` tool, which reports syntax and type problems.
|
||||
- The repository already has a test suite that will be run against your work
|
||||
after you finish, but you cannot see it or run it. Do not create test files.
|
||||
- Read enough of the code to understand it before editing. A change that only
|
||||
makes the symptom go away will usually fail the tests.
|
||||
- Make the smallest correct change. Do not reformat or refactor unrelated code.
|
||||
|
||||
Keep calling tools until the change is complete, then call `finish`."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgenticTask:
|
||||
task_id: str
|
||||
lang: str
|
||||
category: str
|
||||
difficulty: int
|
||||
instruction: str
|
||||
files: Dict[str, str]
|
||||
tests: Dict[str, str]
|
||||
gold: Dict[str, str] = field(default_factory=dict)
|
||||
fail_to_pass: List[str] = field(default_factory=list)
|
||||
pass_to_pass: List[str] = field(default_factory=list)
|
||||
|
||||
@classmethod
|
||||
def from_record(cls, rec: Dict[str, Any]) -> "AgenticTask":
|
||||
def as_map(v):
|
||||
return v if isinstance(v, dict) else json.loads(v or "{}")
|
||||
def as_list(v):
|
||||
return v if isinstance(v, list) else json.loads(v or "[]")
|
||||
return cls(
|
||||
task_id=rec["task_id"], lang=rec["lang"], category=rec["category"],
|
||||
difficulty=int(rec.get("difficulty", 3)),
|
||||
instruction=rec["instruction"],
|
||||
files=as_map(rec["files"]), tests=as_map(rec["tests"]),
|
||||
gold=as_map(rec.get("gold")),
|
||||
fail_to_pass=as_list(rec.get("fail_to_pass")),
|
||||
pass_to_pass=as_list(rec.get("pass_to_pass")),
|
||||
)
|
||||
|
||||
@property
|
||||
def spec(self) -> LangSpec:
|
||||
return LANGS[self.lang]
|
||||
|
||||
|
||||
def materialise(files: Dict[str, str], dest: Path) -> Path:
|
||||
"""Write a file map out as a real directory tree."""
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
for rel, content in files.items():
|
||||
p = dest / rel
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(content)
|
||||
return dest
|
||||
|
||||
|
||||
def repo_overview(files: Dict[str, str]) -> str:
|
||||
"""A compact tree, so a run is not lost merely to not finding the files."""
|
||||
lines = []
|
||||
for rel in sorted(files):
|
||||
n = files[rel].count("\n") + 1
|
||||
lines.append(f" {rel} ({n} lines)")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def user_prompt(task: AgenticTask) -> str:
|
||||
return (f"Repository files:\n{repo_overview(task.files)}\n\n"
|
||||
f"Task:\n{task.instruction}")
|
||||
|
||||
|
||||
@dataclass
|
||||
class EpisodeResult:
|
||||
stop_reason: str = "unknown"
|
||||
turns: int = 0
|
||||
tool_calls: int = 0
|
||||
tool_errors: int = 0
|
||||
edits: int = 0
|
||||
escape_attempts: int = 0
|
||||
finished: bool = False
|
||||
bad_tool_json: int = 0
|
||||
nudges: int = 0
|
||||
prompt_tokens: int = 0
|
||||
completion_tokens: int = 0
|
||||
peak_context: int = 0
|
||||
truncated_turns: int = 0
|
||||
per_tool: Dict[str, int] = field(default_factory=dict)
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
NUDGE = ("Your last reply contained no tool call, so nothing happened. There is "
|
||||
"no human in this session to respond to text. If your change is "
|
||||
"complete, call `finish`. Otherwise continue working by calling the "
|
||||
"tools; do not reply with text alone.")
|
||||
|
||||
|
||||
def run_episode(task: AgenticTask, toolbox: ToolBox, chat,
|
||||
max_turns: int = 50,
|
||||
context_limit: Optional[int] = None,
|
||||
max_nudges: int = 3) -> Tuple[EpisodeResult, List[Dict]]:
|
||||
"""Drive one multi-turn tool-calling episode.
|
||||
|
||||
`chat(messages, tools) -> response dict` is supplied by the caller so this
|
||||
module stays free of any HTTP concerns.
|
||||
"""
|
||||
tools = tool_schemas()
|
||||
messages: List[Dict[str, Any]] = [
|
||||
{"role": "system", "content": SYSTEM_PROMPT.format(lang=task.spec.name)},
|
||||
{"role": "user", "content": user_prompt(task)},
|
||||
]
|
||||
res = EpisodeResult()
|
||||
|
||||
for turn in range(max_turns):
|
||||
res.turns = turn + 1
|
||||
try:
|
||||
reply = chat(messages, tools)
|
||||
except Exception as e:
|
||||
res.stop_reason, res.error = "request_failed", f"{type(e).__name__}: {e}"
|
||||
break
|
||||
|
||||
usage = reply.get("usage") or {}
|
||||
pt = int(usage.get("prompt_tokens") or 0)
|
||||
ct = int(usage.get("completion_tokens") or 0)
|
||||
res.prompt_tokens += pt
|
||||
res.completion_tokens += ct
|
||||
res.peak_context = max(res.peak_context, pt + ct)
|
||||
|
||||
choices = reply.get("choices") or []
|
||||
if not choices:
|
||||
res.stop_reason, res.error = "no_choices", json.dumps(reply)[:300]
|
||||
break
|
||||
msg = choices[0].get("message") or {}
|
||||
if choices[0].get("finish_reason") == "length":
|
||||
res.truncated_turns += 1
|
||||
calls = msg.get("tool_calls") or []
|
||||
|
||||
reasoning = msg.get("reasoning_content")
|
||||
messages.append({
|
||||
"role": "assistant",
|
||||
"content": msg.get("content") or "",
|
||||
**({"reasoning_content": reasoning} if reasoning else {}),
|
||||
**({"tool_calls": calls} if calls else {}),
|
||||
})
|
||||
|
||||
if not calls:
|
||||
if toolbox.finished:
|
||||
res.stop_reason = "finished"
|
||||
break
|
||||
if res.nudges >= max_nudges:
|
||||
res.stop_reason = "no_tool_calls"
|
||||
break
|
||||
res.nudges += 1
|
||||
messages.append({"role": "user", "content": NUDGE})
|
||||
continue
|
||||
|
||||
for call in calls:
|
||||
fn = call.get("function") or {}
|
||||
name = fn.get("name") or ""
|
||||
raw = fn.get("arguments")
|
||||
if isinstance(raw, dict):
|
||||
args = raw
|
||||
else:
|
||||
try:
|
||||
args = json.loads(raw or "{}")
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
res.bad_tool_json += 1
|
||||
args = None
|
||||
output = (toolbox.dispatch(name, args) if args is not None else
|
||||
f"Error: arguments for {name!r} were not valid JSON.")
|
||||
messages.append({"role": "tool",
|
||||
"tool_call_id": call.get("id") or f"call_{res.tool_calls}",
|
||||
"content": output})
|
||||
|
||||
if toolbox.finished:
|
||||
res.stop_reason = "finished"
|
||||
break
|
||||
if context_limit and res.peak_context > context_limit:
|
||||
res.stop_reason = "context_limit"
|
||||
break
|
||||
else:
|
||||
res.stop_reason = "max_turns"
|
||||
|
||||
res.tool_calls = toolbox.calls
|
||||
res.tool_errors = toolbox.errors
|
||||
res.edits = toolbox.edits
|
||||
res.escape_attempts = toolbox.ws.escape_attempts
|
||||
res.finished = toolbox.finished
|
||||
res.per_tool = dict(toolbox.per_tool)
|
||||
return res, messages
|
||||
|
||||
|
||||
def run_tests_against(files: Dict[str, str], task: AgenticTask, runner: Runner,
|
||||
workroot: Path, label: str) -> Dict[str, str]:
|
||||
"""Materialise a tree, drop the hidden tests in, and run them.
|
||||
|
||||
The tests are only ever written into this throwaway copy, never into the
|
||||
tree the agent worked in, so there is no window in which they could have
|
||||
been read or edited.
|
||||
"""
|
||||
dest = workroot / f"grade-{label}"
|
||||
shutil.rmtree(dest, ignore_errors=True)
|
||||
materialise(files, dest)
|
||||
materialise(task.tests, dest)
|
||||
return runner.test(dest).get("tests", {})
|
||||
|
||||
|
||||
def score(task: AgenticTask, outcomes: Dict[str, str]) -> Dict[str, Any]:
|
||||
"""Resolved means every target test passes and nothing previously passing broke."""
|
||||
def passing(tid):
|
||||
return outcomes.get(tid) == "pass"
|
||||
|
||||
f2p_ok = [t for t in task.fail_to_pass if passing(t)]
|
||||
p2p_ok = [t for t in task.pass_to_pass if passing(t)]
|
||||
regressions = [t for t in task.pass_to_pass if not passing(t)]
|
||||
resolved = (len(f2p_ok) == len(task.fail_to_pass)
|
||||
and not regressions and bool(task.fail_to_pass))
|
||||
return {
|
||||
"resolved": resolved,
|
||||
"f2p_passed": len(f2p_ok), "f2p_total": len(task.fail_to_pass),
|
||||
"p2p_passed": len(p2p_ok), "p2p_total": len(task.pass_to_pass),
|
||||
"regressions": regressions[:10],
|
||||
"n_regressions": len(regressions),
|
||||
"partial": (len(f2p_ok) / len(task.fail_to_pass)) if task.fail_to_pass else 0.0,
|
||||
}
|
||||
|
||||
|
||||
def make_sandbox(lang_name: str, sandbox_cls, cache_root=None, base_python=None):
|
||||
"""A sandbox carrying only the toolchain for `lang_name`.
|
||||
|
||||
Keyed per language rather than per suite: running the Python tasks must
|
||||
never provision node, and vice versa.
|
||||
"""
|
||||
lang = LANGS[lang_name]
|
||||
return sandbox_cls(
|
||||
name=f"agentic-{lang_name}",
|
||||
packages=list(lang.packages),
|
||||
provision=list(lang.provision),
|
||||
env=dict(lang.env),
|
||||
address_space_limit=lang.address_space_limit,
|
||||
cache_root=cache_root,
|
||||
base_python=base_python,
|
||||
)
|
||||
|
||||
|
||||
def run_task(task: AgenticTask, chat, sandbox, workroot: Optional[Path] = None,
|
||||
max_turns: int = 50, context_limit: Optional[int] = None,
|
||||
test_timeout: float = 90.0) -> Dict[str, Any]:
|
||||
"""One full attempt: fresh workspace, agent episode, then hidden tests."""
|
||||
root = Path(tempfile.mkdtemp(prefix=f"agentic-{task.task_id}-",
|
||||
dir=str(workroot) if workroot else None))
|
||||
try:
|
||||
ws_root = root / "repo"
|
||||
materialise(task.files, ws_root)
|
||||
ws = Workspace(ws_root)
|
||||
runner = Runner(sandbox, task.spec, timeout=test_timeout)
|
||||
toolbox = ToolBox(ws, linter=lambda t: runner.lint(ws_root, t))
|
||||
|
||||
episode, messages = run_episode(task, toolbox, chat,
|
||||
max_turns=max_turns,
|
||||
context_limit=context_limit)
|
||||
final = ws.snapshot()
|
||||
changed = sorted(k for k in final
|
||||
if final[k] != task.files.get(k)) + \
|
||||
sorted(k for k in task.files if k not in final)
|
||||
|
||||
outcomes = run_tests_against(final, task, runner, root, "final")
|
||||
result = score(task, outcomes)
|
||||
result.update({
|
||||
"task_id": task.task_id, "lang": task.lang,
|
||||
"category": task.category, "difficulty": task.difficulty,
|
||||
"changed_files": changed,
|
||||
"episode": {k: v for k, v in vars(episode).items()},
|
||||
"messages": messages,
|
||||
})
|
||||
return result
|
||||
finally:
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
|
||||
|
||||
TRANSCRIPT_CSS = """
|
||||
body { font-family: system-ui, sans-serif; margin: 0; padding: 16px; background: #fff; color: #222; max-width: 1100px; }
|
||||
h1 { font-size: 16px; margin: 0 0 4px; }
|
||||
.meta { font-size: 13px; color: #555; font-family: 'SF Mono', 'Menlo', 'Consolas', monospace; margin-bottom: 12px; white-space: pre-wrap; }
|
||||
.msg { border: 1px solid #e1e4e8; border-left-width: 4px; margin: 8px 0; padding: 6px 10px; background: #fafbfc; }
|
||||
.msg .role { font-size: 12px; font-weight: 600; color: #555; margin-bottom: 4px; }
|
||||
.system { border-left-color: #888; }
|
||||
.user { border-left-color: #0969da; }
|
||||
.nudge { border-left-color: #9a6700; }
|
||||
.assistant { border-left-color: #1a7f37; background: #fff; }
|
||||
.tool { border-left-color: #8250df; }
|
||||
.tool.err { border-left-color: #cf222e; }
|
||||
pre { font-family: 'SF Mono', 'Menlo', 'Consolas', monospace; font-size: 12px; background: #fff; border: 1px solid #e1e4e8; padding: 6px 8px; margin: 4px 0; white-space: pre-wrap; word-wrap: break-word; }
|
||||
.reasoning pre { color: #57606a; background: #f6f8fa; }
|
||||
.call { font-size: 12px; font-weight: 600; color: #8250df; margin-top: 6px; }
|
||||
details > summary { cursor: pointer; font-size: 12px; color: #555; }
|
||||
.empty { font-size: 12px; color: #888; font-style: italic; }
|
||||
"""
|
||||
|
||||
|
||||
def _pre(text: str) -> str:
|
||||
return f"<pre>{html.escape(text)}</pre>"
|
||||
|
||||
|
||||
def _render_call(call: Dict[str, Any]) -> str:
|
||||
fn = call.get("function") or {}
|
||||
raw = fn.get("arguments")
|
||||
try:
|
||||
args = raw if isinstance(raw, dict) else json.loads(raw or "{}")
|
||||
shown = json.dumps(args, indent=2, ensure_ascii=False)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
shown = f"(invalid JSON)\n{raw}"
|
||||
return (f'<div class="call">→ {html.escape(fn.get("name") or "?")} '
|
||||
f'<span style="font-weight:400;color:#888">{html.escape(call.get("id") or "")}</span></div>'
|
||||
+ _pre(shown))
|
||||
|
||||
|
||||
def render_transcript_html(messages: List[Dict[str, Any]], title: str,
|
||||
meta: str = "") -> str:
|
||||
"""One page per episode: every turn, with reasoning, tool calls and results."""
|
||||
names = {}
|
||||
for m in messages:
|
||||
for c in m.get("tool_calls") or []:
|
||||
names[c.get("id")] = (c.get("function") or {}).get("name") or "?"
|
||||
|
||||
parts = []
|
||||
turn = 0
|
||||
for m in messages:
|
||||
role = m.get("role") or "?"
|
||||
content = m.get("content") or ""
|
||||
if role == "assistant":
|
||||
turn += 1
|
||||
body = ""
|
||||
if m.get("reasoning_content"):
|
||||
body += ('<details class="reasoning" open><summary>reasoning '
|
||||
f'({len(m["reasoning_content"])} chars)</summary>'
|
||||
+ _pre(m["reasoning_content"]) + "</details>")
|
||||
body += _pre(content) if content.strip() else ""
|
||||
body += "".join(_render_call(c) for c in m.get("tool_calls") or [])
|
||||
if not body:
|
||||
body = '<div class="empty">(empty reply)</div>'
|
||||
parts.append(f'<div class="msg assistant"><div class="role">assistant '
|
||||
f'· turn {turn}</div>{body}</div>')
|
||||
elif role == "tool":
|
||||
name = names.get(m.get("tool_call_id"), "?")
|
||||
err = " err" if content.startswith("Error") else ""
|
||||
parts.append(f'<div class="msg tool{err}"><div class="role">tool result '
|
||||
f'· {html.escape(name)}</div>{_pre(content)}</div>')
|
||||
elif role == "system":
|
||||
parts.append('<div class="msg system"><details><summary>system prompt'
|
||||
f'</summary>{_pre(content)}</details></div>')
|
||||
else:
|
||||
cls = "nudge" if content == NUDGE else role
|
||||
label = "user (nudge)" if content == NUDGE else role
|
||||
parts.append(f'<div class="msg {html.escape(cls)}"><div class="role">'
|
||||
f'{html.escape(label)}</div>{_pre(content)}</div>')
|
||||
|
||||
return (f'<!DOCTYPE html>\n<html>\n<head>\n<meta charset="UTF-8">\n'
|
||||
f'<title>{html.escape(title)}</title>\n<style>{TRANSCRIPT_CSS}</style>\n'
|
||||
f'</head>\n<body>\n<h1>{html.escape(title)}</h1>\n'
|
||||
f'<div class="meta">{html.escape(meta)}</div>\n'
|
||||
+ "\n".join(parts) + "\n</body>\n</html>\n")
|
||||
|
||||
|
||||
def write_transcript(messages: List[Dict[str, Any]], dest: Path, title: str,
|
||||
meta: str = "") -> Path:
|
||||
"""Write `<dest>.json` (the raw message history) and `<dest>.html`."""
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
json_path = dest.with_name(dest.name + ".json")
|
||||
with open(json_path, "w") as f:
|
||||
json.dump(messages, f, indent=2, ensure_ascii=False)
|
||||
with open(dest.with_name(dest.name + ".html"), "w") as f:
|
||||
f.write(render_transcript_html(messages, title, meta))
|
||||
return json_path
|
||||
@@ -0,0 +1,488 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Lightweight sandboxed execution for code-generation evals (HumanEval, ClassEval).
|
||||
|
||||
Replaces the usual Docker-per-suite setup with a cached venv plus whatever
|
||||
process isolation the host can give us without root:
|
||||
|
||||
tier 1 bubblewrap read-only root, tmpfs /tmp and $HOME, no network
|
||||
tier 2 unshare -rn no network (home still visible)
|
||||
tier 3 rlimits only last resort, still capped on cpu/memory/procs/files
|
||||
|
||||
Every tier also applies rlimits and runs the candidate program in a throwaway
|
||||
working directory as its own process group, so a runaway solution is killed
|
||||
along with anything it spawned.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import resource
|
||||
import shutil
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import venv
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Sequence
|
||||
|
||||
DEFAULT_CACHE_ROOT = Path(
|
||||
os.environ.get("LLAMA_EVAL_CACHE", Path.home() / ".cache" / "llama-eval")
|
||||
)
|
||||
|
||||
LIMIT_AS_BYTES = 4 * 1024 * 1024 * 1024
|
||||
LIMIT_CPU_SECONDS_SLACK = 5
|
||||
LIMIT_NPROC = 256
|
||||
LIMIT_FSIZE_BYTES = 64 * 1024 * 1024
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExecResult:
|
||||
passed: bool
|
||||
status: str
|
||||
stdout: str = ""
|
||||
stderr: str = ""
|
||||
returncode: Optional[int] = None
|
||||
duration: float = 0.0
|
||||
|
||||
def summary(self, max_chars: int = 400) -> str:
|
||||
"""Short human-readable reason, for grader logs."""
|
||||
if self.passed:
|
||||
return "ok"
|
||||
if self.status == "timeout":
|
||||
return "timeout"
|
||||
tail = (self.stderr or self.stdout).strip()
|
||||
if not tail:
|
||||
return f"{self.status} (rc={self.returncode})"
|
||||
lines = [ln for ln in tail.splitlines() if ln.strip()]
|
||||
reason = lines[-1] if lines else tail
|
||||
return reason[:max_chars]
|
||||
|
||||
|
||||
def resolve_python(spec: Optional[str]) -> Optional[str]:
|
||||
"""Turn '3.13' or a path into a usable interpreter, or None if unavailable.
|
||||
|
||||
Suites can need an older interpreter than the one running this tool, because
|
||||
a dependency has no wheel (or no working source build) for the newest
|
||||
CPython. Accepts an explicit path, a bare version, or an executable name.
|
||||
"""
|
||||
if not spec:
|
||||
return None
|
||||
|
||||
candidate = Path(spec).expanduser()
|
||||
if candidate.exists() and candidate.is_file():
|
||||
return str(candidate)
|
||||
|
||||
uv = shutil.which("uv")
|
||||
if uv and re.fullmatch(r"\d+\.\d+(\.\d+)?", spec):
|
||||
try:
|
||||
r = subprocess.run([uv, "python", "find", spec],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
path = r.stdout.strip()
|
||||
if r.returncode == 0 and path and Path(path).exists():
|
||||
return path
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return shutil.which(spec) or shutil.which(f"python{spec}")
|
||||
|
||||
|
||||
def _detect_isolation(probe_python: Optional[str] = None,
|
||||
ro_binds: Sequence[Path] = ()) -> str:
|
||||
"""Pick the strongest isolation tier this host supports, once.
|
||||
|
||||
The probe must run the *same* interpreter the suite will, with the same
|
||||
binds. Probing a bare sys.executable silently downgrades the tier whenever
|
||||
this tool is itself run from a venv under $HOME -- which is the usual case,
|
||||
since it needs `datasets`. bwrap tmpfs-mounts the home directory, the
|
||||
interpreter disappears, the probe fails, and the sandbox falls back to a
|
||||
tier that does not hide $HOME at all. Nothing about the results looks wrong
|
||||
when that happens, so it has to be got right here.
|
||||
"""
|
||||
py = probe_python or sys.executable
|
||||
probe = "import sys; sys.exit(0)"
|
||||
|
||||
if shutil.which("bwrap"):
|
||||
try:
|
||||
r = subprocess.run(
|
||||
_bwrap_argv(Path.cwd(), [py, "-I", "-c", probe], ro_binds=ro_binds),
|
||||
capture_output=True, timeout=20,
|
||||
)
|
||||
if r.returncode == 0:
|
||||
return "bwrap"
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if shutil.which("unshare"):
|
||||
for tier, argv in (
|
||||
("unshare", ["unshare", "--user", "--map-current-user", "--net"]),
|
||||
("unshare-root", ["unshare", "-rn"]),
|
||||
):
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[*argv, py, "-I", "-c", probe],
|
||||
capture_output=True, timeout=20,
|
||||
)
|
||||
if r.returncode == 0:
|
||||
return tier
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return "rlimit"
|
||||
|
||||
|
||||
UNSHARE_ARGV = {
|
||||
"unshare": ["unshare", "--user", "--map-current-user", "--net"],
|
||||
"unshare-root": ["unshare", "-rn"],
|
||||
}
|
||||
|
||||
|
||||
def _under(path: Path, parent: str) -> bool:
|
||||
return bool(parent) and (str(path) == parent
|
||||
or str(path).startswith(parent.rstrip("/") + os.sep))
|
||||
|
||||
|
||||
def _bwrap_argv(workdir: Path, inner: Sequence[str],
|
||||
ro_binds: Sequence[Path] = ()) -> List[str]:
|
||||
home = os.path.expanduser("~")
|
||||
argv = [
|
||||
"bwrap",
|
||||
"--ro-bind", "/", "/",
|
||||
"--dev", "/dev",
|
||||
"--proc", "/proc",
|
||||
"--tmpfs", "/tmp",
|
||||
]
|
||||
if home and home != "/":
|
||||
argv += ["--tmpfs", home]
|
||||
for path in ro_binds:
|
||||
if _under(path, home):
|
||||
argv += ["--ro-bind", str(path), str(path)]
|
||||
argv += [
|
||||
"--bind", str(workdir), str(workdir),
|
||||
"--chdir", str(workdir),
|
||||
"--unshare-all",
|
||||
"--die-with-parent",
|
||||
"--new-session",
|
||||
"--",
|
||||
]
|
||||
return argv + list(inner)
|
||||
|
||||
|
||||
def _nproc_ceiling() -> int:
|
||||
"""A process cap that bounds a fork bomb without tripping on existing load.
|
||||
|
||||
RLIMIT_NPROC is enforced per real UID across the whole system, not per
|
||||
process tree, so it has to sit above whatever the user is already running.
|
||||
"""
|
||||
try:
|
||||
current = len(os.listdir("/proc"))
|
||||
except OSError:
|
||||
current = 512
|
||||
return max(current * 2, 1024)
|
||||
|
||||
|
||||
def _apply_rlimits(timeout: float, with_nproc: bool,
|
||||
as_bytes: Optional[int] = LIMIT_AS_BYTES):
|
||||
def _preexec():
|
||||
os.setsid()
|
||||
cpu = int(timeout) + LIMIT_CPU_SECONDS_SLACK
|
||||
limits = [
|
||||
(resource.RLIMIT_CPU, (cpu, cpu)),
|
||||
(resource.RLIMIT_FSIZE, (LIMIT_FSIZE_BYTES, LIMIT_FSIZE_BYTES)),
|
||||
(resource.RLIMIT_CORE, (0, 0)),
|
||||
]
|
||||
if as_bytes:
|
||||
limits.append((resource.RLIMIT_AS, (as_bytes, as_bytes)))
|
||||
if with_nproc:
|
||||
ceiling = _nproc_ceiling()
|
||||
limits.append((resource.RLIMIT_NPROC, (ceiling, ceiling)))
|
||||
for res, lim in limits:
|
||||
try:
|
||||
resource.setrlimit(res, lim)
|
||||
except (ValueError, OSError):
|
||||
pass
|
||||
return _preexec
|
||||
|
||||
|
||||
class Sandbox:
|
||||
"""A cached venv plus an isolated runner for untrusted candidate programs."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
name: str,
|
||||
packages: Optional[Sequence[str]] = None,
|
||||
cache_root: Optional[Path] = None,
|
||||
base_python: Optional[str] = None,
|
||||
isolation: Optional[str] = None,
|
||||
provision: Optional[Sequence[str]] = None,
|
||||
env: Optional[Dict[str, str]] = None,
|
||||
address_space_limit: Optional[int] = LIMIT_AS_BYTES,
|
||||
):
|
||||
self.name = name
|
||||
self.packages = sorted(set(packages or []))
|
||||
self.provision = list(provision or [])
|
||||
self.env = dict(env or {})
|
||||
# None drops the RLIMIT_AS cap entirely -- see _apply_rlimits
|
||||
self.address_space_limit = address_space_limit
|
||||
self.cache_root = Path(cache_root or DEFAULT_CACHE_ROOT)
|
||||
self.base_python = base_python or sys.executable
|
||||
self.venv_dir = self.cache_root / f"{name}-venv"
|
||||
self._python: Optional[Path] = None
|
||||
self._isolation = isolation
|
||||
self._base_prefix: List[Path] = []
|
||||
|
||||
@property
|
||||
def python(self) -> Path:
|
||||
if self._python is None:
|
||||
raise RuntimeError("Sandbox.ensure() must be called first")
|
||||
return self._python
|
||||
|
||||
@property
|
||||
def isolation(self) -> str:
|
||||
if self._isolation is None:
|
||||
if self._python:
|
||||
py = str(self._python)
|
||||
roots = [self.venv_dir, *(self._base_prefix or [])]
|
||||
else:
|
||||
py = sys.executable
|
||||
roots = self._detect_base_prefix(Path(py))
|
||||
self._isolation = _detect_isolation(py, roots)
|
||||
return self._isolation
|
||||
|
||||
def _stamp_path(self) -> Path:
|
||||
return self.venv_dir / ".llama-eval-stamp.json"
|
||||
|
||||
def _stamp_matches(self) -> bool:
|
||||
try:
|
||||
stamp = json.loads(self._stamp_path().read_text())
|
||||
except Exception:
|
||||
return False
|
||||
return (
|
||||
stamp.get("packages") == self.packages
|
||||
and stamp.get("py") == self._base_version()
|
||||
and stamp.get("provision") == self.provision
|
||||
)
|
||||
|
||||
def _resolved_env(self) -> Dict[str, str]:
|
||||
return {k: v.replace("{venv}", str(self.venv_dir)) for k, v in self.env.items()}
|
||||
|
||||
def _base_version(self) -> str:
|
||||
r = subprocess.run(
|
||||
[self.base_python, "-c",
|
||||
"import sys;print('.'.join(map(str,sys.version_info[:3])))"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
return r.stdout.strip()
|
||||
|
||||
def ensure(self, force: bool = False, quiet: bool = False) -> Path:
|
||||
"""Create/refresh the venv. Idempotent and cheap when already valid."""
|
||||
py = self.venv_dir / "bin" / "python"
|
||||
if force and self.venv_dir.exists():
|
||||
shutil.rmtree(self.venv_dir)
|
||||
|
||||
if py.exists() and self._stamp_matches():
|
||||
self._python = py
|
||||
self._base_prefix = self._detect_base_prefix(py)
|
||||
return py
|
||||
|
||||
if not quiet:
|
||||
pkgs = ", ".join(self.packages) if self.packages else "stdlib only"
|
||||
print(f"[sandbox] provisioning venv {self.venv_dir} ({pkgs})")
|
||||
|
||||
self.cache_root.mkdir(parents=True, exist_ok=True)
|
||||
if self.venv_dir.exists():
|
||||
shutil.rmtree(self.venv_dir)
|
||||
|
||||
uv = shutil.which("uv")
|
||||
if uv:
|
||||
subprocess.run(
|
||||
[uv, "venv", "--python", self.base_python, str(self.venv_dir)],
|
||||
check=True, capture_output=True,
|
||||
)
|
||||
elif Path(self.base_python).resolve() == Path(sys.executable).resolve():
|
||||
venv.EnvBuilder(with_pip=bool(self.packages), clear=True).create(self.venv_dir)
|
||||
else:
|
||||
subprocess.run(
|
||||
[self.base_python, "-m", "venv", str(self.venv_dir)],
|
||||
check=True, capture_output=True,
|
||||
)
|
||||
|
||||
if self.packages:
|
||||
if uv:
|
||||
cmd = [uv, "pip", "install", "--python", str(py), *self.packages]
|
||||
else:
|
||||
cmd = [str(py), "-m", "pip", "install", "--quiet",
|
||||
"--disable-pip-version-check", *self.packages]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"failed to install {len(self.packages)} package(s) into "
|
||||
f"{self.venv_dir}:\n{(r.stderr or r.stdout)[-3000:]}"
|
||||
)
|
||||
|
||||
for i, snippet in enumerate(self.provision, 1):
|
||||
if not quiet:
|
||||
print(f"[sandbox] provisioning step {i}/{len(self.provision)}")
|
||||
prov_env = dict(os.environ)
|
||||
prov_env["SANDBOX_VENV"] = str(self.venv_dir)
|
||||
prov_env.update(self._resolved_env())
|
||||
r = subprocess.run([str(py), "-c", snippet], env=prov_env,
|
||||
capture_output=True, text=True, timeout=1800)
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"provisioning step {i} failed:\n{(r.stderr or r.stdout)[-3000:]}"
|
||||
)
|
||||
|
||||
self._stamp_path().write_text(json.dumps({
|
||||
"packages": self.packages,
|
||||
"py": self._base_version(),
|
||||
"provision": self.provision,
|
||||
}))
|
||||
self._python = py
|
||||
self._base_prefix = self._detect_base_prefix(py)
|
||||
if not quiet:
|
||||
print(f"[sandbox] ready ({self.isolation} isolation)")
|
||||
if self.isolation == "unshare-root":
|
||||
print("Warning: this host's unshare cannot keep the caller's uid, so "
|
||||
"candidate code runs as root inside the namespace. Tests that "
|
||||
"assert a permission denial will not see one. Install "
|
||||
"bubblewrap (bwrap) for correct results.")
|
||||
return py
|
||||
|
||||
@staticmethod
|
||||
def _detect_base_prefix(py: Path) -> List[Path]:
|
||||
"""Every directory that must stay visible for the venv to start.
|
||||
|
||||
A venv is only symlinks into its base interpreter, so the whole install
|
||||
tree has to survive the tmpfs that hides $HOME -- and binding just the
|
||||
resolved tree is not enough. Version-managed interpreters (uv, pyenv)
|
||||
are commonly reached through an unversioned alias symlink
|
||||
(cpython-3.13-... -> cpython-3.13.14-...), and startup resolves through
|
||||
the alias path, so both spellings need to be mounted. Walk the chain
|
||||
and keep every install root it passes through.
|
||||
"""
|
||||
roots: List[Path] = []
|
||||
|
||||
def add(path: Path):
|
||||
if path.exists() and path not in roots:
|
||||
roots.append(path)
|
||||
|
||||
cur, hops = py, 0
|
||||
while hops < 16:
|
||||
hops += 1
|
||||
add(cur.parent.parent)
|
||||
if cur.is_symlink():
|
||||
target = Path(os.readlink(cur))
|
||||
cur = target if target.is_absolute() else (cur.parent / target)
|
||||
else:
|
||||
break
|
||||
|
||||
try:
|
||||
r = subprocess.run([str(py), "-c", "import sys; print(sys.base_prefix)"],
|
||||
capture_output=True, text=True, timeout=60)
|
||||
if r.returncode == 0 and r.stdout.strip():
|
||||
add(Path(r.stdout.strip()))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return [p for p in roots if str(p) not in ("/", "/usr", "/usr/local")]
|
||||
|
||||
def _isolate(self, argv: Sequence[str], workdir: Path,
|
||||
ro_binds: Sequence[Path] = ()) -> List[str]:
|
||||
"""Wrap a command in the strongest isolation tier this host supports."""
|
||||
if self.isolation == "bwrap":
|
||||
return _bwrap_argv(
|
||||
workdir, argv,
|
||||
ro_binds=[self.venv_dir, *(self._base_prefix or []), *ro_binds],
|
||||
)
|
||||
if self.isolation in UNSHARE_ARGV:
|
||||
return [*UNSHARE_ARGV[self.isolation], *argv]
|
||||
return list(argv)
|
||||
|
||||
def _base_env(self, workdir: Path) -> Dict[str, str]:
|
||||
env = {
|
||||
"PATH": "/usr/bin:/bin",
|
||||
"HOME": str(workdir),
|
||||
"TMPDIR": str(workdir),
|
||||
"LC_ALL": "C.UTF-8",
|
||||
"PYTHONHASHSEED": "0",
|
||||
"PYTHONDONTWRITEBYTECODE": "1",
|
||||
"OPENBLAS_NUM_THREADS": "1",
|
||||
"OMP_NUM_THREADS": "1",
|
||||
}
|
||||
env.update(self._resolved_env())
|
||||
return env
|
||||
|
||||
def run_argv(self, argv: Sequence[str], workdir: Path,
|
||||
timeout: float = 15.0, env: Optional[Dict[str, str]] = None,
|
||||
ro_binds: Sequence[Path] = ()) -> ExecResult:
|
||||
"""Run an arbitrary command in `workdir`, which stays writable.
|
||||
|
||||
Unlike run(), the caller owns the directory and it survives the call.
|
||||
That is what a multi-step agent needs: its edits have to persist across
|
||||
many tool invocations before anything is graded. Taking an argv rather
|
||||
than a source string is what lets a suite drive a linter or a test
|
||||
runner for a language other than Python.
|
||||
"""
|
||||
run_env = self._base_env(workdir)
|
||||
if env:
|
||||
run_env.update(env)
|
||||
|
||||
started = time.monotonic()
|
||||
try:
|
||||
proc = subprocess.Popen(
|
||||
self._isolate(argv, workdir, ro_binds),
|
||||
cwd=str(workdir),
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
errors="replace",
|
||||
env=run_env,
|
||||
preexec_fn=_apply_rlimits(timeout,
|
||||
with_nproc=self.isolation == "rlimit",
|
||||
as_bytes=self.address_space_limit),
|
||||
)
|
||||
except Exception as e:
|
||||
return ExecResult(False, "error", "", f"{type(e).__name__}: {e}")
|
||||
|
||||
try:
|
||||
out, err = proc.communicate(timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
_kill_tree(proc)
|
||||
out, err = proc.communicate()
|
||||
return ExecResult(False, "timeout", out or "", err or "",
|
||||
None, time.monotonic() - started)
|
||||
|
||||
duration = time.monotonic() - started
|
||||
if proc.returncode == 0:
|
||||
return ExecResult(True, "ok", out, err, 0, duration)
|
||||
return ExecResult(False, "failed", out, err, proc.returncode, duration)
|
||||
|
||||
def run(self, code: str, timeout: float = 15.0,
|
||||
env: Optional[Dict[str, str]] = None) -> ExecResult:
|
||||
"""Run `code` to completion in the sandbox. Exit 0 == passed."""
|
||||
workdir = Path(tempfile.mkdtemp(prefix=f"{self.name}-", dir="/tmp"))
|
||||
try:
|
||||
prog = workdir / "candidate.py"
|
||||
prog.write_text(code)
|
||||
return self.run_argv([str(self.python), "-I", "-B", str(prog)],
|
||||
workdir, timeout=timeout, env=env)
|
||||
except Exception as e:
|
||||
return ExecResult(False, "error", "", f"{type(e).__name__}: {e}")
|
||||
finally:
|
||||
shutil.rmtree(workdir, ignore_errors=True)
|
||||
|
||||
|
||||
def _kill_tree(proc: subprocess.Popen):
|
||||
"""Kill the candidate and everything it spawned."""
|
||||
for sig in (signal.SIGKILL,):
|
||||
try:
|
||||
os.killpg(os.getpgid(proc.pid), sig)
|
||||
except (ProcessLookupError, PermissionError):
|
||||
try:
|
||||
proc.kill()
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
+1241
-34
File diff suppressed because it is too large.
Load diff
@@ -13,8 +13,6 @@ from typing import Dict, List, Optional
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import datasets
|
||||
|
||||
# Set cache directory for HuggingFace datasets
|
||||
cache_dir = Path.home() / ".cache" / "huggingface" / "datasets"
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
@@ -76,6 +74,8 @@ class AimeDataset:
|
||||
return question.get("problem", question.get("question", ""))
|
||||
|
||||
def _load_dataset(self):
|
||||
import datasets
|
||||
|
||||
if self.dataset_type == "aime":
|
||||
print(f"Loading AIME dataset (split: {self.split})...")
|
||||
cache_path = Path.home() / ".cache" / "huggingface" / "datasets" / "AI-MO___aimo-validation-aime" / "default" / "0.0.0"
|
||||
@@ -164,6 +164,58 @@ class AimeDataset:
|
||||
return str(normalized) if normalized is not None else answer
|
||||
return str(answer)
|
||||
|
||||
class CodeDataset:
|
||||
"""Stand-in for a code model: replays canonical solutions as chat replies.
|
||||
|
||||
Handles both code suites: HumanEval keys off the function stub, ClassEval
|
||||
off the class skeleton.
|
||||
"""
|
||||
|
||||
DEFAULT_SOURCE = {
|
||||
"humaneval": "openai/openai_humaneval",
|
||||
"classeval": "FudanSELab/ClassEval",
|
||||
}
|
||||
|
||||
def __init__(self, dataset_type: str, source: Optional[str] = None):
|
||||
self.dataset_type = dataset_type
|
||||
self.source = source or self.DEFAULT_SOURCE[dataset_type]
|
||||
self.questions: List[Dict] = []
|
||||
self._load_dataset()
|
||||
|
||||
def _load_dataset(self):
|
||||
print(f"Loading {self.dataset_type} dataset (source: {self.source})...")
|
||||
local = Path(self.source)
|
||||
if local.exists():
|
||||
text = local.read_text()
|
||||
if local.suffix == ".json":
|
||||
rows = json.loads(text)
|
||||
else:
|
||||
rows = [json.loads(ln) for ln in text.splitlines() if ln.strip()]
|
||||
else:
|
||||
import datasets
|
||||
rows = [dict(r) for r in datasets.load_dataset(self.source, split="test")]
|
||||
self.questions = rows
|
||||
print(f"{self.dataset_type} dataset loaded: {len(self.questions)} questions")
|
||||
|
||||
def _stub_field(self) -> str:
|
||||
return "skeleton" if self.dataset_type == "classeval" else "prompt"
|
||||
|
||||
def _get_question_text(self, question: Dict) -> str:
|
||||
return question[self._stub_field()]
|
||||
|
||||
def find_question(self, request_text: str) -> Optional[Dict]:
|
||||
for question in self.questions:
|
||||
stub = question[self._stub_field()].rstrip()
|
||||
if stub and stub in request_text:
|
||||
return question
|
||||
return None
|
||||
|
||||
def get_answer(self, question: Dict) -> str:
|
||||
if self.dataset_type == "classeval":
|
||||
return question["solution_code"]
|
||||
return question["prompt"] + question["canonical_solution"]
|
||||
|
||||
|
||||
class Simulator:
|
||||
def __init__(
|
||||
self,
|
||||
@@ -171,12 +223,17 @@ class Simulator:
|
||||
host: str = "localhost",
|
||||
success_rate: float = 0.8,
|
||||
dataset_split: str = "train",
|
||||
dataset_type: str = "aime"
|
||||
dataset_type: str = "aime",
|
||||
dataset_source: Optional[str] = None
|
||||
):
|
||||
self.port = port
|
||||
self.host = host
|
||||
self.success_rate = success_rate
|
||||
self.dataset = AimeDataset(dataset_split, dataset_type)
|
||||
self.dataset_type = dataset_type
|
||||
if dataset_type in ("humaneval", "classeval"):
|
||||
self.dataset = CodeDataset(dataset_type, dataset_source)
|
||||
else:
|
||||
self.dataset = AimeDataset(dataset_split, dataset_type)
|
||||
self.eval_state = EvalState(
|
||||
id=dataset_type,
|
||||
tasks=[dataset_type],
|
||||
@@ -196,6 +253,12 @@ class Simulator:
|
||||
else:
|
||||
response_text = self._generate_wrong_answer(question)
|
||||
|
||||
if self.dataset_type in ("humaneval", "classeval"):
|
||||
response_text = (
|
||||
f"Here is the implementation:\n\n```python\n{response_text}\n```\n\n"
|
||||
"Let me know if you want it optimised further."
|
||||
)
|
||||
|
||||
comp_tokens = random.randint(10000, 60000)
|
||||
tps_gen = random.uniform(90.0, 110.0)
|
||||
t_gen_ms = comp_tokens / tps_gen * 1000
|
||||
@@ -229,6 +292,12 @@ class Simulator:
|
||||
def _generate_wrong_answer(self, question: Dict) -> str:
|
||||
expected_answer = self.dataset.get_answer(question)
|
||||
|
||||
if self.dataset_type == "classeval":
|
||||
return re.sub(r"(\n\s+def [^\n]+:\n)", r"\1 return None\n",
|
||||
question["skeleton"])
|
||||
if self.dataset_type == "humaneval":
|
||||
return question["prompt"] + " return None\n"
|
||||
|
||||
if expected_answer.isdigit():
|
||||
wrong_answer = str(int(expected_answer) + 1)
|
||||
else:
|
||||
@@ -335,7 +404,7 @@ def main():
|
||||
"--dataset",
|
||||
type=str,
|
||||
default="aime",
|
||||
choices=["aime", "aime2025"],
|
||||
choices=["aime", "aime2025", "humaneval", "classeval"],
|
||||
help="Dataset type (default: aime)"
|
||||
)
|
||||
parser.add_argument(
|
||||
@@ -344,6 +413,12 @@ def main():
|
||||
default="train",
|
||||
help="AIME dataset split to use (default: train)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dataset-source",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Override the dataset location (local .jsonl path or HuggingFace repo id)"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
@@ -353,7 +428,8 @@ def main():
|
||||
host=args.host,
|
||||
success_rate=args.success_rate,
|
||||
dataset_split=args.dataset_split,
|
||||
dataset_type=args.dataset
|
||||
dataset_type=args.dataset,
|
||||
dataset_source=args.dataset_source
|
||||
)
|
||||
|
||||
server = HTTPServer((args.host, args.port), RequestHandler)
|
||||
|
||||
Reference in new issue
Block a user