Georgi Gerganov
a17e94d229
* tests : consolidate the context state tests into test-llama-context test-save-load-state, test-recurrent-state-rollback and test-state-restore-fragmented check the same object from three places: the KV cache and the recurrent state of a llama_context, saved, loaded, copied, rolled back. Each had its own main, its own loop over the models, its own copy of the token and the comparison helpers, and a run of all three loaded the same models several times. They are one binary now, and the case is the unit of the test: tests/llama-context/test-case.h the interface a case implements tests/llama-context/cases-kv.cpp the cases of the KV cache tests/llama-context/cases-rs.cpp the cases of the recurrent state tests/llama-context/test-util.cpp tokens, generation, comparison tests/llama-context/test-runner.cpp the models of a case, on threads tests/llama-context/test-log.cpp the log file of a model A run is a table: a row per model, a column per case, a cell holds the verdict of that pair. The models load on several threads (-j, --threads, min(cores, 4) by default), the table is printed when it is complete and the log of a model that failed is printed after it, in the order of the rows. A case returns a verdict and the reason for it, and the runner is the only one who reports - the test does not use tests/testing.h, a run over 130 models is a matrix of verdicts and not a tree of subtests. Staging no longer travels through the environment: llama_model_set_attn_rot replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one thread used to change for the models the other threads were loading, and llama_memory_get_buffers is added for the cases that copy the buffers of a memory. common_log_set_stdo keeps the console quiet while the table is being built. ggml_aligned_malloc allocates with posix_memalign when the thread sanitizer is on. On macOS it allocates with the Mach VM, which the sanitizer does not intercept, so it never sees a page range change hands and reports the first write into recycled memory as a race - 30 of them in this test, and every one of them false. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * cont : run single-threaded by default * cont : try to build the tests on all platforms * cont : trim down some body fat * cont : fix * cont : fix the fix
84 lines
2.5 KiB
C++
84 lines
2.5 KiB
C++
#pragma once
|
|
|
|
#include "test-util.h"
|
|
|
|
#include <memory>
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
//
|
|
// the interface of a single context state case
|
|
//
|
|
|
|
enum class test_status {
|
|
PASS,
|
|
FAIL,
|
|
SKIP,
|
|
};
|
|
|
|
// the verdict as printed in the result table, colored
|
|
const char * test_status_str(test_status status);
|
|
|
|
// how many columns the verdict takes in the table - the escapes of the color are not printed
|
|
size_t test_status_width(test_status status);
|
|
|
|
// a skipped case does not fail the run, a failed case always does
|
|
test_status merge_status(test_status a, test_status b);
|
|
|
|
enum test_group {
|
|
TEST_GROUP_STATE, // KV state save / load / copy
|
|
TEST_GROUP_RS, // recurrent state, also run against a dirty cache
|
|
};
|
|
|
|
struct case_result {
|
|
test_status status = test_status::PASS;
|
|
std::string reason; // why the case was skipped or failed, empty when it passed
|
|
|
|
static case_result pass();
|
|
static case_result fail(const std::string & why = {});
|
|
static case_result skip(const std::string & why);
|
|
};
|
|
|
|
struct ctx_test {
|
|
virtual ~ctx_test() = default;
|
|
|
|
// the column header of the --models table, keep it short
|
|
virtual const char * name() const = 0;
|
|
|
|
// one line of what the case verifies, printed by --list
|
|
virtual const char * description() const { return ""; }
|
|
|
|
virtual test_group group() const { return TEST_GROUP_STATE; }
|
|
|
|
// the case compares against the logits produced by the baseline case
|
|
virtual bool needs_baseline() const { return false; }
|
|
|
|
// the case is also run against a cache pre-filled with a known pattern, so that a restore
|
|
// reading cells it never wrote cannot pass
|
|
virtual bool uses_fill() const { return group() == TEST_GROUP_RS; }
|
|
|
|
// non-empty means the case does not apply to this model, the text is the reported reason
|
|
virtual std::string skip_reason(model_run & mr) const;
|
|
|
|
// the scenario, the checks are written with CTX_REQUIRE
|
|
virtual case_result run(model_run & mr) const = 0;
|
|
};
|
|
|
|
// record an expectation that did not hold and give up on the case - the runner reports the
|
|
// reason, the body does not print anything
|
|
#define CTX_REQUIRE(cond, msg) \
|
|
do { \
|
|
if (!(cond)) { \
|
|
return case_result::fail(msg); \
|
|
} \
|
|
} while (0)
|
|
|
|
using ctx_test_ptr = std::unique_ptr<const ctx_test>;
|
|
|
|
// the cases of one source file, in table order
|
|
std::vector<ctx_test_ptr> make_kv_cases(); // cases-kv.cpp
|
|
std::vector<ctx_test_ptr> make_rs_cases(); // cases-rs.cpp
|
|
|
|
// all cases, in table order
|
|
const std::vector<ctx_test_ptr> & test_cases();
|