Slot selection takes free slots by id or by last use only, so child
tasks often land on a slot holding a parent state while empty slots
are idle, and overwrite it without saving it to the prompt cache.
Free slots are now ordered empty first, then least recently used, for
both the LRU fallback and the child slots, and a child task saves the
state it overwrites. With 4 grouped decision requests in flight and
12 states in rotation, prompt cache reads go from 5 to 36.
[no release]
* server: save only restorable slot states to the prompt cache
The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.
Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.
* server: test that decision slots skip the prompt cache
Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.
* server: restore any reusing task from the prompt cache
Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.
Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
* init conversion
* convert: ok
* model loaded
* add server code
* improve conversion script
* support shared prompt prefix
* add docs, imorove UX a bit
* add vision support
* add openjev tiny model for testing
* add dev docs
* support lev & kev
* clean up
* fix lev noul
* fix py lint
* nits docs
* clarify about not supporting date_facts
Pascal
Xuan-Son Nguyen