master
40
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
85cc432e2c |
server: add router device memory margin parameter for dynamic unloading (#21231)
* server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold estimate with to-be-loaded model size included use no_alloc to get memory requirements for model load only set model memory_mb if not previously calculated use memory margin instead of total size limit, apply to each device separately add server memory debug logging move llama_context_device_memory function to llama-ext.h fix model count exceeded check improve memory_per_device map naming improve variable naming, fix style also strip models memory margin from child processes cont : clean-up replace device memory map with buft memory map. Use llama_get_memory_breakdown extract duplicated check into helper function move model memory estimation to subprocess precompute name->buft map, map GPU host types to CPU buft cleanup unused variable remove duplicated init calls * adapt to upstream changes * wip * common_memory_reports internface * SERVER_ROLE_MEASURE * server-models-sched * fix build * server : fix memory accounting of measure children and claim spin A measure child runs no_alloc, so the free memory it reports does not include its estimate: subtracting self there erased the memory held by other processes, down to zero with --fit on, and the next load could overshoot the margin. Subtract self only for instance reports. A queued load only checked --models-max before claiming, so when the memory margin blocked it behind a busy model, load() threw and the waiter retried in a tight loop. Refuse the claim while the limits are exceeded and no idle model can be unloaded, so the waiter sleeps on the condition variable until one goes idle. --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Xuan-Son Nguyen
|
f3269ef96a |
server: refactor server entrypoint (#31113)
* server_role * cont |
||
|
Xuan-Son NguyenandPascal
|
1242be66f6 |
server: add realtime transcription API (#30636)
* server: add realtime transcription API * drop transcription_sessions * proxy ws * add docs * server: let the API key middleware guard the websocket httplib runs the pre-routing handler before the websocket upgrade, so the API key middleware answers 401 at handshake like any other route. This covers the router proxy, whose children run keyless, and accepts the same key forms as HTTP (Bearer, raw key, X-Api-Key). * server: keep realtime slots out of the prompt loop A realtime slot waiting for audio is GENERATING with nothing to decode, so it fell through the prompt loop and took slot_batched. A request on a higher slot then failed can_batch_with on the task type and starved for the whole session. Realtime slots have no prompt: the loop sets them up once and always leaves them. * server: start the realtime session once its slot is live The audio stream was created in the prompt loop, which skips a slot that cannot batch with the one generating, and session.created went out before the task was posted: early appends failed and the reader quit, leaving the socket open and the slot held until the HTTP timeout. The slot now sets up its stream at launch and signals it with the begin result, session.created waits for it, and a reader that stops closes the socket so the session ends with it. * server: name the committed item in input_audio_buffer.committed Per the Realtime spec, item_id is the item made from the committed audio and previous_item_id the one committed before it, null for the first. committed now comes before the transcript that completes that item, and both go out with the item switch under one lock, so a delta cannot land between them. * server: use the GA transcription session shape The session object follows the GA Realtime schema: type transcription, with the input format, transcription model, turn detection and noise reduction under audio.input. The rate stays the model sample rate, which is what the client must send for now. --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
 
|
c0fd786264 |
ui : Discover & Download Models from HF Hub (#29584)
* common : resolve <quant>-<sidecar> download tags and list cached sidecars A Q4_0-mtp style tag now resolves the sidecar file when no model file matches it, so a solo draft or mmproj download actually pulls the file. Cached sidecar files list as their own entries so the state survives a restart, and removing such a tag deletes only the sidecar. Assisted-by: pi:zai-org/GLM-5.3 * ui : optional sanitized raw HTML in markdown Add an allowHtml prop to MarkdownContent: raw HTML found in the markdown is rendered after DOMPurify sanitization instead of being escaped as literal text. Default stays escaped. Assisted-by: pi:GLM-5.3-Flash * ui : add the discover models components The Discover surface: the curated-catalog list with its search and skeletons, the details pane with its readme, metadata, Hub stats and download options, and the standalone download progress bar that replaces the plain one. The quant download button sizes its select with a new xs trigger, so the select gains that size and the muted box look. The store stays incomplete when a repo fails, so the next mount retries it. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash * ui : put the discover view in the models manager The manager dialog gains its second view: the Discover Models call to action fades the table out, slides the title with its compass mark, and fades the discover surface in; the arrow button returns to the table. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash # Conflicts: # tools/ui/src/lib/components/ui/button/button.svelte * ui : add the Enable Discover Models setting The flag gates browsing and downloading HuggingFace GGUF models, so it is declared on the discover layer; the repo-tree lookup in the manager layer above reads it from here. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash * ui : resume model downloads from the pause point A stopped download keeps its partial file instead of deleting it, so re-posting the tag resumes from disk. The first progress record now carries the resume offset, the paused snapshot survives into the resumed entry, and a pause settles as soon as it is requested. Assisted-by: pi * ui : lead the discover list with downloads The discover sidebar lists running downloads above the suggested models, and steps aside while a query is active. A paused download stays in the manager table, where it is resumed or dropped. Assisted-by: pi * ui : make resuming a paused download work Re-posting the tag of a paused download hit the router's "already exists" check, because a paused download parks its entry as DOWNLOADED until the next reload: a download entry that is not running is now replaced and resumed. The UI keeps the paused row when the request still fails, instead of dropping it and looking finished. Assisted-by: pi * ui : remember unfinished model downloads A partial file is named after its content hash, so nothing on disk says which model it belongs to once the router restarts. The router now keeps the tags of the downloads it started, clears a tag when the download finishes or is deleted, and lists what is left as still downloading without starting anything. The UI adopts those tags as paused on every list fetch, so a refresh or a fresh open shows an incomplete download and can resume it from the files already on disk. Assisted-by: pi * ui : show a paused download's state on its chip A paused quant chip reads as paused at rest, the way a running one shows its spinner, and offers the resume action on hover. The selector's download row gets a View in Discover action beside its cancel. Assisted-by: pi * ui : collapse the download options by default The copyable download command starts collapsed so the option rows lead; expanding it stays one click away. Assisted-by: pi * ui : keep decision models out of the hub lists Decision models answer with calibrated probabilities instead of text, so llama-ui cannot run them: the hub data layer drops them from the catalog and every search, keyed by the decision-model tag or a text-classification pipeline. Assisted-by: pi * ui : read download tags as quant and sidecar A `<repo>:<quant>-<sidecar>` tag now parses into its quant and sidecar instead of one opaque quant token, and ModelsService.draftSidecarsByRepo collects the router listing's sidecar-only entries per repo, replacing the inline copy in the models store. Assisted-by: pi * ui : cluster draft downloads under their main model Tracked downloads group by repo: the main weights lead and the draft sidecars ride inside their item as compact sub-lines, on the selector, the discover list and the manager table. The main model's controls pause, resume and cancel the whole cluster, and the manager's download rows select into the configuration pane. Assisted-by: pi * ui : act on tracked downloads from the table rows The table passes its clusters' draft sub-lines to the model rows. A row's actions cascade pause and resume, an in-flight row offers Cancel download while a paused one keeps Delete from disk, and the confirm dialog's copy follows the state. Assisted-by: pi * ui : give downloads a configuration pane of their own Selecting a tracked download opens the pane in a download mode: live progress with pause, resume and cancel, and See in Discover, instead of metadata and load actions that a model without files cannot offer. Assisted-by: pi * ui : make the discover details self-fetching and responsive ModelsDiscoverDetailsContainer owns the details, tree and readme fetches so hosts outside the discover view can render the pane, and the details layout adapts to narrow panes with container queries instead of viewport breakpoints. Assisted-by: pi * ui : download a matching draft with the main model A main quant download now pulls the quant-matched draft of the most preferred kind the repo ships (DFlash before MTP), fire-and-forget, so speculative decoding is ready when the main lands. Manual draft chips stay available; the options row also collapses to one column in narrow panes. Assisted-by: pi * ui : keep discover off by default and couple it with the hub setting Discover Models starts unchecked. Enabling it turns the Hugging Face Hub setting on and disabling it turns it back off, so the checkbox never lies; with discover off the hub setting stays available on its own for the display data. Stored and reset state are corrected the same way on load. Assisted-by: pi * ui : open the manage models dialog on the manager view The dialog now opens on the manager instead of returning to the last view, discover entries still land on the discover view while the setting allows it, and the discover surfaces follow the setting's state. Assisted-by: pi * ui : give the discover search room for its focus ring The sticky search bar insets from the panel edges so the focus ring stays inside the scroll container, and rows scroll behind an opaque bar instead of through it. Assisted-by: pi * common: settle sidecar names on the listing and drop unfinished partials An infix or trailing sidecar token is ambiguous: draft heads use it (Model-MTP-BF16.gguf), so do full models with embedded heads (Model-Q8D-MTP.gguf). Such a name is a sidecar only next to token-free main weights of the same family, while mmproj or imatrix anywhere in the name, or a leading token, decides alone (mmproj_f16.gguf). A quant tag always carries its bit count, so a sidecar named after no quant (mtp-Model-it.gguf) gets the bare tag: the listing, the plan and remove all use it. An imatrix tag resolves into a plan slot of its own, downloaded but never wired into the params. remove resolves the tag the way a download does and deletes the partial files of every planned blob, so cancelling a running download or dropping one left unfinished by an earlier run leaves nothing on disk. The router passes its token and releases its lock around the call, and records the unfinished downloads with an atomic write. llama download succeeds on a sidecar-only plan and prints its paths. * ui : move dialog views and draft states into enums and constants The models dialog's views become the ModelsDialogView enum, the draft subline's states join ModelRowDownloadState, the view fade and the copy-feedback reset move to ui constants, and the dialog's compass marks come from DISCOVER_ICON. Assisted-by: pi * ui : drop the unused discover surface The dead openDiscoverModels twin, the commented-out command divider, the details pane's unused onClose and its header's close button, and a redundant type cast leave; the command's copy feedback joins the shared reset constant. Assisted-by: pi * ui : trim the download option comments and highlight the chat template The comment verbosity the options panel and its chips carried shrinks to the non-obvious why, and the chat template dialog renders through SyntaxHighlightedCode as jinja instead of a plain pre block. Assisted-by: pi * common: only a .gguf file is a sidecar, and mmproj/imatrix as whole words A non-GGUF file named like a sidecar (quantization-mmproj-Q8_0.json) is picked as the mmproj, and a sidecar word inside a longer word of a model name (Phi-2-OmniMatrix.i1-Q4_K_M.gguf) drops the model. * ui: dedupe draft sidecar badges per quant The draft sidecar badges deduped by kind and repo without the quant, so with a draft cached at two quants only one badge survived, and a delete of the other main quant found no badge to remove its sidecar with. Also restores the getDownloadEntries doc lost when the clusters method landed above it. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: adopt the incomplete download status and drop the paused-ids registry The store inferred an unfinished download from a downloading entry it held no local state for, which misread a download running in another tab as paused and offered a resume the server rejects. The listing carries the "incomplete" status now, so the adoption keys off it directly. The paused ids no longer need to survive in localStorage: the server remembers them across restarts, and the progress snapshot was stale after a reload anyway. The stale localStorage key is removed. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: quant-match the pause and cancel cascades Pause, resume and cancel cascaded to every draft sidecar of the repo while delete only took the quant-matched ones, so canceling a Q4_0 download also killed a Q8_0 draft that belongs to another quant's main. Every cascade acts on the quant-matched drafts now, matching what a delete takes with it. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: collapse the duplicated draft subline markup The downloaded and in-flight branches repeated the identical connector, label and badge markup; one block with a conditional tail keeps the two states in sync. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: scrub positioned styles out of sanitized model card html The style attribute stays allowed for the KaTeX output, which let a third party card draw a fixed overlay over the app. The sanitize hook drops position:fixed and position:absolute values and leaves the KaTeX layout styles alone. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: run the discover mount fetch once and reuse the shared repoOf helper fetch() toggles its loading flag synchronously inside the mount effect, which re-ran the fetch on every toggle and retried failed catalog repos endlessly; the effect now untracks the call. The local repoOf copy and the duplicate services import give way to the shared helpers. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: narrow the spec-type map to the draft sidecars SPEC_TYPE carried empty entries for the aux sidecars it is never indexed with; the map and the command component now type the badge as a draft sidecar through a draftBadgeOf helper instead of a cast. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: name the decision-model tags and markdown cache keys The decision-model hub tag and pipeline tag move into the huggingface constants next to the other pipeline tags, and the markdown transform cache prefixes get names instead of bare one-letter tokens. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: read the discover setting through one truthiness check Three places read the setting with !== false and one with Boolean(); all four read it the same way now. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: compare the models dialog view against its enum The template compared the view state against the 'manage' literal in three spots while the script used the enum. Assisted-by: pi (zai-org/GLM-5.3-Flash) * nit: Remove `+` before draft sidecar badge * server: list unfinished downloads as incomplete A download that never finished was reported as "downloading", the same status a live download carries, so a client could not tell a running download from a dead one. The tags remembered in unfinished-downloads now list under a dedicated "incomplete" status, without an instance. Assisted-by: pi (zai-org/GLM-5.3-Flash) * server: download a model with its sidecars as -hf does Resolution goes back to master. POST /models takes the sidecar kinds to pull with the model (mmproj by default) and the download child gets them as -hf flags, so the server picks the files. /models lists real models only, each with the sidecars it has on disk, and incomplete downloads keep their sidecars across restarts. A stopped download keeps its partial files for the next attempt, and DELETE purges them through the same plan the download follows. * ui: download sidecars as parameters of their model The download options keep their chips, but a chip no longer posts a sidecar tag: the store maps it to its model and the sidecar kinds to pull, the quant chip takes its companion draft in the same download, and draft badges come from the sidecars each cached model lists. The server adds the sidecars a cached model lacks when it is posted again, and DELETE removes a single sidecar of a cached model. * ui : mirror the server's sidecar lookup in draft badges The server resolves a sidecar by the case-sensitive `<kind>-` token in the same directory as the weights. The UI matched the token loosely, so a badge could offer a draft the server never pulls. Assisted-by: DeepSeek-V4.1-Flash * ui : post the model's sidecar kinds with its download The store keeps one download per model: the chip's kinds ride along with the model's own sidecars, and a download row no longer expands into one row per sidecar. Draft badges carry no quant of their own yet, so a sidecar of a cached model is never labelled with the model's quant. Assisted-by: DeepSeek-V4.1-Flash * ui : join a quant's draft into its download chip One chip per quant, its draft badged inside it and its size covering both files. The draft is the file the server would pick: the same directory, the exact quant, else the closest bit depth, so a 4-bit model can pair with a 16-bit head. Assisted-by: DeepSeek-V4.1-Flash * ui : drop the draft download rows from the manager and selector A download is one row per model again: the drafts ride with it, and the rows read the sidecars on disk off the listing instead of pairing sidecar entries with their model. Pause, resume and cancel act on the model's own download. Assisted-by: DeepSeek-V4.1-Flash * ui : keep the discover download list one row per model The list leads with one row per downloading model, its sidecars as sub-lines. A row badges only the draft kinds the server can resolve, so a badge never outlives the file its lookup would pick. Assisted-by: DeepSeek-V4.1-Flash * ui : keep the discover chips readable in dark mode The metadata chips, the chat template button and the View on Hugging Face link carried a faint border and no surface over the dialog background, so they were near invisible. They take the tertiary button surface now, and their icons size with size-* so the variant's base rule leaves them alone. Assisted-by: DeepSeek-V4.1-Flash * ui : carry the draft through the download options One chip per main quant, its draft badged on it and the chip's downloaded state covering both files. The drafts sit behind an opt-in checkbox with an inline type pick, and each row opens its own command: it pins the row's quant and only names the draft type, so -hfd is gone. Assisted-by: DeepSeek-V4.1-Flash * ui : line the details skeleton up with the loaded pane Same root padding and container context as the loaded view, so the container queries resolve alike. The download options card pads its blocks instead of itself, the readme gets its card, and a description line stands in for the catalog paragraph. Assisted-by: DeepSeek-V4.1-Flash * common : drop the partials of a removed sidecar Removing a sidecar alone left the .downloadInProgress files of the sidecars a download had planned in blobs/. The removal purges them through the same plan the download follows, and the test covers a finished draft beside an unfinished one, whose partial goes while the model keeps its files. Assisted-by: DeepSeek-V4.1-Flash * server : name why a download is refused A model that is already downloading, or that is running, answered "already exists". It now says which: "is already downloading", or "is running, unload it first", both before the resumable check. The router test covers the two. Assisted-by: DeepSeek-V4.1-Flash * ui : post a download once per click Two quick clicks on the same chip posted twice, and the second answer was "already exists". The store tracks the model as pending, returns early while it is pending or in flight, and reads it as in flight before the server answers, so the chip shows the download at once. The new client test pins the flows: one POST per double click, the chip reading downloaded once the list reports the model, a resume keeping its sidecars, a cancel dropping the whole download, and a reload restoring the paused state. Assisted-by: DeepSeek-V4.1-Flash * ui : keep a plain download row the height of the others The sub-lines wrapper of a download row was rendered even with nothing in it, so its padding made the row taller than a plain one and the overlaid progress bar sat in that empty strip. It renders only with sub-lines now, in the selector and in the discover list, and the selector row takes the phone padding its neighbours have. Assisted-by: DeepSeek-V4.1-Flash * ui : badge a repo's draft once The args of a loaded model and the listing of a cached one report the same draft in different words, one with the file's quant and one with the kind alone, so the merge kept both and a repo row showed the same kind twice. The badges merge per kind now, the args one winning as it knows the quant, and the test pins both sources of a real repo. Assisted-by: DeepSeek-V4.1-Flash * server : report the sidecars of a preset or a download on its way A model the preset also names is registered as a preset, so its drafts were never reported and no row could badge them. A download on its way reported none either, though the record of the kinds it pulls is right there: those win over the plan now, the plan resolves through the preset's hf-repo, and a models dir stays out of it. Assisted-by: DeepSeek-V4.1-Flash * ui : badge a download's drafts and open its model from the row A draft still coming down badges the model id now, the way a landed one does, while an on-disk draft keeps its checked sub-line. The row itself opens the model, the status slot beside it pauses or resumes, and the manager resolves a focus on a download row so its panel opens. Assisted-by: DeepSeek-V4.1-Flash * common : report a removed sidecar that only had a partial Removing a sidecar alone dropped the result of the partial purge, so a sidecar that only ever landed as a .downloadInProgress blob answered false and the UI had no reason to refetch. The purge result is folded into the return now, and the test covers a partial-only removal. Assisted-by: DeepSeek-V4.1-Flash * ui : ignore a pause while the download POST is still on its way The chip reads as downloading before the server answers, so a quick second click unloaded a model that had not started and left the store neither in flight nor paused. A pause returns early while the POST is pending, before it touches the feed or the paused marker. The client test holds the POST open and pins it. Assisted-by: DeepSeek-V4.1-Flash * ui : offer a missing draft as its own chip A model already on disk without its draft had no way to add it: the quant chip carried the draft, so it read as a download of both. The quant chip keeps to its own state now, its joined draft shows only while it rides along or is on disk, and a downloaded quant missing its draft gets a chip of its own - badge, size, download - that posts the model with that kind, so the server keeps the files already there and pulls the draft. Assisted-by: DeepSeek-V4.1-Flash * server : list a paused sidecar download by the files on disk A model on disk whose sidecar download stops keeps its unfinished record, and the listing reported the kinds that record plans as if they were on disk: the UI checked a draft that never landed and offered no way to add it. The planned kinds now stand in only while the download runs. * chore: Clean up comments * server : list sidecars only for a model that resolves from a HF repo A preset on a local file has no HF repo, yet the listing resolved its name as one: the plan threw on the invalid repo format and GET /models failed for every model. The sidecars now come from the repo a preset names or a cached model is listed under, and a local file lists without them. * common : fix download cancel regression Signed-off-by: Adrien Gallouët <angt@huggingface.co> * common : preserve model partials when removing sidecars Signed-off-by: Adrien Gallouët <angt@huggingface.co> * server : one sidecar table and an unfinished downloads class Address review from @ggerganov. The sidecar kinds live in one table in common/download, mapping each name to its download option and plan file; the server validates, builds its options and lists cached sidecars through it instead of its own lists. The unfinished downloads move into server_unfinished_downloads, a record with its own mutex and a struct per entry, and server_models exposes them through unfinished_downloads_get and unfinished_downloads_remove. * common: one sidecar table, and removals that keep shared files The resolution, the cached listing and the removal follow a single table of sidecar kinds, so a kind is declared once with its file keyword, whether it is a draft, its option and its plan file. The server reads the same table for the kinds it accepts, the default sidecars and the child flags. Removing a tag keeps the files another cached tag still loads, such as an mmproj named after the removed quant. A sidecar removed on its own goes for every tag that resolves it. An unfinished download records the exact paths of its partials when it starts. A removal deletes those without asking the hub, and keeps a partial another recorded download also writes. The record is taken once the child is spawned, and the files go under the models lock, so a download of the same tag cannot start while they are removed. * server: drop stale partials and refuse a tag that resolves nothing Recording a download again deletes the partials of the earlier record that the new plan no longer writes, such as those of an older hub revision or of a sidecar the new request leaves out. A posted tag that resolves no file of the repo is refused before the download starts. Removing a model follows a single path, whether the reload erased its entry during the wait or not. * server: fix the CI race between the router download tests The pytest workers run in parallel on one shared cache, so a download test could find the file another test had just fetched and see no progress, or start from the unfinished downloads record another test wrote. Each download test now runs on an empty cache of its own. * server: refuse a DELETE of an unknown name and decide downloads in load() A DELETE of a name that is neither a model nor a recorded download went to common_download_remove, and without a tag it wiped the whole repo cache, including the files of a loaded quant or of a download on its way. Such a name is now not found, as it was before. A POST that downloads over an existing entry checked its state outside the lock, so a load of the same model in between got its instance killed. The checks now run in load(), under the lock, right before the spawn. The fake load time of the tests applies to model loads only, since a download worker loads none. * server: add incomplete_sidecars to the /models API A sidecar added to a cached model and then paused kept its record, but the listing showed only the complete model, so a fresh UI lost the pause. The model entry now carries incomplete_sidecars, the kinds its unfinished download has not written yet; a POST with the kind resumes one and a DELETE with it drops its partial. * common : drop a kept partial when a resume gets 416 Now that partials survive between attempts, an old or complete partial makes every retry fail with the same 416. Delete it so the next attempt can start over from zero. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * common: drop a kept partial when a resume gets 200 A server that ignores the range answers 200 to a resume, and the kept partial then fails every attempt the same way a 416 does. Drop it as well, so the next attempt starts over without a range. * server: refuse a download that shares a partial with one on its way Two downloads of quants that share a file, the mmproj or an mtp picked by the closest bits, appended to the same partial and renamed a file with interleaved bytes. A download whose plan writes a partial that a running download also writes is now refused until that one is done. * common : check the saved etag before resuming a partial download Before resuming, compare the etag saved with the partial to the one the server now serves. If it is missing or different, start over, so a resumed download never mixes bytes from two versions of the file. Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co> Co-authored-by: Pascal <admin@serveurperso.com> Co-authored-by: Adrien Gallouët <angt@huggingface.co> |
||
|
Aleksander GrygierandPascal
|
baef3ed9a1 |
ui: Models Manager Follow-up Improvements (#30228)
* common : read a GGUF's trained context from its metadata common_get_gguf_n_ctx_train opens only the file's metadata (no_alloc, like common_get_decision_type) and reads <arch>.context_length, so a caller can learn the trained context without loading the model. It accepts both u32 and u64 values and returns 0 when the file is missing, unreadable, invalid, or reports no context length. Assisted-by: pi:zai-org/GLM-5.3-Flash * server : report the trained context in the models listing update_caps already resolves the model file offline to read its modalities, so it now reads the trained context from the same GGUF metadata, and GET /models reports it as context_length when it is known. A router listing then carries the context without any Hub request, which lets the UI sort and filter by it offline. Assisted-by: pi:zai-org/GLM-5.3-Flash * ui : take the trained context from the models listing The router now reports context_length per model, so the option mapping fills contextLength from it and the manager reads it before the Hub record. The Context column, the context sort and the context filter then work with the Hugging Face Hub API turned off. A browser suite guards the sort and the search, the Hub-cache driven context filter and the re-sort when details arrive after the sort was clicked. Assisted-by: pi:zai-org/GLM-5.3-Flash * ui : mark favorite models with a heart A favorited model shows a rose heart in the selector even before its row is hovered, and the crossed heart takes its place on hover, so unfavoriting stays one hover away. The manager table marks its favorited rows with the same heart after the badges and capabilities. Assisted-by: pi:zai-org/GLM-5.3-Flash * fix: UI text nit * fix: UI nits * fix: Favorite models grouping in models table * feat: Remove sorting from Status column in Models Table * server: read the GGUF metadata once per model Read the decision type and the trained context in a single GGUF open, accept only a UINT32 context length like the model loader, and reset n_ctx_train with the other caps so a failed refresh drops it. * fix: Post-review fixes --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Adrien Gallouët
|
4d60b4d087 |
common, server : report model input/output modalities in GET /models (#29987)
Signed-off-by: Adrien Gallouët <angt@huggingface.co> |
||
 PascalandXuan-Son Nguyen
|
a7fb71fab8 |
log, server: self contained colors, split child commands from logs in router mode (#29895)
* log, server: make router child lines carry their own colors The logger writes the color reset after the trailing newline, so the reset opens the next line. On the shared pipe of a router child it lands in front of the next state command, which the router then misses, and the line break that works around it shows up as an empty log line on every progress update. The reset now goes before the trailing newlines, so every line is self contained and the command goes back to its plain framing. The router passes its effective color setting to its children, whose output ends up in its terminal, and leaves that option out when comparing presets on reload. * log: enable ANSI colors on the Windows console A Windows console renders ANSI sequences only in virtual terminal mode, which nothing turns on for the logger, so llama-server prints raw escape codes on the Windows 10 console while llama-cli, whose console code enables it, shows colors. The logger now enables virtual terminal mode on stdout and stderr when it turns colors on, and keeps colors off when a console cannot render them. Pipes and files take the sequences as is. * server: separate the router child commands from its logs The child sent its state commands on the same pipe as its logs, so the router had to pick them out of the log stream by a line prefix, and any unterminated write in front of a command made the router miss it. This resolves the TODO at the spawn that called for splitting stdout and stderr. The child now keeps stdout for the commands and points everything else written to stdout at stderr, before anything is written. The router reads both pipes, handles the commands from stdout and forwards stderr as the log, and warns about any other line on the command pipe. * server: address review from ngxson The single server_child is now created first in the entry point and its constructor keeps stdout for the commands, so the stream is a member of the instance instead of a static, and init() is gone. The instance is passed down to the server, while the CLI entry point creates its own. * Update tools/server/server.cpp --------- Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> |
||
|
Xuan-Son NguyenandPascal
|
82d6bb284d |
server: refactor subproc handling (#28555)
* server: refactor subproc handling * fix Windows build * download: keep concurrent downloads of one blob apart Every process writes the same path + .downloadInProgress, so a second download of the same blob finds that file, takes it for its own partial transfer and asks for the bytes after it, which produces a corrupt result. The in-progress file now carries the pid of the process writing it. std::rename also replaces an existing destination on POSIX but fails on Windows, so a download whose blob appeared in the meantime is dropped after every retry and an etag rewrite silently keeps the old value. std::filesystem::rename has the POSIX behaviour everywhere, and the error now carries the reason reported by the system. * Revert "download: keep concurrent downloads of one blob apart" This reverts commit 917b83f149c625527f872fb2cf41289358fa5371. * tests: serialize the router tests that download the same model Parallel workers share one cache, so the two tests fetch the same blob into the same in-progress file and race to rename it. They now take a file lock around the download, like the session fixture does for the preset models. * Revert "tests: serialize the router tests that download the same model" This reverts commit c368a4a98c677938ca87002edb6186ff2c02fd83. --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Xuan-Son NguyenandPascal
|
160bd031b2 |
server: fix LRU hang on multiple requests same model (#28539)
* server: fix LRU hang on multiple requests same model * server: keep a queued model out of the victim pool until its waiters leave A waiter that gave up while its model was still loading left the model idle with no request behind it, and nothing recounted the free slots, so a second request queued behind it stayed queued forever. tick() was only driven by requests: join, claim and the end of a proxied request. Keep the queue entry alive after a successful claim so the model coming up is never picked as a victim before its waiters use it, and recount the slots on every status change and whenever a waiter abandons the queue. The model is then evicted as soon as it comes up with nobody left to serve. --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Xuan-Son Nguyen
|
9855ad69d3 |
server: (router) lazy-load startup_models after main setup (#27424)
* server: (router) lazy-load startup_models after main setup * only allow is_first_load to populate it * nits * nits 2 |
||
|
Xuan-Son Nguyen
|
ee4c505a4f | server: add dedup-cache-models preset option (#27346) | ||
|
Xuan-Son Nguyen
|
dff15d4ac9 |
server: (router) add LRU scheduler (#26572)
* add lru_sched * handle coalescing (req leaves waiting queue) * add tests * fix stream case * address review comments |
||
|
Xuan-Son Nguyen
|
e1470ee6a2 | server: (router) do not evict busy models (#26567) | ||
|
Pascal
|
d73c1d6b22 |
server + ui: fix stream routes for model names containing a slash (#26137)
* server + ui: refactor resumable stream routes to query string conv_id The conversation id can embed a model name containing slashes (ggml-org/...) in router mode, which the decoded path splits before the :conv_id param is captured, so stop and resume never matched the session. Move the id to the conv_id query string on the public routes and on the internal router -> child hop, where slashes survive encoding. Handlers are unchanged since query and path params land in the same map. Add a regression test with a slashed model name. * server: move stream route docs to server-stream.h Address review: ngxson wants the main server.cpp registration code kept clean and simple, with route-level explanations living in the header. Move the query string rationale and the lookup ownership note next to the handler declarations in server-stream.h, and shorten the wiring comment to a pointer. * server: cancel a pending request when its stream is stopped during model load The conversation was registered in the conv map only after the blocking autoload wait, so a stop issued while the model loaded found nothing to cancel and the request went on to generate an orphan once the load ended. Register the conversation before the wait and give the entry a ticket: a stop erases the entry, and the parked request checks its ticket after the wait and aborts with 400 instead of starting. A newer request on the same conversation replaces the entry, so only the stopped request is cancelled. Add a regression test that stops during the load window. * server + ui: resume a stream after a page reload during model load A pending request died with the client socket when the page was reloaded while its model was loading, so no session ever existed and the conversation had nothing to recover. A session request that waited for a load now detaches from the client socket and reaches the child regardless, the session buffer receives the generation, and the resume route answers 503 while the owner is loading so the client retries instead of dropping its state. The WebUI persists the pending stream at send time, quietly polls on 503, and attaches once the session exists. Add a regression test that drops the client during the load window. * ui: show the model load progress again after a page refresh The resume wait was invisible, so a conversation refreshed while its model was loading showed nothing until the first byte. On a 503 from the resume probe, mark the conversation as loading again so the assistant row persisted at send time renders the processing info, and target the model frozen in the persisted stream state for the progress, since the row has no model yet and the dropdown may not be restored. * fix CI * fix CI bis |
||
|
PascalandXuan Son Nguyen
|
1a87dcdc45 |
server + ui: SSE Replay Buffer (#23226)
* server: SSE replay buffer, survives client disconnect
Opt in on POST /v1/chat/completions when the client sends
X-Stream-Resume: 1 and a non empty X-Conversation-Id. The conv id is
the session identity end to end, no extra opaque token. The drain
runs detached server side and buffers SSE bytes, the generation
survives HTTP disconnect, F5, or lets users switch from iOS Safari
to another app without losing the actively generated response.
Routes:
GET /v1/stream/<conv_id>?from=N replay
GET /v1/streams[?conversation_id=X] list, drives sidebar spinners
DELETE /v1/stream/<conv_id> Stop, idempotent
Router parent fans out to children for list and delete, probes on GET
to route to the owner, fans out DELETE on POST so "one session per
conv" holds across model swaps.
WebUI: the layout snapshots /v1/streams at mount and on
visibilitychange, the sidebar reflects live inferences across all
convs. The chat page reattaches on mount, append vs fresh is detected
from existing content so continue mid stream keeps its prefix.
update_slots: on llama_memory_seq_rm refusal at a deep position, full
clear of the seq and reprefill from zero instead of GGML_ABORT.
OAI strict path unchanged when the opt in headers are absent.
* server: create stream session only after post_tasks succeeds
* server, ui: drop X-Stream-Resume, X-Conversation-Id alone enables the replay buffer
* server: drop magic 17, derive the X-Conversation-Id header length from sizeof at build time
* refactor: address review feedback from ngxson
* server-context: cleaning
* server-stream: fix use-after-free on rd
Guard stop_producer with a shared alive flag, flipped by on_stream_end
before rd dies. Prevents a late cancel (session eviction by a later
POST on the same conv_id, or a DELETE arriving after the producer
ended) from touching a destroyed rd.
* ui: fix cross-conversation contamination
Scope streaming flags per conv so one finishing does not unflag the
others, guard discoverActiveStream against concurrent runs to avoid
duplicate attaches, and stop racing syncRemoteRunningStreams for the
sidebar set.
* server-http: keep request alive in detached SSE drain
The response next() lambda may reach into *request via &req long
after on_complete reset the request shared_ptr. Capture request in
the detached thread so it outlives the drain.
* ui: address review feedback from coder543
Forward Authorization to /v1/stream and /v1/streams fetches, the resumable routes
must obey --api-key like the rest of the API.
Wrap reader.read() in a try/catch, the underlying connection drop rejects with
TypeError instead of resolving done=true, treat it as a premature end of stream
so the existing resume loop kicks in.
Freeze the model at session start in chatStreamingStates.model and thread it
through cancel and resume, the dropdown selection may have changed since the
POST and the server side identity is fixed at that time.
* format
* ui: remove unused selectedModelName
* server-stream: poll session->is_cancelled() in stream_aware_should_stop
Address review feedback from coder543. The cancel propagation through
rd.stop() relies on the slot eventually processing the cancel task and
posting a result that notifies the recv condvar, remove_waiting_task_ids
does not notify directly. Add a defensive poll on session->is_cancelled()
so the producer-side next() loop exits on its next iteration after
cancel() without waiting for the cancel task to round trip through a slot.
* server-stream, ui: replace GET /v1/streams with POST /v1/streams/lookup
Address review feedback from coder543. Listing live sessions leaks the
conversation_id of every concurrent user, which defeats the random UUID
unguessability. The new route takes {conversation_ids: [...]} in the
body and returns matches only for the ids the caller already owns, so
foreign UUIDs stay private. The router fans out the same POST to every
child and aggregates, the WebUI passes the convs visible in its sidebar.
* ui: read conv ids from IndexedDB in syncRemoteRunningStreams
The conversations store is not hydrated yet at +layout onMount, so the
sidebar spinners stayed off for background convs until the user clicked
on them. Read straight from the DB to dodge the init race.
* server-models: deduplicate stream lookup timeouts behind one constant
* ui: extract visibility kick grace into a stream constant, bump to 1000 ms
* make it safer & more simple
* server-stream: survive client disconnect via stream_pipe::finish_producer
After the RAII rewrite the generation stopped the moment the client
disconnected. httplib bails its content provider on the is_peer_alive
check at the top of write_content_chunked, so returning true from the
provider never keeps it producing: the response resets, rd is destroyed
and its task gets cancelled.
Reinstate the disconnect survival inside the pipe. stream_pipe gains
finish_producer, which pumps the response next() into the ring buffer
until the generation ends, and mark_producer_done for the clean wire
end. server-http only triggers them: mark before sink.done on a clean
close, finish in on_complete when the peer left early. No detach, no
stream logic in server-http beyond the trigger, and the strict OAI path
is untouched when no pipe is attached.
Known limitation: finish_producer pumps synchronously on the http
worker, so a disconnected stream keeps its worker busy until the
generation ends. A follow-up will move the drain off the http worker so
no worker is held.
* server-stream: drain disconnected streams on a manager owned thread
The previous commit pumped the post disconnect drain synchronously in
on_complete, on the http worker, so a disconnected stream kept its
worker busy until the generation ended. Under a wave of reloads or tab
closes that pins workers from the pool.
Move the drain off the http worker. on_complete now hands the response
to stream_session_manager::adopt_orphan, which pumps it to completion on
a manager owned thread and releases the worker at once. One thread per
disconnected stream still generating, stored in a list, joined and
reaped on the next adopt, by the GC, and at shutdown. No detach, the
thread lifecycle is fully owned by the manager. needs_drain gates the
handoff so a cleanly finished stream never spawns a thread, and the
strict OAI path stays untouched when no pipe is attached.
stop_gc now cancels sessions before finalizing them, so an in flight
drain sees is_cancelled and exits instead of blocking the shutdown join
until the generation ends naturally.
* ui: add missing JSDoc
* server-stream: drain on the http worker, drop the manager thread
Address @ngxson review: httplib runs a large dynamic pool and a worker
blocked in next() sits on a condvar instead of burning cpu, so draining
the rest of the generation on that worker is fine and much simpler than
a dedicated thread.
on_complete calls finish_producer directly again. Removes adopt_orphan,
the orphan thread list and its reaping, the stop_gc session cancel that
only existed to unblock those threads, and the now dead drain_shutdown
flag.
* server-stream: split stream_pipe into producer and consumer classes
Address @ngxson review: one class covering both ends was messy. stream_pipe
is now a base holding the session and is_cancelled, with stream_pipe_producer
(write, mark_producer_done, finish_producer, cleanup, finalizes on destruct)
and stream_pipe_consumer (read only, no finalize) deriving from it.
Drops the is_producer_ discriminator and its runtime guards, the type now
encodes the role. res.spipe is retyped to shared_ptr<stream_pipe_producer>
since it is only ever a producer. No behavior change.
* server-stream: rename producer methods to unix pipe semantics
Address @ngxson review: mark_producer_done becomes done(), finish_producer
becomes close(), matching a unix pipe write end. The producer_done_ member
follows as done_. write() is unchanged. No behavior change.
* server, ui: route resumable streams via a conv map, persist resume identity
Address ngxson review: drop the polling probe, proxy_post records a conv_id ->
model map and the stream routes resolve the owning child with one lookup. The
map is the single source of truth, the ::model suffix stays for child session
uniqueness but the router never parses it.
UI: the server keys a session by the POST time identity (conv::model), but reload
probed with the bare conv id and missed model tagged sessions, so F5 stopped the
stream and sidebar spinners stayed off. Persist the model and rebuild the exact
identity on resume, single conv and bulk sidebar both send it.
Add unit coverage for the identity round trip.
* ui: resolve continue target by id to stop cross-conversation flash on switch
* ui: skip stream resume when the abort is intentional
* server: move the conv id to model map into a self contained tracker
Address review from ngxson: server_models held two mutexes side by side, the
global one and a bare conv_model_mu guarding a loose map, which made the locking
hard to follow. Wrap the map and its lock in a small conv_model_tracker struct
that owns its mutex, one mutex per struct. The remember, lookup and forget
methods move inline into the tracker, server_models exposes a single conv_models
member and the routes call models.conv_models.lookup and friends. No behavior
change, the map stays the single source of truth for routing resumable streams
to a child.
* ui: replace stream magic values with enums and shared constants
Address review from allozaur: lift the inline literals around the resumable
stream code into named symbols so the intent is explicit and reusable.
* ui: fold the stream resume and discovery helpers into ChatService
Address review from allozaur: drop the two standalone stream-*.service files.
They were used only by the chat service and store, carried no shared state, and
did not follow the static class pattern the other services use, so a separate
abstraction was not warranted. Move the helpers onto ChatService as static
methods. No behavior change, tests now exercise them through ChatService.
* docs: document the SSE replay buffer in server README-dev
Add the resumable streaming section, list stream_session_manager in the
backend component inventory, and link PR 23226 in the related PRs.
* ui: align attachServerStream call with onCompletionId param in handleStreamResponse
* server-http: rename del_ to del to match get and post
* ui: address review feedback from allozaur
* ui: drop duplicate SSE constants, keep sse.ts canonical
* ui: use svelte:document for the visibilitychange listener
address review from allozaur: replace the manual document.addEventListener
in onMount with a declarative <svelte:document onvisibilitychange>. svelte
handles attach, detach and SSR, so the typeof document guard and the onMount
cleanup go away. onMount keeps only the first load snapshot.
* server: trim redundant stream drain comments
Address review from ngxson
* server: balance and clean up stream comments
remove redundant comments and tighten the verbose ones across the resumable
stream code, keeping the concurrency and lifetime rationale that is not obvious
from the code. also fix two stale comments in server.cpp and server-models.h
that still described the old ::model suffix probe and fan out routing, now
replaced by the conv_id -> model map
Address review from ngxson
* ui: balance and clean up stream comments
dedup repeated rationale (frozen conv::model identity, the lookup privacy note,
the abort patterns) down to one canonical spot, tighten the verbose blocks, and
keep the concurrency and resume-offset reasoning. fix stale comments in
stream-identity.ts and chat.service.ts that still described the old loopback
probe and fan out routing, now the conv_id -> model map.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
|
||
|
Xuan-Son Nguyen
|
721354fbdf |
server: (router) move model downloading to dedicated process (#24834)
* server: real-time model load progress tracking via /models/sse * update docs * server: move model download to child process * rm unused * fix most problems * clean up * nit fixes * fix test case * do not detact() thread * shorter MODEL_DOWNLOAD_TIMEOUT in test * throttle |
||
|
Xuan-Son Nguyen
|
d6d899580d |
server: real-time model load progress tracking via /models/sse (#24828)
* server: real-time model load progress tracking via /models/sse * update docs * add mutex for notify_to_router * correct docs |
||
|
Xuan-Son Nguyen
|
2b686a9120 |
server: refactor child --> router communication (#24821)
* server: refactor child --> router communication * fix wakeup case * add docs * improve update_status() * nits |
||
|
Xuan-Son Nguyen
|
175147e8f6 | server: remove all internal mentions about "webui" (#24817) | ||
|
Xuan-Son Nguyen
|
8c2d6f6475 |
server: add --agent arg, remove redundant webui naming compat (#24801)
* server: add --agent arg, remove redundant webui naming compat * corrent env * fix the test * llama-gen-docs * nits: wordings |
||
|
Xuan-Son Nguyen
|
4b4d13ae72 |
server: (router) add model management API (#23976)
* wip * server: (router) add SSE realtime updates API * nits * wip * add download API * add download api * update docs * add delete endpoint * fix std::terminate * fix crash * fix 2 * add tests * nits |
||
|
Xuan-Son Nguyen
|
06d26dfdff |
download: add option to skip_download (#23059)
* download: add option to skip_download * fix * fix 2 * if file doesn't exist, respect skip_download flag |
||
|
Aleksander GrygierandXuan Son Nguyen
|
59778f0196 |
ui: Restructure repo to use tools/ui folder and ui / UI / llama-ui / LLAMA_UI naming (#23064)
* webui: Move static build output from `tools/server/public` to `build/ui` directory * refactor: Move to `tools/ui` * refactor: rename CMake variables and preprocessor defines - Rename LLAMA_BUILD_WEBUI -> LLAMA_BUILD_UI (old kept as deprecated) - Rename LLAMA_USE_PREBUILT_WEBUI -> LLAMA_USE_PREBUILT_UI (old kept as deprecated) - Backward compat: old vars auto-forward to new ones with DEPRECATION warning - Rename internal vars: WEBUI_SOURCE -> UI_SOURCE, WEBUI_SOURCE_DIR -> UI_SOURCE_DIR, etc. - Rename HF bucket: LLAMA_WEBUI_HF_BUCKET -> LLAMA_UI_HF_BUCKET - Emit both LLAMA_BUILD_WEBUI and LLAMA_BUILD_UI preprocessor defines - Emit both LLAMA_WEBUI_DEFAULT_ENABLED and LLAMA_UI_DEFAULT_ENABLED * refactor: rename CLI flags (--webui -> --ui) with backward compat - Add --ui/--no-ui (old --webui/--no-webui kept as deprecated aliases) - Add --ui-config (old --webui-config kept as deprecated alias) - Add --ui-config-file (old --webui-config-file kept as deprecated alias) - Add --ui-mcp-proxy/--no-ui-mcp-proxy (old --webui-mcp-proxy kept as deprecated) - Add new env vars: LLAMA_ARG_UI, LLAMA_ARG_UI_CONFIG, LLAMA_ARG_UI_CONFIG_FILE, LLAMA_ARG_UI_MCP_PROXY - C++ struct fields: params.ui, params.ui_config_json, params.ui_mcp_proxy added alongside old fields - Backward compat: old fields synced to new ones in g_params_to_internals * refactor: update C++ server internals with backward compat - Rename json_webui_settings -> json_ui_settings (both kept in server_context_meta) - Rename params.webui usage -> params.ui (both synced, old still works) - JSON API emits both "ui"/"ui_settings" and "webui"/"webui_settings" keys - Server routes use params.ui_mcp_proxy || params.webui_mcp_proxy - Preprocessor guards use #if defined(LLAMA_BUILD_UI) || defined(LLAMA_BUILD_WEBUI) * refactor: rename CI/CD workflows, artifacts, and build script - Rename webui-build.yml -> ui-build.yml; artifact webui-build -> ui-build - Rename webui-publish.yml -> ui-publish.yml; var HF_BUCKET_WEBUI_STATIC_OUTPUT -> HF_BUCKET_UI_STATIC_OUTPUT - Rename server-webui.yml -> server-ui.yml; job webui-build/checks -> ui-build/checks - Update server.yml: job/artifact refs webui-build -> ui-build - Update release.yml: all webui-build/publish refs -> ui-build/publish; HF_TOKEN_WEBUI_STATIC_OUTPUT -> HF_TOKEN_UI_STATIC_OUTPUT - Update server-self-hosted.yml: webui-build -> ui-build - Update build-self-hosted.yml: HF_WEBUI_VERSION -> HF_UI_VERSION - Rename webui-download.cmake -> ui-download.cmake (internal refs updated) - Update labeler.yml: server/webui -> server/ui path label * docs: update CODEOWNERS and server README docs - Update CODEOWNERS: team ggml-org/llama-webui -> ggml-org/llama-ui, path /tools/server/webui/ -> /tools/ui/ - Update server README.md: CLI tables show --ui flags with deprecated --webui aliases - Update server README-dev.md: "WebUI" -> "UI", paths updated to tools/ui/ * fix: Small fixes for UI build * fix: CMake.txt syntax * chore: Formatting * fix: `.editorconfig` for llama-ui * chore: Formatting * refactor: Use `APP_NAME` in Error route * refactor: Cleanup * refactor: Single migration service * make llama-ui a linkable target * fix: UI Build output * fix: Missing change * fix: separate llama-ui npm build output into build/tools/ui/dist subfolder + use cmake npm build instead of downloading ui-build.yml artifacts in CI * refactor: UI workflows cleanup --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
Xuan-Son Nguyen
|
7bfe120c21 |
mtmd, server, common: expose modalities to /v1/models (#22952)
* mtmd, server, common: expose modalities to /v1/models * fix build * rename to mtmd_caps |
||
|
Xuan-Son Nguyen
|
9dcf835528 |
server: (router) expose child model info from router's /v1/models (#22683)
* server: (router) expose child model info from router's /v1/models * update docs |
||
|
Xuan-Son Nguyen
|
935a340292 | server: implement /models?reload=1 (#21848) | ||
  tha80andRalph Paßgang
|
983ca8992e |
server: (router) Forward form-data to model server (Fixes #22044) (#22118)
* This commit enables the router to forward form-data to model server. Fixes #22044 (enabling to use the /v1/audio/transcriptions in router mode) * * Applied the suggestion from Copilots first comment: using the non-throwing json::parse overload. * Addressed Copilots third comment by extending the files representation to also include filename and content-type * Addressed Copilots fourth comment by making the RNG thread_local * Changed variable body from std::string to std::ostringstream in build_multipart_body as suggested by ngxson in https://github.com/ggml-org/llama.cpp/pull/22118#discussion_r3127099053 * Added sanitize_field lambda in build_multipart_body for key, filename and content_type as suggested by ngxson in https://github.com/ggml-org/llama.cpp/pull/22118#discussion_r3127104647 * explicitly checking if value/item is string before calling value/item.get<std::string>() as requested by ngxson in https://github.com/ggml-org/llama.cpp/pull/22118#discussion_r3127111279 * Added double quote to the sanitize lambda and throw on json parse failure --------- Co-authored-by: Ralph Paßgang <ralph@trust-it.de> |
||
|
Xuan-Son Nguyen
|
49bfddeca1 |
server: allow router to report child instances sleep status (#20849)
* server: allow router to report child instances sleep status * refactor * move sleeping to state * nits |
||
 Evan HuusandXuan Son Nguyen
|
23fbfcb1ad |
server: Parse port numbers from MCP server URLs in CORS proxy (#20208)
* Parse port numbers from MCP server URLs * Pass scheme to http proxy for determining whether to use SSL * Fix download on non-standard port and re-add port to logging * add test --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
Pascal
|
2e7e638523 |
server : support multiple model aliases via comma-separated --alias (#19926)
* server : support multiple model aliases via comma-separated --alias * server : update --alias description and regenerate docs * server : multiple model aliases and tags - address review feedback from ngxson - --alias accepts comma-separated values (std::set, no duplicates) - --tags for informational metadata (not used for routing) - aliases resolve transparently in router via get_meta/has_model - /v1/models exposes aliases and tags fields * regenerate docs * nits * server : use first alias as model_name for backward compat address review feedback from ngxson * server : add single-model test for aliases and tags |
||
 Vladislav Sayapin
|
da143b9940 | server : fix router child env in containerized environments (#18562) | ||
  wbtekandGreg Slocum
|
5b1248c9af |
server : Cmdline arg -to changes http read timeout from current 600sec default (#18279)
* Prevent crash if TTFT >300sec, boosted to 90 days * server : allow configurable HTTP timeouts for child models * server : pass needed timeouts from params only --------- Co-authored-by: Greg Slocum <fromgit@wbtek.slocum.net> |
||
|
Xuan-Son Nguyen
|
f5acfb2ffa |
server: (router) add stop-timeout option (#18350)
* server: (router) add stop-timeout option * also allow stop while loading * add docs * unload_lru: also wait for unload to complete |
||
|
Xuan-Son NguyenandPascal
|
9e39a1e6a9 |
server: support load model on startup, support preset-only options (#18206)
* server: support autoload model, support preset-only options * add docs * load-on-startup * fix * Update common/arg.cpp Co-authored-by: Pascal <admin@serveurperso.com> --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Xuan-Son Nguyen
|
98c1c7a7bf |
presets: refactor, allow cascade presets from different sources, add global section (#18169)
* presets: refactor, allow cascade presets from different sources * update docs * fix neg arg handling * fix empty mmproj * also filter out server-controlled args before to_ini() * skip loading custom_models if not specified * fix unset_reserved_args * fix crash on windows |
||
|
Pascal
|
6ce3d85796 |
server: (webui) add --webui-config (#18028)
* server/webui: add server-side WebUI config support Add CLI arguments --webui-config (inline JSON) and --webui-config-file (file path) to configure WebUI default settings from server side. Backend changes: - Parse JSON once in server_context::load_model() for performance - Cache parsed config in webui_settings member (zero overhead on /props) - Add proper error handling in router mode with try/catch - Expose webui_settings in /props endpoint for both router and child modes Frontend changes: - Add 14 configurable WebUI settings via parameter sync - Add tests for webui settings extraction - Fix subpath support with base path in API calls Addresses feedback from @ngxson and @ggerganov * server: address review feedback from ngxson * server: regenerate README with llama-gen-docs |
||
|
Xuan-Son Nguyen
|
bde461de8c |
server: (router) allow child process to report status via stdout (#18110)
* server: (router) allow child process to report status via stdout * apply suggestions |
||
  
|
f32ca51bfe |
server: add presets (config) when using multiple models (#17859)
* llama-server: recursive GGUF loading Replace flat directory scan with recursive traversal using std::filesystem::recursive_directory_iterator. Support for nested vendor/model layouts (e.g. vendor/model/*.gguf). Model name now reflects the relative path within --models-dir instead of just the filename. Aggregate files by parent directory via std::map before constructing local_model * server : router config POC (INI-based per-model settings) * server: address review feedback from @aldehir and @ngxson PEG parser usage improvements: - Simplify parser instantiation (remove arena indirection) - Optimize grammar usage (ws instead of zero_or_more, remove optional wrapping) - Fix last line without newline bug (+ operator instead of <<) - Remove redundant end position check Feature scope: - Remove auto-reload feature (will be separate PR per @ngxson) - Keep config.ini auto-creation and template generation - Preserve per-model customization logic Co-authored-by: aldehir <aldehir@users.noreply.github.com> Co-authored-by: ngxson <ngxson@users.noreply.github.com> * server: adopt aldehir's line-oriented PEG parser Complete rewrite of INI parser grammar and visitor: - Use p.chars(), p.negate(), p.any() instead of p.until() - Support end-of-line comments (key=value # comment) - Handle EOF without trailing newline correctly - Strict identifier validation ([a-zA-Z_][a-zA-Z0-9_.-]*) - Simplified visitor (no pending state, no trim needed) - Grammar handles whitespace natively via eol rule Business validation preserved: - Reject section names starting with LLAMA_ARG_* - Accept only keys starting with LLAMA_ARG_* - Require explicit section before key-value pairs Co-authored-by: aldehir <aldehir@users.noreply.github.com> * server: fix CLI/env duplication in child processes Children now receive minimal CLI args (executable, model, port, alias) instead of inheriting all router args. Global settings pass through LLAMA_ARG_* environment variables only, eliminating duplicate config warnings. Fixes: Router args like -ngl, -fa were passed both via CLI and env, causing 'will be overwritten' warnings on every child spawn * add common/preset.cpp * fix compile * cont * allow custom-path models * add falsey check * server: fix router model discovery and child process spawning - Sanitize model names: replace / and \ with _ for display - Recursive directory scan with relative path storage - Convert relative paths to absolute when spawning children - Filter router control args from child processes - Refresh args after port assignment for correct port value - Fallback preset lookup for compatibility - Fix missing argv[0]: store server binary path before base_args parsing * Revert "server: fix router model discovery and child process spawning" This reverts commit e3832b42eeea7fcb108995966c7584479f745857. * clarify about "no-" prefix * correct render_args() to include binary path * also remove arg LLAMA_ARG_MODELS_PRESET for child * add co-author for ini parser code Co-authored-by: aldehir <hello@alde.dev> * also set LLAMA_ARG_HOST * add CHILD_ADDR * Remove dead code --------- Co-authored-by: aldehir <aldehir@users.noreply.github.com> Co-authored-by: ngxson <ngxson@users.noreply.github.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: aldehir <hello@alde.dev> |
||
|
Pascal
|
5ceed62421 |
server: fix duplicate HTTP headers in multiple models mode (#17698)
* llama-server: fix duplicate HTTP headers in multiple models mode (#17693) * llama-server: address review feedback from ngxson - restrict scope of header after std::move - simplify header check (remove unordered_set) |
||
|
ec18edfcba |
server: introduce API for serving / loading / unloading multiple models (#17470)
* server: add model management and proxy * fix compile error * does this fix windows? * fix windows build * use subprocess.h, better logging * add test * fix windows * feat: Model/Router server architecture WIP * more stable * fix unsafe pointer * also allow terminate loading model * add is_active() * refactor: Architecture improvements * tmp apply upstream fix * address most problems * address thread safety issue * address review comment * add docs (first version) * address review comment * feat: Improved UX for model information, modality interactions etc * chore: update webui build output * refactor: Use only the message data `model` property for displaying model used info * chore: update webui build output * add --models-dir param * feat: New Model Selection UX WIP * chore: update webui build output * feat: Add auto-mic setting * feat: Attachments UX improvements * implement LRU * remove default model path * better --models-dir * add env for args * address review comments * fix compile * refactor: Chat Form Submit component * ad endpoint docs * Merge remote-tracking branch 'webui/allozaur/server_model_management_v1_2' into xsn/server_model_maagement_v1_2 Co-authored-by: Aleksander <aleksander.grygier@gmail.com> * feat: Add copy to clipboard to model name in model info dialog * feat: Model unavailable UI state for model selector * feat: Chat Form Actions UI logic improvements * feat: Auto-select model from last assistant response * chore: update webui build output * expose args and exit_code in API * add note * support extra_args on loading model * allow reusing args if auto_load * typo docs * oai-compat /models endpoint * cleaner * address review comments * feat: Use `model` property for displaying the `repo/model-name` naming format * refactor: Attachments data * chore: update webui build output * refactor: Enum imports * feat: Improve Model Selector responsiveness * chore: update webui build output * refactor: Cleanup * refactor: Cleanup * refactor: Formatters * chore: update webui build output * refactor: Copy To Clipboard Icon component * chore: update webui build output * refactor: Cleanup * chore: update webui build output * refactor: UI badges * chore: update webui build output * refactor: Cleanup * refactor: Cleanup * chore: update webui build output * add --models-allow-extra-args for security * nits * add stdin_file * fix merge * fix: Retrieve lost setting after resolving merge conflict * refactor: DatabaseStore -> DatabaseService * refactor: Database, Conversations & Chat services + stores architecture improvements (WIP) * refactor: Remove redundant settings * refactor: Multi-model business logic WIP * chore: update webui build output * feat: Switching models logic for ChatForm or when regenerating messges + modality detection logic * chore: update webui build output * fix: Add `untrack` inside chat processing info data logic to prevent infinite effect * fix: Regenerate * feat: Remove redundant settigns + rearrange * fix: Audio attachments * refactor: Icons * chore: update webui build output * feat: Model management and selection features WIP * chore: update webui build output * refactor: Improve server properties management * refactor: Icons * chore: update webui build output * feat: Improve model loading/unloading status updates * chore: update webui build output * refactor: Improve API header management via utility functions * remove support for extra args * set hf_repo/docker_repo as model alias when posible * refactor: Remove ConversationsService * refactor: Chat requests abort handling * refactor: Server store * tmp webui build * refactor: Model modality handling * chore: update webui build output * refactor: Processing state reactivity * fix: UI * refactor: Services/Stores syntax + logic improvements Refactors components to access stores directly instead of using exported getter functions. This change centralizes store access and logic, simplifying component code and improving maintainability by reducing the number of exported functions and promoting direct store interaction. Removes exported getter functions from `chat.svelte.ts`, `conversations.svelte.ts`, `models.svelte.ts` and `settings.svelte.ts`. * refactor: Architecture cleanup * feat: Improve statistic badges * feat: Condition available models based on modality + better model loading strategy & UX * docs: Architecture documentation * feat: Update logic for PDF as Image * add TODO for http client * refactor: Enhance model info and attachment handling * chore: update webui build output * refactor: Components naming * chore: update webui build output * refactor: Cleanup * refactor: DRY `getAttachmentDisplayItems` function + fix UI * chore: update webui build output * fix: Modality detection improvement for text-based PDF attachments * refactor: Cleanup * docs: Add info comment * refactor: Cleanup * re * refactor: Cleanup * refactor: Cleanup * feat: Attachment logic & UI improvements * refactor: Constants * feat: Improve UI sidebar background color * chore: update webui build output * refactor: Utils imports + move types to `app.d.ts` * test: Fix Storybook mocks * chore: update webui build output * test: Update Chat Form UI tests * refactor: Tooltip Provider from core layout * refactor: Tests to separate location * decouple server_models from server_routes * test: Move demo test to tests/server * refactor: Remove redundant method * chore: update webui build output * also route anthropic endpoints * fix duplicated arg * fix invalid ptr to shutdown_handler * server : minor * rm unused fn * add ?autoload=true|false query param * refactor: Remove redundant code * docs: Update README documentations + architecture & data flow diagrams * fix: Disable autoload on calling server props for the model * chore: update webui build output * fix ubuntu build * fix: Model status reactivity * fix: Modality detection for MODEL mode * chore: update webui build output --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |