Models & Tuning

Tune every llama-server flag on a per-model basis, save named presets, and compare settings across models — all from the Models tab, without hand-editing models.ini.

What it does

Each row in the Models list is a section of models.ini (or an auto-discovered model not yet added to it). Expanding a row opens a live knob editor built from the running llama-server binary's own --help output, so the available flags always match the binary you actually built — not a hardcoded list that can drift out of date across llama.cpp versions.

backend/argspec.py (build_schema()) runs <server_bin> --help, parses the column-aligned help text into typed, grouped knobs (bool, int, float, enum, path, string), and caches the result per (server_bin path, binary mtime) — the cache self-invalidates automatically if you repoint server_bin at a different binary or rebuild it (backend/routes.py schema()). A handful of flags the router itself owns (host, port, model, hf-repo, and similar) are filtered out of the editor as RESERVED — they aren't safe to set per model.

The editor has two density levels, controlled by ui_mode in config.json:

• Lite shows a curated set of common knobs (n-gpu-layers, ctx-size, cache-type-k/cache-type-v, flash-attn, batch-size, ubatch-size, threads, tensor-split, temp, top-p).
• Advanced exposes every flag the binary reports via --help, grouped under the section headers from the help text itself.

Because the flag count is entirely a function of your llama-server build, LlamaForge does not hardcode a number for it — it is whatever --help reports at the time the schema is built.

How to use it

1. Open the Models tab and click a model's row to expand its editor.
2. Set the knobs you want to override. Unset fields inherit from the [*] global-defaults section of models.ini.
3. Click Save + Reload. This writes the changed keys into the model's models.ini section (config.set_keys()), and if the model is currently loaded, unloads it first, then tells the router to reload (reload_presets(): POST /models/reload, or GET /models?reload=1 on builds without that route) so the next load picks up the new settings — no dashboard or router restart required.
4. To reuse a set of knobs on other models, click Save current + in the preset bar to name the model's current settings as a preset. Click a saved preset chip on any other model's row to apply it (same save + reload path). Delete a preset with the × on its chip.
   • To make a preset a model's default, click the ◉/○ dot on its chip to bind it (POST /api/presets/bind). Binding writes the preset's knobs into that model's section now, and — the point of it — editing the preset later re-syncs every model bound to it. The bound chip is highlighted; clicking the dot again unbinds and leaves the knobs in place. Knobs you set by hand afterward still win, since they're written last.
   • To save a one-click launch profile, click Save as profile in a model's editor. A profile is the model plus an optional preset plus an optional pinned engine build (one of the official builds installed from Build / Update; leave it on "whichever build is active" to follow updates). Profiles appear as ▶ chips above the model list: one click switches to the pinned build if it isn't active (waiting for the router to come back), applies the preset, and loads the model. Builds a profile pins are never pruned by later updates; if a pinned build or preset has been removed, the launch stops with an explanation and changes nothing.
   • To share a profile, click ↗ on its chip and copy the recipe: readable JSON with the GGUF file name, the Hugging Face repo it came from (when it was downloaded through LlamaForge or sits in the Hugging Face cache), its knobs, and the llama.cpp build it was made on. Anyone can paste it into + import recipe to get the same preset and profile; if the model isn't on their machine, Download from <repo> & import fetches the same file (all shards) and finishes the import. Recipes are untrusted input, so treat one from a stranger like code: every knob is shown before you apply it, and only tuning knobs are imported. Anything that touches paths or files (including any knob your engine types as a path), other hosts (rpc), keys, logging, CORS, server-side tools/MCP, the chat template, or model auto-downloads (*-default presets) is dropped and listed, as are knobs your engine doesn't know. GPU topology and CPU pinning (main-gpu, tensor-split, split-mode, device, threads, cpu-*) are never shared either: they only make sense on the machine that made the recipe. A value containing a newline rejects the whole recipe.
   • browse recipes opens the community gallery: tested setups from the repo's recipes/ folder, each with the hardware it ran on, notes on its knobs, and an on this machine badge when the model file is already registered. Import prefills the import box with that recipe and runs it. The list is fetched live from GitHub (cached for 6 hours, refresh forces a fetch) and falls back to the copy bundled with your release when offline or rate-limited. Gallery files go through the same filter as a pasted recipe. To add one, export it with ↗, add an about block, and open a PR (recipes/README.md); CI rejects a gallery recipe that carries any knob the filter would drop.
5. To compare settings across models, click Compare above the model list, tick the checkbox on two or three rows (three is the most), then open the comparison. The table lists every knob key any selected model has explicitly set and highlights cells that differ between models; a blank cell marked "inherit" means that model falls back to the [*] default.
6. To auto-tune a model's knobs based on your hardware, use the Refine bar beside Presets: pick an intent (balanced / speed / context / coding), click Run, and it runs one short real completion (~200 tokens) per candidate and applies the fastest config. It is a quick check, not llama-bench. A results table shows tok/s per candidate and which was chosen.
7. To remove a stale or unwanted llama.cpp entry, click Unregister. This unloads it if necessary and removes only its models.ini section and preset binding; the GGUF file remains on disk. vLLM's separate Delete action still removes its managed WSL files.

1. To run several llama.cpp models at once, turn on Multi-model (llama.cpp) in Setup. Load then loads a model as the main model, on the fastest GPU that fits, and Load as worker (offered only beside a model that is already up) loads it beside the main one on the GPUs the main model doesn't use, and never unloads anything. An open row's editor says whether the model is the main model or a worker, and for an unloaded one whether it Fits (with the footprint, marked measured, predicted or a rough estimate) or Won't fit, with an Unload ... and load button when evicting workers would make room. Make main promotes a worker without moving or reloading anything. On the Stowage skin the same plan is drawn in the bay plan, where a model that isn't loaded shows as a dashed booked box. If the router running now was started single, the Models tab offers Restart router, which unloads whatever is loaded.
2. To run one model on a different llama.cpp build than the router's, use the Build selector in its editor (shown once there is more than one build to choose). follow the router is the default; picking an installed build pins the model to it (model_builds in config.json). A build other than the router's runs the model in its own llama-server process on its own port, so clients reach it at its own endpoint and Embers can't use it. Changing the build unloads the model, and if a pinned build is later uninstalled the editor says Build gone and the model won't load until you pick another.

1. While a model loads, its row shows loading N% and a bar. The router streams llama.cpp's load progress on /models/sse (weights, then a draft model and a vision projector when the model has them), and the panel listens only while a load is in progress. An older router without that stream shows elapsed seconds instead.
2. To control reasoning, use the Thinking row in the editor: Auto (the chat template decides), On or Off, an effort level when the template has them (see below), and a budget (unlimited, 512, 2k or 8k tokens). It writes reasoning, reasoning-effort and reasoning-budget to the model's section; Auto and unlimited remove the key, so llama.cpp's default applies. The note below it comes from the GGUF's chat template: a thinking switch means Off really works, while a template that only shows thinking tags may keep thinking. The row is hidden for models whose template shows no reasoning at all, unless something is already set. The effort list holds only the levels the template knows (Qwen3.8: low, medium, xhigh), read from the GGUF's chat template; a level the template renames (Qwen3.8 turns high into xhigh) is left out, like llama.cpp's own capability check does. It needs a llama.cpp build with --reasoning-effort.
3. To turn a vision model's projector off, click Vision: Off. That removes the mmproj line from its section and keeps the path in vision_off in config.json, so the projector's VRAM is free and the file stays on disk. Rescans and downloads leave it off; On puts the line back (and says so if the file has gone). Both switches apply at once: a loaded model is unloaded, after you confirm, and its next load uses the change.
4. To check a model's chat template, click Check on the Template row. LlamaForge compares the template inside your GGUF with two references: the same file in the Hugging Face repo you downloaded it from (it reads only that file's header, never the weights), and the base model named on the repo's card (chat_template.jinja, else tokenizer_config.json). A stale template is the usual cause of broken tool calls or thinking that won't turn off.
   • The repo has a newer template: the file was re-uploaded with template fixes. Use the repo's template saves it to chat-templates/ and points chat-template-file at it, so llama.cpp uses it instead of the embedded one. The GGUF is not touched. What changed shows the diff.
   • Differs from the base model: only a hint. GGUF makers often change templates on purpose (unsloth adds fixes), so Try the base template only if tool calls or thinking misbehave.
   • While an override is set, the row says which file is in use, and Use the file's own removes it. Like the other switches, applying unloads a loaded model after you confirm.
   • It works for models downloaded through LlamaForge or sitting in the Hugging Face cache; other files have no repo to compare against, and the row says so.
5. To see the sampler settings a model's makers recommend, click Recommended on the Samplers row. LlamaForge reads generation_config.json from the repo you downloaded the GGUF from, else from the base model on its card (GGUF repos rarely carry it), and maps temperature, top_p, top_k, min_p, repetition_penalty and presence_penalty onto temp, top-p, top-k, min-p, repeat-penalty and presence-penalty.
   • Values that already match yours are dimmed (as numbers, so 1.0 matches 1); hover a value to see yours, including whether it comes from [*]. The source repo is linked.
   • Use these writes only the values that differ into the model's own section. Like the other switches, it unloads a loaded model after you confirm.
   • A config with do_sample: false means greedy decoding, shown as temp 0. Recommendations written only in a model card's prose aren't read.
6. LlamaForge watches for re-uploads. GGUF makers often replace a file in a release's first weeks (template fixes, broken quants), and nothing tells you. Once a day, when you open the Models tab, it asks Hugging Face for the current sha256, size and last commit of each model's file (one small request per model, no downloads).
   • re-uploaded on a row: the repo's file differs from yours. Open the model to see when it changed and the commit message, then Re-download. The new copy downloads next to the old one and replaces it only when complete; a loaded model is unloaded first.
   • Files in the Hugging Face cache are compared by sha256, exactly. LlamaForge's own downloads have no hash on record, so a different size counts as re-uploaded. A newer commit at the same size shows a softer note (often metadata only); the Template row's Check shows whether the chat template changed.
   • For a file in the Hugging Face cache, the note gives the hf download command instead of a button, so the cache stays consistent.
7. To see inside a model, click X-ray on its info card. LlamaForge reads the GGUF header (nothing is loaded) and draws one bar per layer: the embedding, every block, then the output head. Each bar is split into what the layer holds (attention, FFN, MoE experts, the shared expert, SSM, per-layer inputs, an MTP head, norms), and the part that stays in system RAM under the model's settings is hatched.
   • The summary gives the weights on the GPU and in system RAM, using the same placement rule as the fit check: the embedding always stays in RAM, n-gpu-layers offloads the last N blocks, and cpu-moe, n-cpu-moe and override-tensor pull tensors back. A model with a tied output (no output.weight) also gets a GPU copy of the embedding, shown on its own.
   • The chips rank the quant types by bytes. Mixed quants show up here: a block whose bar is longer than its neighbours was given a bigger type.
   • Drag the GPU layers slider to see what moves at another setting, and for a MoE the Experts on CPU slider (--n-cpu-moe: the first N blocks keep their experts in system RAM). A line says whether the weights fit your VRAM, keeping 10% (at least 1 GB) back for the KV cache and buffers. Dragging is a what-if: Back to N returns to the model's own values.
   • Load it with shows the flags for the view, such as -ngl 42 --n-cpu-moe 4, with Copy. Fit my VRAM moves the sliders to where the most weight fits: for a MoE, every layer on the GPU with the fewest blocks' experts in RAM. Apply to this model asks first, then writes the two settings into the model's models.ini section, replacing any other spelling of them (ngl, cpu-moe...) and leaving the rest, comments included, as it was. A loaded model is unloaded, so the next load uses them.
   • Compare with lists the other quants in the model's Hugging Face repo. Pick one and LlamaForge reads only its GGUF header over an HTTP Range request (a few MB, none of the weights) and draws both stacks side by side, on one scale and under the same settings and GPU layers. The line above them says what the other quant costs: the weights, and how much more or less lands on the GPU and in system RAM. It needs the file to have come from Hugging Face, through LlamaForge or the HF cache.
   • Share image turns the view on screen, slider included, into a 1200x630 card with the model's name, architecture, quant mix and GPU / RAM split, but no paths. Download image or Copy image it. With Compare with on, it makes the two-up card instead: both quants side by side on one scale, with what the other one costs (weights, GPU, system RAM) as the headline.

Screenshot

Reference

Troubleshooting

If the editor shows "Could not read knobs from llama-server --help", server_bin in config.json is missing, wrong, or the binary failed to run (missing DLLs is common on Windows). Fix the path from the Setup tab or config.json directly — the schema is retried automatically on the next open, no restart needed. If --help runs but returns no arguments, the help text format wasn't recognized; check the binary is actually llama-server and not a different tool.

See also models.ini Format for the on-disk file this editor writes to, and config.json Reference for server_bin and ui_mode.