Usage Stats

Per-model token counts, run counts, and generation speed, with a daily activity chart.

What it does

The dashboard itself never sees inference traffic — clients talk to the llama.cpp router directly — so it cannot provide per-client or per-IP accounting. It records per-model aggregates only. llama.cpp's own Prometheus counters reset on every router restart and keep no per-model history. backend/stats.py's StatsTracker works around both limits with a background poller (run_forever(), every POLL_SECS = 5 seconds) that:

1. Calls the router's /models endpoint to learn whether it's up and which models are loaded: one by default (--models-max 1), several when multi-model is on. Each model has its own /metrics?model=<id>, so token deltas stay attributed to the right one. A model pinned to another build runs in its own llama-server process outside the router, and the poller scrapes that process's /metrics directly.
2. Scrapes /metrics?model=<id> for each loaded model's cumulative llamacpp:prompt_tokens_total and llamacpp:tokens_predicted_total counters, diffs them against the previous poll, and adds the delta to that model's running totals — only when that model stayed loaded across both polls, and only when the delta is non-negative (a drop means the router restarted and the counters reset, so it's treated as zero rather than subtracted).
3. Separately polls vLLM's /metrics (vllm:prompt_tokens_total / vllm:generation_tokens_total) on vllm_port the same way, best-effort and silent on failure, so vLLM usage lands in the same per-model store.
4. Persists everything to stats.json at the repo root via an atomic write (write to .tmp, then os.replace), throttled to at most once every FLUSH_SECS = 15 seconds while dirty.

Each model's record in stats.json ({"models": {...}, "daily": {...}, "first_seen": ...}) tracks prompt, generated, loaded_secs, gen_secs, runs, and last_used. A "run" increments whenever generation transitions from idle to active (_idle flag), which approximates a request count without the router exposing one directly. The Tok/s column (decode_tps) is llama.cpp's own generation speed (llamacpp:predicted_tokens_seconds, timed per slot by the server) averaged over every token generated, kept as dec_tokens / dec_secs; it reads "-" until a model has generated under LlamaForge 0.18.11 or later. The older avg_tps (generated / gen_secs, where gen_secs only counts poll windows that had generation) is still in the API, but parallel requests and late polls inflate it, so the UI no longer shows it. Daily totals are kept for DAILY_KEEP = 30 days and trimmed on each write.

LAN sharing and the API key. The router binds to 127.0.0.1 (local only) by default. The Network Access panel — backed by GET/POST /api/network — can set its canonical LAN scope (0.0.0.0) so other devices can reach http://<lan-ip>:<port>/. Every newly configured LAN router requires a usable key and LlamaForge-owned starts fail closed without one; router_ctl.start() passes the selected key to llama-server as --api-key. The dashboard's own proxy calls send the configured key as Authorization: Bearer <key> when applicable, and external clients must use the current key. Ordinary state is redacted: Client/Agent previews and generated-key responses are deliberate, no-store exceptions rather than ambient key visibility.

How to use it

1. Open the Stats tab. Under the token-scale line sits each GPU's memory: in the Stowage skin the same bay plan as the Models page (every loaded model stowed at its measured size, used + free = total), in Hearth and Classic a tile with a segment meter. Next to them, a System RAM meter shows used and free host memory (used leaves out cache the OS can reclaim); it shows even without a GPU and hides only when the OS won't report memory.
2. The row of figures shows total tokens processed, tokens generated, cumulative inference time, distinct models used, approximate run count, and the most-used model.
3. Live Throughput shows each loaded model's generation and prompt-eval tok/s and active request count in real time (polled every 4 seconds while the tab is open). In Stowage, each model's square is the colour of its container in the bay plan above it.
4. Activity is a stacked prompt/generated bar chart; toggle 14d / 30d to change the window.
5. Per-model Usage lists every model with logged usage — total tokens, generation tok/s as llama.cpp timed it ("-" until measured), run count, time loaded, and when it was last used. Click a column chip to sort by it.
6. Click Share my rig for a 1200x630 card (and markdown) of your GPUs, RAM, OS and your five most-used models with their measured tok/s and token counts, to post wherever you like. Model names lose any folder part; no paths, keys or hosts are on it (GET /api/rig).
7. Click Reset stats to zero the whole store (POST /api/stats/reset) — this is destructive and cannot be undone.
8. To share the router on your LAN: go to Setup → Network Access, select local-network access, then keep a usable existing key or explicitly generate or replace one before Apply & Restart Router. The dashboard remains loopback-only.

Screenshot

Reference

Troubleshooting

If "Live Throughput" shows the router offline but models load fine, confirm the router process is actually running on router_port — router_running comes from GET /api/network, and the poller re-baselines (self._prev = None) whenever /models fails to answer. If per-model totals look stuck at zero for a model you know ran, check that it wasn't reloaded mid-generation: a token delta is only counted when the same model ID was loaded on the previous poll, so a reload during a burst discards that window. If external clients get 401/403 after enabling LAN access, verify they're sending Authorization: Bearer <key> with the current key from the deliberate configuration/generation action.

See also Setup for the Network Access panel this page's LAN/API-key section documents, and Models & Tuning for per-model configuration.