Configure local model serving

Task

Configure llama.cpp, Ollama, or vLLM so a local model serves Ethos agent turns with a real context window, a working prefix cache, and tool calls that arrive as tool calls.

Result

The server keeps the model resident, reuses the prompt prefix between turns, accepts a context large enough for your personality (the directory of files that decides an agent's tools, memory, and model) — and you verify all of it with ollama ps, ethos bench context, and ethos personality show.

This page is about making Ethos work well on a local server. For general local-LLM setup, use the runtime's own docs: llama.cpp server, Ollama, vLLM.

Prereqs

• A local endpoint wired as the provider — provider: ollama or provider: vllm per Configure an LLM provider.
• Enough VRAM for the model at the context length you configure. KV cache is real memory: tens of thousands of context tokens can cost several GB on top of the weights.

Steps

1. Pick the model family for tool discipline, not parameter count

Tool-calling reliability tracks the model family, not the parameter count. In Docker's 21-model, 3,570-case tool-calling evaluation (third-party numbers — not Ethos measurements), qwen3:8B scored 0.933 F1 while llama3.3:70B scored 0.607 — an 8B model beating a 70B one by half again, on tool calling specifically. If your agent fumbles tool calls, try a different family before trying a bigger model.

Then check the context window the model natively supports — advertised context is not native context:

Model	Advertised	Native	How the gap is bridged
Qwen3	131,072	32,768	YaRN rope scaling — quality degrades past native
gpt-oss	131,072	4,096	Extension over a 4k native window

Configure your serving context from the native column. A 131k context on a 32k-native model accepts your tokens and quietly degrades on them.

2. Configure the runtime

Pick one runtime. Ollama is the recommended path — it needs two environment variables and nothing else; choose llama.cpp when you want direct control over cache and quantization flags, vLLM when you need throughput under concurrent requests.

Option A — Ollama (Recommended)

Ollama's default context is small (4,096 on most setups, tiered by VRAM) — far below what an agent turn needs. Raise it and keep the model resident:

OLLAMA_CONTEXT_LENGTH=64000 OLLAMA_KEEP_ALIVE=-1 ollama serve

• OLLAMA_CONTEXT_LENGTH ≥ 64000 — the setting that fixes the 4,096 default. Ethos system prompts plus tool definitions plus history do not fit in 4k.
• OLLAMA_KEEP_ALIVE=-1 — never unload the model, so the prefix cache survives between turns instead of dying with the process's idle timeout.

Verify the context length took effect — do not assume it:

ollama ps

NAME        ID            SIZE     PROCESSOR    CONTEXT    UNTIL
qwen3:8b    500a1f067a9f  6.5 GB   100% GPU     64000      Forever

CONTEXT must show your configured value and UNTIL must show Forever. If CONTEXT still shows 4096, the variable did not reach the server process — restart ollama serve with the variable in its environment, not the client's.

Option B — llama.cpp

llama-server -m <model>.gguf -c 64000 --jinja --cache-reuse 256 --checkpoint-min-step 2048

Flag	Default	Why it matters for agents
--jinja	off	Applies the model's chat template. Without it, chat formatting — and therefore tool calling — is wrong.
--cache-reuse 256	0 (off)	Prefix reuse does nothing until this is set. The default silently recomputes the whole prompt every turn.
--checkpoint-min-step 2048	8192	Checkpoints are only written every N tokens. Agent turns are usually shorter than the 8192 default, so with the default they get no checkpoint at all. Set it below your typical turn size. This is the single most commonly-missed llama.cpp setting for agent workloads.

Expected startup output includes the listen line:

main: server is listening on http://127.0.0.1:8080 - starting the main loop

Point Ethos at it with provider: vllm semantics — any OpenAI-compatible baseUrl works: baseUrl: http://localhost:8080/v1.

Option C — vLLM

vllm serve Qwen/Qwen3-8B --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser hermes

• --enable-prefix-caching — reuses the byte-stable prompt prefix Ethos maintains across turns. The repo's CLAUDE.md local-serving note covers why the prefix stays byte-identical; this flag is the serving side of that contract.
• --enable-auto-tool-choice --tool-call-parser <parser> — the parser must match the model family:

Model family	--tool-call-parser
Qwen2.5 / Qwen3	hermes
Llama 3.x	llama3_json
Mistral	mistral

Run vllm serve --help for the full parser list. The wrong parser fails soft: tool calls arrive as plain text in the response body, so the model appears to narrate JSON instead of calling tools, and every tool-dependent turn dead-ends.

3. Quantize the KV cache — K needs ≥q8, V tolerates coarser

If context memory does not fit, quantize the KV cache asymmetrically (llama.cpp flags shown; the rule is runtime-independent):

llama-server -m <model>.gguf -c 64000 --jinja --cache-reuse 256 \
  -ctk q8_0 -ctv q4_0 -fa on

The rule, with its numbers (measured on Qwen2.5-7B with llama.cpp, third-party): K needs ≥q8 — q8_0 keys preserved 98.02% top-p match, while q4_0 keys collapsed to 11.75%. V tolerates coarser quantization. The keys carry the attention geometry, so they degrade first; the values carry content, and content survives a coarser representation.

Do not confuse KV quantization with weight quantization. Weight quantization is safe for tool calling — a q4 GGUF calls tools fine. The failure people attribute to "4-bit models can't call tools" is usually a quantized K cache, not quantized weights. Conflating the two leads people to run unnecessarily large models.

4. Optional — speculative decoding

If your runtime supports a draft model (llama.cpp --model-draft, vLLM speculative config), turn it on. It is output-identical — the target model verifies every draft token, so quality is unchanged — and delivers 2–3× throughput at concurrency 1–4, which is exactly the single-user local regime. Zero Ethos configuration is involved; it is purely a serving-side win.

Verify

Close the loop against your actual personality, not a synthetic prompt.

Measure what one turn really costs on your setup:

Run it from the project directory your agent works in, or name that directory with --cwd:

ethos bench context --cwd ~/code/my-repo

Static context tax per personality (chars/4 token estimate)
  project context measured in /Users/you/code/my-repo
  personality               soul ch  tools  schema ch  tool_loading_chars  project ch  ~tokens
  engineer                     1184     30      25580                9693      105052    33247
  researcher                   1267     20      16467               16994      105052    30989

project ch is the AGENTS.md/CLAUDE.md block a turn launched in that directory sends. A personality that declares its own fs_reach.workdir is measured in that directory instead. ~tokens is the whole static prefix, project context included. Compare it with the window: a prefix above 40% of it puts that personality in small-window mode.

Then check the personality against the window you just configured:

ethos personality show researcher

## Prompt size
- System-prompt tokens: ~4726 (measured static floor — serialized tool schemas included)
- Project context (AGENTS.md/CLAUDE.md): depends on the working directory — no fs_reach workdir declared, not included above

The character sheet (ethos personality in the CLI reference) prints the personality's identity, routing, memory scope, and toolset — the inputs that make up the static context you just sized the server for. Its ## Prompt size section adds the project context for a personality with a declared fs_reach.workdir. For one without, run ethos bench context in the directory you work in. If you raised OLLAMA_CONTEXT_LENGTH for a personality that didn't fit, ethos bench context followed by ethos personality show <id> is the "does it fit now?" answer.

Small-window mode is decided per personality, on each turn, from that personality's own static prefix in the directory its turns run in. The same measurement sets its tool-result budget and the history a manual /compact reads. One personality can run in small-window mode while another in the same process does not. When a personality engages it, the log says so once:

small-window mode on for personality `bigctx` (static prefix above 40% of the window): static prefix ~27,537 tokens is 43% of the 64,000-token window. Largest: project context (AGENTS.md/CLAUDE.md) (~26,263 tokens). …

Sub-directory context files found later in progressive mode are not part of that prefix. The per-turn compaction gate counts them with the rest of the prompt.

Troubleshoot

• Model replies but never calls tools (vLLM). Wrong --tool-call-parser, or --enable-auto-tool-choice missing. The tool call is in the response text — check the raw completion. Match the parser to the model family per the table above.
• Every turn is slow, even short ones (llama.cpp). --cache-reuse is unset (defaults to 0) or turns are shorter than --checkpoint-min-step (defaults to 8192). Set both.
• ollama ps shows CONTEXT 4096 after configuring 64000. The env var was set in the client shell, not the server's. Stop the server, export OLLAMA_CONTEXT_LENGTH=64000, start ollama serve in that shell (or set it in the service unit).
• [context_window_too_small] context window too small: … The message was NOT sent. The system prompt and tool schemas, plus your message, do not fit the window Ethos is using for this model, so the turn stopped before any request went out. Ethos sizes an Ollama model from ollama ps when the model is loaded. When the model is not loaded, or ollama ps shows no context, Ethos falls back to the model's catalog window, capped at 32,768 tokens, and the startup log says so. If the server really serves more, set OLLAMA_CONTEXT_LENGTH as in step 2, then either load the model before starting Ethos or set contextWindow: <tokens> in ~/.ethos/config.yaml to match. If the window really is that small, shrink the static prefix instead: tool_loading: on in config.yaml, a small_window_toolset in the personality's context_engine_options, or a smaller AGENTS.md/CLAUDE.md in the working directory. ethos bench context prints the static size to compare.
• small-window mode on for personality <id> appears after a /personality switch or a web or gateway turn. That personality's static prefix, usually its project context, is over 40% of the window, so its turns use the compact prelude, index-only memory and skills, a shorter history, and a smaller tool-result budget. Other personalities are unaffected. To keep it out of small-window mode, trim the AGENTS.md/CLAUDE.md in its working directory or fs_reach.workdir, or raise the window. ethos bench context --cwd <dir> shows the project ch figure to compare.
• Garbage output after enabling KV quantization. You quantized K below q8 (-ctk q4_0). Keep K at q8_0 or above; push V down instead.
• Model answers degrade on long sessions despite a 131k window. You are past the model's native window (step 1). Reduce the configured context to the native value, or pick a model whose native window fits your workload.

Numbers and provenance

Serving flags drift with upstream releases — this page's facts are pinned:

Fact	Source	Checked
0.933 vs 0.607 F1	Docker's tool-calling evaluation, 21 models / 3,570 cases (third-party; not reproduced by Ethos)	2026-08
98.02% / 11.75% top-p match	llama.cpp KV-cache quantization measurement on Qwen2.5-7B (third-party)	2026-08
--cache-reuse default 0, --checkpoint-min-step default 8192	llama.cpp server flags current at 2026-08	2026-08
Ollama 4,096 default context, ollama ps CONTEXT column	Ollama releases current at 2026-08	2026-08
Qwen3 32k native / 131k YaRN; gpt-oss 4k native / 131k advertised	Model cards, current at 2026-08	2026-08

If a number no longer matches your runtime's --help output, trust the runtime and file a docs issue.