Skip to Content
Configuration ReferenceOption reference

Option reference

The CLI flag uses kebab case and the TOML key uses snake case. This page groups the public surface; run skippy serve --help for the exact accepted values and current defaults.

Use this reference after choosing a serving behavior. Start with automatic defaults, inspect them with --print-effective-config, then override only the controls supported by a measurement or deployment requirement.

[model]

TOML keyPurpose
model, model_pathSelect a catalog, Hugging Face, or local source
model_idSet the public model identifier where supported
quant, checkpoint_imatrixConfigure supported on-load checkpoint quantization
mmprojSelect a multimodal projector
mmap, mlockControl model weight mapping and page locking

[execution]

TOML keyPurpose
deviceSelect a backend device reported by skippy doctor
n_gpu_layersControl GPU layer placement
threads, threads_batchSet CPU threads for generation and prompt processing
batch_size, microbatch_sizeSet native batch token limits
split_mode, main_gpuConfigure supported multi-GPU placement

[kv]

TOML keyPurpose
ctx_sizeSet the live context capacity
kv_cache_type_k, kv_cache_type_vSelect f16, q8_0, or q4_0 live KV precision
kv_cache_offloadOffload supported KV work to the backend
flash_attentionSelect automatic, enabled, or disabled Flash Attention
cache_idle_slotsLimit retained idle sessions

[cache]

TOML keyPurpose
prefix_cacheSelect auto, on, off, or record
prefix_cache_budgetSet the resident retention target
prefix_cache_max_entries, prefix_cache_min_tokensBound retained entries and minimum useful prefixes
prefix_cache_ramEnable the optional host-RAM tier above disk
kv_cache_disk, kv_cache_disk_dirEnable and locate durable prompt state
kv_cache_min_freeReserve free space on the cache volume

[scheduling]

TOML keyPurpose
generation_concurrencySet active generation lanes
continuous_batchingServe concurrent lanes in the iteration scheduler
generation_queue_capacityBound queued requests
generation_admission_timeout_secsBound queue wait time
prefill_chunk_size, prefill_chunk_policyConfigure chunked prefill

[sampling]

Request defaults include temperature, top_k, top_p, min_p, typical_p, repetition penalties, seed, stop, logit_bias, sampler order, and request_max_tokens. Explicit request values override these defaults.

[chat]

Chat behavior includes chat_template, system_prompt, reasoning controls, grammar, json_schema, Jinja rendering, and request-template policy. Treat allow_request_chat_template as a trust decision for every caller that can reach the server.

[speculative]

Native MTP, draft-model, n-gram, DFlash, and adaptive speculation controls. Start with speculative_strategy = "auto"; explicit strategies must be compatible with the admitted target model and runtime.

[media]

Multimodal projector and request media limits. Model loading uses mmproj in [model]; [media] controls request-side media policy.

[api]

Public bind address, response limits, compatibility guardrails, prompt mode, and context compaction. Common controls include bind_addr, default_max_tokens, guardrails, compact, compact_trigger_percent, and compact_target_percent.

[distributed]

Prepared stage plan, ordered topology, binary stage transport, worker-only mode, activation codec, flow control, and downstream timeouts. Split identity and tensor closure remain owned by the generated stage plan.

[runtime]

Native runtime bundle roots, cache location, release selection, and selection policy.

[diagnostics]

Human/JSON output, debug logs, metrics export, telemetry level, and startup timeout.

[network]

Wire-delay, bandwidth, jitter, and stall simulation for controlled distributed experiments.

The repository’s canonical field-by-field table is docs/SERVING_SETTINGS.md. Keep that source and this public index consistent when adding or retiring a serving option.

Last updated on