Serving configuration
Serving configuration defines how Skippy loads a model, allocates resources,
schedules requests, exposes APIs, and records reusable state. Every supported
operator control is available through skippy serve --help.
Use CLI options for temporary overrides and experiments. Use environment
variables for process-level deployment settings. Use --settings serve.toml
for a complete, reviewable serving configuration.
Create a settings file
version = 1
[model]
model = "Qwen3-0.6B-Q4_K_M"
mmap = true
[execution]
n_gpu_layers = -1
threads = 4
batch_size = 512
[kv]
ctx_size = 8192
kv_cache_type_k = "f16"
kv_cache_type_v = "f16"
[cache]
prefix_cache = "auto"
kv_cache_disk = "off"
[scheduling]
generation_concurrency = 2
continuous_batching = true
[api]
bind_addr = "127.0.0.1:9337"
default_max_tokens = 2048Run with an override and inspect the resolved launch:
skippy serve --settings serve.toml --temperature 0.2
skippy serve --settings serve.toml --print-effective-configHow settings resolve
Settings resolve in this order:
- Explicit command-line options.
SKIPPY_SERVE_<OPTION_NAME>environment variables.- Values in the serving settings file.
- Admitted model/stage policy and built-in defaults.
Paths in a settings file are relative to that file. Paths supplied on the CLI remain relative to the working directory. Unknown keys, wrong sections, invalid values, and conflicting model sources are errors.
Environment variables use the CLI representation and the
SKIPPY_SERVE_<OPTION_NAME> form. For example,
SKIPPY_SERVE_THREADS=8 sets [execution].threads; --threads 4 overrides it.
Configuration surfaces
| Surface | Best for | Example |
|---|---|---|
| CLI | One run or a measured override | --temperature 0.2 |
| Environment | Deployment-level process settings | SKIPPY_SERVE_THREADS=8 |
| TOML | Versioned complete serving policy | --settings serve.toml |
| Admitted stage plan | Planner-owned split identity and tensor closure | --config stage-0.json |
Continue to the option reference.