Runtime and hardware
A native runtime bundle contains the backend-specific code that executes an admitted model. Skippy selects a bundle for the machine’s operating system, architecture, accelerator backend, toolkit, and Skippy ABI.
Use CPU for broad compatibility, Metal on supported Apple hardware, and CUDA, ROCm, or Vulkan when the installed accelerator and published runtime support that backend. Inspect the machine and available bundles before changing a serving plan:
skippy doctor
skippy runtime list
skippy runtime list --availableInstall the recommended compatible runtime, or name a backend explicitly:
skippy runtime install
skippy runtime install metalSkippy release lanes cover CPU, Metal, CUDA, ROCm, and Vulkan variants where published for the target. A backend build is not evidence that the model fits: context size, live KV precision, concurrency, runtime workspaces, and other processes all consume memory.
How runtime selection works
The CLI evaluates installed candidates against the host, backend, release
metadata, native ABI, and recorded file hashes. An explicitly selected runtime
bundle still must pass these checks. skippy doctor reports rejected
candidates and the reason they were not selected.
Model weights, live KV state, compute workspaces, and active generation lanes share finite device or host memory. Weight quantization only changes the model weights. It does not automatically quantize the live KV cache.
When a model fails to fit, first reduce context or concurrency, choose a smaller quantization, or select a smaller model. Use pipeline parallelism when the verified model must be divided across explicit workers.
Hardware and runtime reference
| Goal | Control |
|---|---|
| Inspect host and backend support | skippy doctor |
| List installed runtimes | skippy runtime list |
| List published compatible runtimes | skippy runtime list --available |
| Install the recommended runtime | skippy runtime install |
| Install a named backend | skippy runtime install BACKEND |
| Select devices | --device |
| Control GPU layer placement | --n-gpu-layers |
| Set live KV precision | --kv-cache-type-k, --kv-cache-type-v |
| Inspect the resolved launch | --print-effective-config |
If selection fails, run skippy doctor before installing another bundle. If
loading fails for memory, reduce context or concurrency before changing the
runtime.