Skip to Content
Running ModelsRuntime and hardware

Runtime and hardware

A native runtime bundle contains the backend-specific code that executes an admitted model. Skippy selects a bundle for the machine’s operating system, architecture, accelerator backend, toolkit, and Skippy ABI.

Use CPU for broad compatibility, Metal on supported Apple hardware, and CUDA, ROCm, or Vulkan when the installed accelerator and published runtime support that backend. Inspect the machine and available bundles before changing a serving plan:

skippy doctor skippy runtime list skippy runtime list --available

Install the recommended compatible runtime, or name a backend explicitly:

skippy runtime install skippy runtime install metal

Skippy release lanes cover CPU, Metal, CUDA, ROCm, and Vulkan variants where published for the target. A backend build is not evidence that the model fits: context size, live KV precision, concurrency, runtime workspaces, and other processes all consume memory.

How runtime selection works

The CLI evaluates installed candidates against the host, backend, release metadata, native ABI, and recorded file hashes. An explicitly selected runtime bundle still must pass these checks. skippy doctor reports rejected candidates and the reason they were not selected.

Model weights, live KV state, compute workspaces, and active generation lanes share finite device or host memory. Weight quantization only changes the model weights. It does not automatically quantize the live KV cache.

When a model fails to fit, first reduce context or concurrency, choose a smaller quantization, or select a smaller model. Use pipeline parallelism when the verified model must be divided across explicit workers.

Hardware and runtime reference

GoalControl
Inspect host and backend supportskippy doctor
List installed runtimesskippy runtime list
List published compatible runtimesskippy runtime list --available
Install the recommended runtimeskippy runtime install
Install a named backendskippy runtime install BACKEND
Select devices--device
Control GPU layer placement--n-gpu-layers
Set live KV precision--kv-cache-type-k, --kv-cache-type-v
Inspect the resolved launch--print-effective-config

If selection fails, run skippy doctor before installing another bundle. If loading fails for memory, reduce context or concurrency before changing the runtime.

Last updated on