Introducing Skippy
Skippy is an AI inference engine for local and distributed workloads. It is the inference engine that powers Mesh LLM .
Built on llama.cpp, Skippy adds production-focused model and runtime
management, OpenAI- and Anthropic-compatible APIs, explicit pipeline
parallelism, multi-tier KV caching, memory-aware scheduling, and
hardware-specific optimizations.
Skippy supports CPU inference and GPU acceleration through Metal, CUDA, ROCm, and Vulkan. Run it on a single machine or divide one verified model across an ordered set of workers. A single CLI and configuration model takes you from local inference to multi-machine serving.
Serve your first model
Build the development product and start a recommended model:
just build
./target/debug/skippy models recommended
./target/debug/skippy serve --model Qwen3-0.6B-Q4_K_M --promptThe prompt and HTTP API are available together. From another terminal:
curl http://127.0.0.1:9337/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen3-0.6B-Q4_K_M","messages":[{"role":"user","content":"Hello"}]}'How Skippy serves a request
Skippy resolves the model and a compatible native runtime, plans memory for weights and live context state, and opens one serving lifecycle for both API styles. Each request is tokenized, admitted by the scheduler, evaluated by the local runtime or ordered pipeline stages, and sampled by stage zero.
The same lifecycle owns model identity, prefix-cache compatibility, runtime selection, request limits, and telemetry. That keeps local and distributed serving behavior consistent even when the model is divided across machines.
Learn
- Install and verify Skippy
- Run local, Hugging Face, and multimodal models
- Split a model across explicit stages
- Tune prompt-prefix and KV storage
- Review every serving setting
- Contribute safely
Developed by
