Skip to Content
Getting StartedMaking inference requests

Making inference requests

Skippy exposes OpenAI-compatible and Anthropic Messages APIs on the same address and model lifecycle. Both APIs submit work to the same loaded model, scheduler, cache, and sampling pipeline.

Use the OpenAI-compatible API for clients built around Chat Completions. Use the Anthropic-compatible API for clients built around Messages. Wait for the server ready event before using either endpoint.

OpenAI chat completions

curl http://127.0.0.1:9337/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen3-0.6B-Q4_K_M", "messages": [{"role": "user", "content": "Reply with OK"}], "max_tokens": 32 }'

Anthropic Messages

curl http://127.0.0.1:9337/v1/messages \ -H 'Content-Type: application/json' \ -H 'anthropic-version: 2023-06-01' \ -d '{ "model": "Qwen3-0.6B-Q4_K_M", "messages": [{"role": "user", "content": "Reply with OK"}], "max_tokens": 32 }'

Use skippy serve --model ... --prompt when you want an interactive prompt and the HTTP API in the same process. Use skippy prompt --endpoint URL to connect to an already-running server.

How requests are processed

The model value must match an identifier returned by GET /v1/models. Skippy renders the selected chat template, tokenizes the complete conversation, applies the request or server sampling defaults, and admits the generation to an available lane. Explicit request values take precedence over serving defaults.

The response reports prompt and generated token usage. Streaming requests use the same lifecycle but emit incremental events. Invalid model IDs, unsupported request features, conflicting token limits, and context overflow return an API error rather than silently changing the request.

Endpoint reference

APIEndpointRequired request fields
OpenAI Chat CompletionsPOST /v1/chat/completionsmodel, messages
Anthropic MessagesPOST /v1/messagesmodel, messages, max_tokens
Model discoveryGET /v1/modelsNone

Send Content-Type: application/json for request bodies. Anthropic-compatible clients should also send an anthropic-version header. Use the model-specific request fields for sampling, reasoning, tools, structured output, streaming, and output limits.

Deterministic caching

Skippy can reuse the model state produced for a shared prompt prefix. This avoids recomputing repeated system prompts, tool definitions, documents, and conversation history.

prompt_cache_key is a stable identifier chosen by your application. It places related requests in the same cache namespace. A key can isolate tenants or agent configurations, keep compatible requests together, and deliberately start a fresh namespace when a prompt or toolset changes.

The key does not identify the cached prompt and does not guarantee a hit. Requests must still use a compatible model and runtime and share an identical leading token sequence. Reuse ends at the first token that differs.

You can omit prompt_cache_key for normal prefix caching. Requests without one use the default cache namespace and can still reuse matching prefixes.

OpenAI-compatible requests

Add prompt_cache_key to a chat completion request:

{ "model": "Qwen3-0.6B-Q4_K_M", "prompt_cache_key": "support-agent:v1", "messages": [ { "role": "system", "content": "You are a support agent. Follow the company support policy." }, { "role": "user", "content": "How do I reset my password?" } ] }

A later request can use the same key and system prompt with a different user message:

{ "model": "Qwen3-0.6B-Q4_K_M", "prompt_cache_key": "support-agent:v1", "messages": [ { "role": "system", "content": "You are a support agent. Follow the company support policy." }, { "role": "user", "content": "How do I update my email address?" } ] }

Skippy can reuse the matching system-prompt prefix and prefill only the changed tail.

Anthropic-compatible requests

The Anthropic-compatible API accepts the same top-level field:

{ "model": "Qwen3-0.6B-Q4_K_M", "prompt_cache_key": "support-agent:v1", "max_tokens": 256, "system": "You are a support agent. Follow the company support policy.", "messages": [ { "role": "user", "content": "How do I reset my password?" } ] }

Anthropic cache_control controls retention behavior. It does not select the cache namespace or replace prompt_cache_key.

Choose a stable key

Use caseExample
One agent configurationsupport-agent:v1
Tenant and agent isolationtenant-42:support-agent:v1
Versioned tool definitionsresearch-agent:tools-v3
Default shared namespaceOmit prompt_cache_key

Change the key when the isolation boundary or stable prompt configuration changes. Do not use a unique request ID, timestamp, or random value because a new value prevents reuse from earlier requests. Do not include secrets in the key.

Skippy trims surrounding whitespace and treats an empty key as absent.

Last updated on