Open source · Rust · llama.cpp
One OpenAI-compatible endpoint for the GGUF models on your machine, with deliberate compatibility, model management, and single-GPU operations instead of hand-managed llama-server processes, ports, and GPU memory.
Request lifecycle
A request names the model it needs. The switchboard uses kind-aware routing, protects in-flight work, changes the resident model, and rolls back safely when a load cannot complete.
Control plane
The narrow single-GPU scope makes switching predictable: one resident model, memory-aware eviction, idle priority loading, and optional automatic GPU layers.
Serve supported OpenAI Chat Completions (POST /v1/chat/completions) and Completions (POST /v1/completions), plus Embeddings, Models, and usage endpoints. Compatibility is best-effort against the installed llama.cpp build; Responses and Audio are partial and backend or model dependent. Also accept streaming and non-streaming Anthropic Messages API requests.
Responses is partial: it translates function tools and calls with strict SSE events through Chat Completions; built-in and hosted tools are not supported. Audio is partial and proxied only when llama-server exposes the requested endpoint. Tool behavior remains model-dependent, so the selected GGUF and its chat template still matter.
Search Hugging Face, inspect files, and pull models with hardware-aware quant selection, checksum and GGUF validation, and a resumable download fallback. A successful pull registers the model and refreshes a running server. In the current development build, a post-pull speed test measures prompt and generation throughput, with --no-bench as the opt-out.
Install as a system-wide Linux service under a dedicated ggs service account. The current development build adds shared model-directory permissions. Refresh the live registry when the available models change.
Drain the single slot, swap or roll back the resident model, route by model kind, evict under memory pressure, retry with OOM context fallback, and load idle priority models before they are needed.
Use optional automatic GPU layers alongside Swagger UI, Prometheus, health status, the live registry, portable models.json metadata, and usage history.
Product evidence
The live registry, API surface, and model state remain visible instead of disappearing behind a desktop wrapper.
Operational boundary
Experimental software for development machines and single-GPU home labs.
Trusted LAN deployment; it is not an internet-facing multi-tenant gateway.
One model stays resident at a time, with controlled swaps between requests.
llama.cpp required with llama-server installed and GGUF model files available locally.
Quick setup
Run ./deploy.sh for the primary quick setup. It installs GGUF Switchboard system-wide so all users can access the installed project and models directory. In the current development build, the models directory has shared writable permissions for all intended users.
git clone https://github.com/pradeepgudipati/gguf-switchboard.git
cd gguf-switchboard
./deploy.shOpen source
Review the architecture, compatibility notes, and source before running it on your machine.