PGPradeep Gudipati

Open source · Rust · llama.cpp

Swap models. Keep momentum.

One OpenAI-compatible endpoint for the GGUF models on your machine. Stop coordinating llama-server processes, ports, and GPU memory by hand.

View on GitHubRead the documentation
  • One resident model
  • OpenAI-compatible API
  • 100% local
GGUF Switchboard routing local language models through a central control plane

Request lifecycle

One route. Deliberate model changes.

A request names the model it needs. The switchboard protects in-flight work, changes the resident model, and keeps the endpoint stable.

  1. 01Request model B
  2. 02Drain model A
  3. 03Load model B
  4. 04Forward request
GGUF Switchboard lifecycle showing model selection, draining, loading and request forwarding
The switch lifecycle preserves one stable API while the resident model changes.

Control plane

Built for constrained local inference.

The narrow single-GPU scope makes switching predictable and the operational model easy to understand.

Single-slot hot swap

Drain in-flight requests, unload the active model, load the requested GGUF, then forward the call.

OpenAI-compatible surface

Use chat, completions, embeddings, responses, audio endpoints, streaming, and model discovery from existing clients.

Memory-aware operation

Evict under system memory pressure and retry context sizing only when llama.cpp reports an out-of-memory condition.

Operational visibility

Inspect Swagger UI, Prometheus metrics, health status, the live registry, and built-in usage history.

Architecture layers connecting OpenAI clients to GGUF models through GGUF Switchboard and llama.cpp
Existing OpenAI clients connect to one endpoint; llama.cpp remains the inference backend.

Product evidence

Inspect the system you are running.

The live registry, API surface, and model state remain visible instead of disappearing behind a desktop wrapper.

A live run: a curl request against the switchboard, nvidia-smi showing GPU memory shift as the resident model is evicted and swapped, and the streamed JSON response.

Operational boundary

Know exactly what it is built for.

Experimental software for development machines and single-GPU home labs.

Trusted LAN deployment; it is not an internet-facing multi-tenant gateway.

One model stays resident at a time, with controlled swaps between requests.

llama.cpp required with llama-server installed and GGUF model files available locally.

Start locally

Clone, build, discover, run.

git clone https://github.com/pradeepgudipati/gguf-switchboard.git
cd gguf-switchboard
cargo build --release
./target/release/gguf-switchboard discover-models ~/models -o models.toml
./target/release/gguf-switchboard config.toml

Open source

Take control of your local model runtime.

Review the architecture, compatibility notes, and source before running it on your machine.

View on GitHubRead the documentation