PGPradeep Gudipati

Open source · Rust · llama.cpp + vLLM

One machine. Many local models. One API.

Run GGUF through llama.cpp and SafeTensors through vLLM behind one OpenAI-compatible endpoint. Your coding tools select a model; the switchboard manages the local runtime, process, and resident-model change.

Install GGUF SwitchboardExplore documentation
  • One resident model
  • GGUF + SafeTensors
  • OpenAI-compatible endpoint
GGUF Switchboard

Developer problem

Stop coordinating local runtimes by hand.

Coding agents, embedding workflows, and SDK clients may need different local models. They should not each own a process, port, runtime command, and slice of constrained GPU memory.

  1. 01Client requests a model
  2. 02Select llama.cpp or vLLM
  3. 03Drain and change the resident model
  4. 04Forward through the same API
GGUF Switchboard lifecycle showing model selection, draining, loading and request forwarding
The switch lifecycle preserves one stable API while the resident model changes.

Control plane

A control plane for constrained local inference.

GGUF Switchboard combines runtime selection, model lifecycle, hardware-aware acquisition, load planning, and observable API behavior without becoming a cloud-provider router.

One endpoint, two runtime paths

Connect local coding tools and SDKs to one OpenAI-compatible base URL. GGUF models run through llama.cpp; SafeTensors models run through vLLM.

Switch by requested model

Name a registered model in the request. The scheduler selects its backend, drains in-flight work when needed, changes the resident model, and keeps the client endpoint stable.

Find models that fit

Search Hugging Face for GGUF or SafeTensors models and compare fit, speed, balance, and precision signals against detected hardware before downloading.

Plan around constrained VRAM

Estimate model, context, KV-cache, quantization, and runtime allocations before load. Bounded OOM fallback can retry with a smaller context after an OOM-class failure.

Verify tool-calling behavior

Use the Conformance Console and load-time probes to diagnose tool-calling and chat-template behavior before routing agent workloads to a model.

Inspect the runtime

Use Swagger UI, health and model APIs, Prometheus metrics, portable model metadata, and persistent usage history to understand what the local service is doing.

Architecture connecting AI clients through GGUF Switchboard to llama.cpp for GGUF and vLLM for SafeTensors
Existing clients connect to one endpoint; the configured model selects the llama.cpp or vLLM runtime path.

Integrations

Connect the tools already in your workflow.

Use a normal OpenAI-compatible base URL, then follow the focused guide for your coding tool or SDK.

OpenCode

Configure OpenCode to use GGUF Switchboard as a local OpenAI-compatible provider.

Open OpenCode guide

Cursor

Configure Cursor to use GGUF Switchboard as a local OpenAI-compatible provider.

Open Cursor guide

Cline

Configure Cline to use GGUF Switchboard as a local OpenAI-compatible provider.

Open Cline guide

Continue

Configure Continue to use GGUF Switchboard as a local OpenAI-compatible provider.

Open Continue guide

OpenAI SDK

Configure OpenAI SDK to use GGUF Switchboard as a local OpenAI-compatible provider.

Open OpenAI SDK guide

Product evidence

Inspect the system you are running.

The live registry, API surface, and model state remain visible instead of disappearing behind a desktop wrapper.

GGUF Switchboard API dashboard with the model selector open and runtime status visible
The main dashboard keeps model selection, runtime health, hardware utilization, API discovery, and the Conformance Console in one operational surface.
GGUF Switchboard dashboard header before a model is loaded
Before a model is loaded
GGUF Switchboard dashboard header with a model loaded and hardware status visible
After a model is loaded
Explore the OpenAI-compatible API, select a configured local model, and inspect live runtime state from the browser console.
A live run: a curl request against the switchboard, nvidia-smi showing GPU memory shift as the resident model is evicted and swapped, and the streamed JSON response.

Test tool-call conformance before agent use.

The Conformance Console sends controlled tool-call probes to a local or external OpenAI-compatible model, records the response, and shows whether the model and chat template produce usable function calls. It diagnoses tested tool-calling behavior; it does not certify universal API compatibility.

GGUF Switchboard Conformance Console results for tool-call probes
Conformance results make tool selection, argument shape, and failure details inspectable before a model is assigned to an agent workflow.
Review the Conformance ConsoleCheck API compatibility

Operational boundary

Know exactly what it is built for.

Primary platform is Linux with NVIDIA GPUs and CUDA; macOS and Windows via WSL2 use separate guides.

Trusted LAN deployment only. There is no built-in authentication, so do not expose the service directly to the public internet.

Single slot keeps one model resident at a time; this is switching, not concurrent multi-model serving.

Estimated fit can still fail because context, KV cache, quantization, and runtime allocations vary.

Compare adjacent tools

Quick setup

Install, pull a model, send a request.

The primary Linux path installs both supported engines. Follow the platform guides for macOS or Windows through WSL2.

git clone --branch main https://github.com/pradeepgudipati/gguf-switchboard.git
cd gguf-switchboard
./deploy.sh

Open source

Give every local AI tool one endpoint.

Install the switchboard, then connect your coding agent or SDK with the focused integration guide.

Install GGUF SwitchboardExplore documentation