The problem: starting a model is only the first step
A local model can answer a prompt, yet still require substantial work before a coding tool or application can depend on it. Model files, runtime settings, GPU memory, context length, and API behavior all affect whether the system is usable.
Switching models adds another operational problem. An application should be able to request a registered model without taking responsibility for unloading the previous one, starting the correct runtime, and diagnosing a failed load.
My approach: separate the client from the model lifecycle
I built a control plane around existing inference engines. GGUF uses llama.cpp; SafeTensors uses vLLM. Each backend retains its own execution path, while clients use a stable OpenAI-compatible endpoint.
A request identifies a registered model. When a switch is necessary, the system drains the current model, loads the requested one, and forwards the request. If the switch fails, it attempts to restore the previous model when possible.
The decisions that shaped the system
I used established inference runtimes and focused the product on orchestration. This keeps model execution with the engines built for it, while the switchboard handles model registration, selection, and lifecycle behavior.
Hardware fit is a planning estimate. Model weights are only part of memory consumption: context, KV cache, quantization, and runtime allocations also matter. Hardware-aware search and load planning help assess candidates, with bounded context reduction after an out-of-memory load failure.
API compatibility needs behavioral evidence. A familiar endpoint does not prove that a model and chat template will handle an agent's tool calls correctly. The conformance console provides diagnostics for evaluating that combination.
The working result
GGUF Switchboard brings model discovery, runtime selection, switching, and diagnostics into one local operating workflow. The public repository includes installation instructions, architecture documentation, integration guides, and compatibility notes.
Compatibility remains dependent on the client, model, template, and backend. Hardware estimates do not guarantee a successful load. The primary deployment target is Linux with NVIDIA CUDA, and the service belongs on a trusted network or behind an authenticated proxy because it has no built-in authentication.
What this demonstrates
This project turns an inference experiment into an operable developer tool. The important work sits at the boundaries: handling resource limits, coordinating runtime transitions, exposing failures, and checking the behavior that applications actually need.
It is a practical example of using proven components while taking ownership of the integration and operational experience around them.