Running an LLM locally has become surprisingly easy.
Running several specialized models as a reliable local AI development stack is a different problem.
I wanted to use local models with coding agents such as OpenCode and other OpenAI-compatible tools while keeping everything on hardware I already owned. My test machine is not an AI server:
- NVIDIA RTX 3060 with 12GB VRAM
- AMD Ryzen 7 5800
- 64GB RAM
- Ubuntu Linux
The obvious question was: how much useful local AI can I actually run on a 12GB consumer GPU?
The answer turned out to be quite a lot. But the interesting problem was not getting a model to run. It was managing many models when the GPU could realistically hold only one of them at a time.
My goal was not local chat
There are already excellent tools for chatting with local models. I wanted my models to operate as part of my software development environment.
That meant supporting several kinds of work:
- coding
- reasoning
- tool calling
- structured output
- embeddings
- AI agents
- experiments with new models
I also wanted coding agents, SDKs, and internal applications to consume those models through a standard API.
For this workload, a good response to a prompt is only the starting point. I also care about whether a model can understand an existing codebase, call tools with valid arguments, accept useful repository context, respond quickly enough for interactive coding, and behave consistently behind an API.
Then there is resource usage. Can I move between specialized models without manually cleaning up GPU memory every time? That question eventually shaped the whole architecture.
12GB VRAM is more capable than it looks
A 12GB GPU cannot keep every 20B, 30B, or 70B model resident at full precision. Parameter count alone, however, is a poor way to decide what will work.
The practical result also depends on:
- quantization
- model architecture
- context size and KV cache
- runtime overhead
- GPU offloading
- available system RAM and memory bandwidth
- the inference engine
- the workload itself
A well-quantized 7B to 12B model can be very comfortable on this hardware. Larger models can sometimes run by moving part of the workload into system RAM.
But there is an important distinction:
"It runs" and "I would use it every day" are different benchmarks.
For interactive coding, I would rather use a strong smaller model at a practical speed than force the largest possible model onto the machine.
My practical range is roughly 7B to 12B
On this RTX 3060, that is the range I currently find most useful. Modern models in this class, paired with sensible 4-bit quantization, can leave room for context and runtime overhead while remaining responsive.
Recent Qwen-family models have been particularly interesting in my coding and agent experiments. The broader lesson matters more than any specific model:
Do not optimize for the largest model you can load. Optimize for the strongest model you can run comfortably.
That difference becomes obvious when the model moves from a benchmark into a daily development workflow.
GGUF and SafeTensors solve different operating problems
For constrained consumer hardware, GGUF with llama.cpp is often my default. It gives me multiple quantization choices, predictable weight sizes, GPU layer offloading, CPU and RAM fallback, and mature consumer-GPU support.
There are also situations where I want the original SafeTensors model through vLLM, especially on larger GPU machines or when throughput matters.
I stopped treating the choice as a format contest. Neither option is universally better. They optimize for different operating conditions.
My desired architecture became:
+-- llama.cpp --> GGUF
|
AI client --> API --> scheduler
|
+-- vLLM ------> SafeTensors
The client should not need to care which runtime happens to be underneath it.
Then I hit the real problem
Once I had several useful models installed, the constraint became clear. I might want a coding model, a general reasoning model, a model verified for tool calling, a dedicated embedding model, and whichever model I was currently testing.
The RTX 3060 could not keep all of them resident in VRAM. It did not need to. Most of the time, only one was required.
What I wanted was request-driven model residency:
Request asks for Model B
|
v
Is Model B already running?
/ \
yes no
| |
serve stop new Model A work
|
v
drain active requests
|
v
unload Model A
|
v
verify memory release
|
v
load Model B
|
v
health check
|
v
serve
At that point, my problem was no longer primarily inference. It was scheduling.
Why not just use Ollama?
Ollama is excellent. If someone asks for an easy way to download and run a local model, it is one of the first tools I would suggest:
ollama run <model>
It provides an API, OpenAI compatibility, and model residency controls. For many local AI workloads, that is everything required.
My question had changed from "How do I run a model?" to "How do I operate multiple specialized models as shared infrastructure on constrained hardware?"
Ollama already keeps models resident and unloads them through configurable behavior. Claiming otherwise would be inaccurate. What I wanted was explicit control over the entire transition lifecycle.
If two coding agents are generating responses with Model A and another application requests Model B, I want the system to stop accepting new work for Model A, allow its active requests to finish, unload it, confirm that resources have been released, load Model B, perform a health check, and only then route traffic. A failed transition also needs a defined recovery path.
That is closer to a single-slot scheduler than a model launcher.
Why not LM Studio?
LM Studio is another tool I use and like. Its interactive model discovery and experimentation experience is excellent.
It should not be dismissed as merely a GUI. LM Studio supports CLI and headless operation, local APIs, model loading and unloading, GPU offload configuration, memory estimation, and idle TTL and automatic eviction.
The distinction I found useful was this:
LM Studio helps me work with local models. I wanted other software to treat my collection of local models as infrastructure.
That requirement pushed model selection, runtime choice, transition behavior, and health into a service boundary rather than a desktop workflow.
I did not want one runtime to define the infrastructure
Sometimes I want GGUF through llama.cpp. Other times I want SafeTensors through vLLM. These inference stacks have different strengths, but a coding agent, Node application, or Python service should not need to understand my inference topology.
I wanted clients to use one OpenAI-compatible endpoint:
OpenCode ---------+
Coding agents ----+
Python SDK -------+----> one OpenAI-compatible API
Node SDK ---------+
Internal apps ----+
|
v
model scheduler
/ \
v v
llama.cpp vLLM
| |
v v
GGUF SafeTensors
The requested model should determine the backend. The client should simply request the model.
Hardware should be part of model discovery
Hugging Face contains an enormous number of models, repositories, and quantizations. Searching for "Qwen" is easy, but it does not answer the question I actually have:
I have an RTX 3060 with 12GB VRAM and 64GB RAM. Which Qwen models and quantizations make sense for this machine?
The answer changes completely on another machine. I also run local AI on a system with two RTX 4090 GPUs, 48GB combined VRAM, and 128GB RAM. A useful model search should account for that difference before I download tens of gigabytes of weights.
This led me to add hardware-aware discovery to GGUF Switchboard:
ggs models search "qwen"
The goal is not merely to find repositories. It is to make the machine's available resources part of the deployment decision.
"It fits" is not enough
Suppose a model's weights occupy 9.5GB and the GPU has 12GB of VRAM. Calling that a fit ignores the KV cache, context, GPU buffers, runtime overhead, and every other process using the GPU.
Context matters especially for coding agents. A single request might contain a system prompt, tool definitions, repository context, relevant source files, conversation history, and the current task. A configuration that survives a simple chat test can run out of memory during a real coding session.
I now prefer leaving VRAM headroom. A slightly smaller model that remains stable throughout a long session is more useful than a larger model that sits permanently on the edge of an out-of-memory failure.
A useful discovery tool should therefore answer more than "Will it load?" It should help estimate memory use, choose a quantization, reason about practical context, identify CPU offloading, and explain the performance trade-off.
Coding benchmarks do not prove that an agent will work
A model can write an excellent TypeScript function and still fail inside an agentic workflow.
An agent also needs to produce a tool request with the right function and correctly structured arguments:
{
"name": "read_file",
"arguments": {
"path": "src/server.ts"
}
}
Models differ in tool-use behavior. Chat templates, runtime support, structured output, and quantization can all affect the result.
For an agentic coding workflow, I need to know whether a model understands the available tools, chooses the correct one, produces valid arguments, follows the expected schema, and exposes the result correctly through the inference runtime.
That is why I added tool-conformance testing to Switchboard. The evaluation question is no longer just "Can this model generate good code?" It is also "Can I trust this model to participate in an agentic software workflow?"
I wanted the clients to become boring
I did not want every application to maintain a map of model-specific ports:
Coding model --> localhost:8081
Reasoning model --> localhost:8082
Experimental model --> localhost:8083
vLLM model --> localhost:8000
Embedding model --> localhost:8084
That topology leaks infrastructure details into every client. Changing a model then means editing several applications.
I wanted all clients to use localhost:9090, with model routing and runtime choice behind that endpoint. The client configuration becomes boring. That is a feature.
The switchboard that emerged
The architecture I built is intentionally direct:
+------------------------------------------------+
| AI applications |
| |
| Coding agents / Python / Node / internal apps |
+-----------------------+------------------------+
|
OpenAI-compatible API
|
v
+------------------------------------------------+
| GGUF Switchboard |
| |
| Model registry |
| Hardware-aware discovery |
| Request routing |
| Lifecycle management |
| Drain, switch, and recovery |
| Tool-conformance testing |
+----------------------+-------------------------+
|
+--------+--------+
| |
v v
llama.cpp vLLM
| |
v v
GGUF SafeTensors
The inference engines remain inference engines. The switchboard handles a different question: what should run, where should it run, and when should it run?
It is not an Ollama replacement, and I do not think of it as an LM Studio replacement either.
| Tool | What I primarily use it for |
|---|---|
| LM Studio | Interactive model discovery and experimentation |
| Ollama | Simple local model execution and developer experience |
| llama.cpp | Flexible GGUF inference |
| vLLM | High-throughput GPU inference |
| GGUF Switchboard | Hardware-aware discovery, routing, and lifecycle orchestration across models and runtimes |
There is overlap between these tools. The difference is what each optimizes for.
The simplest summary of why I built another layer is this:
Ollama and LM Studio make running local models easy. GGUF Switchboard explores how to make operating a collection of local models boring.
On a machine with 12GB of VRAM, boring infrastructure is exactly what I wanted.
What I would actually run on an RTX 3060 12GB
My current strategy is straightforward:
For daily coding: stay around strong 7B to 12B models with sensible quantization. Responsiveness matters more than maximizing parameter count.
For agent workflows: prioritize reliable tool use over benchmark scores. A slightly weaker coding model that calls tools correctly can be more useful inside an agent.
For context-heavy development: leave VRAM headroom. Do not choose a configuration that consumes nearly all available memory before receiving meaningful context.
For constrained hardware: I generally prefer GGUF with llama.cpp because of its flexibility.
For larger GPU systems: SafeTensors with vLLM becomes more attractive when models fit comfortably and throughput matters.
For larger models: experiment, but do not confuse successfully loading a model with wanting it to respond to a coding agent all day.
What surprised me most
A relatively inexpensive 12GB consumer GPU is enough to build a genuinely useful local AI development environment.
It will not compete with infrastructure running frontier-scale models. It does not need to. Modern smaller models have become remarkably capable for coding assistance, specialized tasks, local agents, experiments, privacy-sensitive workloads, and self-hosted applications.
That shifts the engineering question. A few years ago, I was asking whether I could run an LLM locally. Now I am asking how to operate several local models reliably as infrastructure.
That is the problem GGUF Switchboard is exploring.
Try the same experiment
GGUF Switchboard is open source on GitHub. The repository includes installation instructions, architecture documentation, model discovery, backend configuration, and integration guides.
I have also published demonstrations of model switching and GPU memory behavior on the GGUF Switchboard product page.
If you already have an NVIDIA GPU, a useful place to start is:
ggs models search "qwen"
Then ask what makes sense for your hardware, not merely what can technically load.
What comes next
I plan to document more local-inference experiments rather than only the switchboard itself.
One comparison I want to examine is GGUF with llama.cpp versus SafeTensors with vLLM on consumer GPUs. I also want to compare the RTX 3060 system with the dual RTX 4090 machine using measured model fit, context headroom, latency, and tool behavior instead of theoretical model-size calculations.
If you run local coding agents, I am interested in the GPU, model, quantization, and runtime you actually use every day. Not what technically loads. What you find useful.