On my RTX 3060, VRAM is the constraint. On a 2×4090 server, once the model fits comfortably and requests overlap, throughput becomes the constraint.
Those machines shouldn’t run LLMs the same way.
After running local coding models on a 12GB RTX 3060 and larger multi-GPU machines, I’ve stopped asking which format is “better.” The useful question is:
What am I optimizing for on this machine?
On the 3060, I want a useful coding model, enough context for real work, and room to finish a session without running out of memory. On the larger machine, I can start asking how many requests the server can handle while keeping response times acceptable.
That leads me toward different inference stacks.
In my previous article, I explained why constrained VRAM pushed me to build GGUF Switchboard: one OpenAI-compatible endpoint, with model routing and lifecycle management across llama.cpp and vLLM.
Supporting both raises the next question: which one should I actually use?
Here is my decision framework. These are starting points for choosing a configuration, not measured claims that one engine always beats the other.
The choice I actually make
| Situation | My starting point | What could change the decision |
|---|---|---|
| 8–12GB consumer GPU | GGUF + llama.cpp | A supported vLLM configuration fits comfortably and performs better on my workload |
| Model barely fits | GGUF + llama.cpp | A smaller model or different quantization offers a better overall result |
| CPU and system RAM needed | GGUF + llama.cpp | Offloading makes the model too slow for interactive use |
| One developer, mostly sequential coding requests | Usually llama.cpp | Long prompts or overlapping agent requests make another stack faster |
| Large GPU with comfortable VRAM | Consider vLLM | Low traffic gives me little reason to change an already useful setup |
| Multiple concurrent users | Evaluate vLLM early | Actual demand remains small, or model support favors llama.cpp |
| Aggregate throughput is the priority | Start with vLLM | Real workload measurements favor another configuration |
| Flexible quantization and hardware placement | GGUF + llama.cpp | The required architecture or feature lacks support |
| Shared inference server | Usually evaluate vLLM first | Memory constraints or workload characteristics favor llama.cpp |
| Mixed hardware | Choose per machine; use Switchboard to keep the client interface consistent | Backend capabilities still need validation |
The two questions behind this table are straightforward:
Can I serve the model with the context I need?
Can I serve the number of requests I need?
The first usually dominates my 3060 decisions. The second becomes more relevant on a shared GPU server.
First, separate the format from the engine
GGUF and SafeTensors store model data. llama.cpp and vLLM execute inference.
SafeTensors does not mean “full precision,” and GGUF does not mean “CPU inference.” Quantization, runtime support, kernels, and configuration all matter. SafeTensors itself is a tensor serialization format, rather than a serving strategy. Hugging Face documentation
In this article, I am comparing the combinations I use:
- GGUF with llama.cpp
- SafeTensors checkpoints with vLLM
These pairings are practical choices, not exclusive technical boundaries. vLLM also documents a GGUF path, but currently labels it experimental and under-optimized. vLLM GGUF documentation
That distinction matters when someone says “GGUF uses less memory” or “SafeTensors is faster.” A comparison between a heavily quantized model and a higher-precision checkpoint changes more than the file format.
On the RTX 3060, I optimize for usable headroom
My smaller machine has an RTX 3060 with 12GB VRAM and 64GB system RAM.
The challenge is fitting an entire coding session into that GPU budget.
The weights are only part of it. The runtime also needs space for the KV cache and working buffers. A coding request can bring tool definitions, source files, conversation history, and a substantial output budget.
A model that loads successfully can still be a poor daily configuration.
This is why GGUF with llama.cpp is usually my starting point. llama.cpp supports several quantization levels and CPU/GPU hybrid inference, giving me options when the model exceeds available VRAM. llama.cpp documentation
I can choose a smaller quantization, keep more of the model on the GPU, or place some work in system RAM. Each choice has a cost, but the flexibility is useful on hardware I already own.
Offloading deserves particular care. More system RAM can make a model runnable without making it responsive enough for coding.
My preference remains the same as in the first article: a smaller model that handles a realistic session comfortably is more useful than a larger model that spends the session at the edge of failure.
On the 2×4090 machine, I ask what the extra capacity is for
More VRAM expands the choices. It does not automatically make vLLM the right answer.
If I use the larger machine for one mostly sequential coding session, llama.cpp can still be a sensible choice. Owning two GPUs does not create concurrent demand.
The decision changes when several requests overlap: multiple developers, parallel agents, or background jobs sharing the same model.
Then I care about aggregate throughput alongside individual response times. vLLM’s scheduling and batching controls make it an attractive candidate for that workload. Its documentation explicitly describes trade-offs between batch size, latency, cache capacity, and throughput. vLLM optimization guide
There is also a deployment choice inside the multi-GPU setup.
Two 24GB GPUs do not behave like one transparent 48GB allocation. The serving configuration must distribute the model and its work. If the model fits on one GPU, separate replicas may be worth testing against spreading one model across both GPUs. Communication overhead and the workload determine which arrangement helps. vLLM parallelism and scaling
My first question on that machine is therefore: am I using the second GPU to fit a larger model, or to serve more work?
Those are different objectives.
Higher VRAM usage needs interpretation
A comparison can show vLLM occupying substantially more GPU memory than llama.cpp. That observation alone does not explain the difference.
I want to know the weight precision, context limits, concurrency settings, and how much memory the engine reserves for cache.
vLLM preallocates GPU cache according to its memory configuration. A larger reservation can support more concurrent work; it does not necessarily represent the minimum memory needed for one request. vLLM cache and preemption guidance
There is real overhead to consider as well. vLLM documents memory controls for context length, sequence count, and CUDA graphs. Reducing those requirements can change the configuration’s performance characteristics. vLLM memory guidance
On a dedicated server, allocating memory to useful serving capacity can be exactly what I want. On a 12GB development GPU, the same allocation policy can be inconvenient.
This is why I treat community comparisons as leads to investigate. Without the model revision, quantization, context, concurrency, and runtime settings, I cannot tell whether the result applies to my machine.
A coding agent needs more than tokens per second
For interactive coding, “faster” has several meanings.
I care about how long the model takes to start responding, how smoothly it generates, how long the complete task takes, and whether its tool calls work.
A high output rate does not compensate for malformed tool arguments or repeated failed edits.
Concurrency also needs a closer look. One developer can launch several agents. One agent can spend much of its time in a sequential loop: request inference, execute a tool, then request inference again.
I choose based on overlapping inference requests, rather than the number of people using the machine.
Nor is concurrency exclusive to vLLM. llama.cpp’s server supports parallel decoding and continuous batching. The question is how the implementations perform with my model and request pattern. llama.cpp server documentation
For a mostly sequential workflow on constrained hardware, I usually begin with llama.cpp. For sustained overlapping requests, I evaluate vLLM early.
What would make me switch?
I would compare a realistic coding workload before changing the runtime.
| What I measure | Why it matters |
|---|---|
| Memory use at realistic context | Establishes whether the configuration remains usable beyond a short prompt |
| Time to first token | Captures the wait before a response begins |
| Generation speed per request | Shows how interactive the response feels |
| Aggregate throughput under concurrency | Shows how much shared work the server completes |
| Slowest-request behavior | Exposes delays that an average can hide |
| Successful tool calls and completed tasks | Checks whether the speed produces useful work |
| Cold startup and model-switch time | Matters when models do not stay resident |
I would keep the base model, prompts, output limits, and test hardware consistent, and record any differences in quantization or templates. If I compare different precisions, I would describe the result as a deployment comparison rather than an isolated engine comparison.
I also want both single-request and concurrent tests.
A configuration that wins with several simultaneous requests may offer little benefit when I am waiting for one coding response. A configuration that feels excellent alone may struggle once other requests arrive.
The result that would change my mind is better task performance within my memory and latency budget.
Where GGUF Switchboard fits
This is why I wanted Switchboard to support both stacks.
On the constrained machine, I can choose GGUF with llama.cpp. On a larger server, I can choose a supported SafeTensors checkpoint with vLLM. The application still requests a model through the same style of OpenAI-compatible interface.
Switchboard handles routing and model lifecycle. The inference engine determines how it executes the request.
That separation lets me change deployment choices without teaching every coding client how to launch each backend. It does not make the backends identical: context limits, tool behavior, supported features, and performance still need testing.
For mixed hardware, the useful abstraction is a consistent client interface while each machine runs an appropriate configuration.
GGUF Switchboard exists to make that operating choice easier. Explore GGUF Switchboard for demonstrations and an overview of the supported workflows.
My default is a starting point
On the RTX 3060, I usually start with GGUF and llama.cpp because I need control over model fit, quantization, and memory placement.
On the 2×4090 server, I consider vLLM when the model has sufficient headroom and overlapping requests make throughput worth optimizing.
I would change either choice when the workload measurements justify it.
The question I now ask before downloading a model is:
What limits useful work on this machine: memory capacity, response latency, or serving capacity?
Answer that first. The runtime choice becomes much clearer.