Four mismatched GPUs, two replicas, one Qwen service

Four 16 GiB cards did not become one 64 GiB GPU. They became two complete 32 GiB replicas behind one service.

That distinction is the whole build. One machine has 2× RTX 5080. The other has 2× RTX 4070 Ti Super. Each pair runs Qwen 3.8 27B with tensor parallelism of 2, and GPUStack hides the difference from clients. I can give one pair the multimodal configuration and the other the longer text configuration, or put both pairs behind the same configuration and weight requests by measured throughput.

The result serves my coding and research agents, and it supplies the local vision backend for Frigate. I split the camera evaluation into a companion post, What Qwen actually writes about my security cameras. This one is about fitting, routing, and tuning the model.

Version note, August 22, 2026: These results are pinned to GPUStack 2.2.3, vLLM 0.25.1, and the checkpoint revision I tested. The checkpoint documentation and current vLLM path have moved since these measurements. Treat the flags and the MTP result as a record of this stack, not defaults for a newer one.

Two ways to use the same four cards

The published vLLM recipe assumes a larger Blackwell card or a pair of 5090s. Two 16 GiB cards are enough for one replica in my setup. The second pair adds another workload profile, aggregate capacity, or redundancy. It does not add its VRAM to the first pair.

GPUStack model routes are what make the mismatch disappear at the API boundary. The 58/42 weights are manual request weights, not dynamic scheduling based on predicted tokens or GPU time. They make sense when both targets host the same workload profile. A long agent request and a short camera description are not equivalent units of work, which is one reason I also keep the split mode available.

The official model supports a much larger native context. The 65,536 and 98,304 token values are deployment limits imposed by these cards and configurations. The 98K text route has been especially valuable for agents. Repository context, instructions, tool schemas, intermediate results, and action history can remain in one request without immediately forcing a summary or retrieval hop.

MTP made decode 4.6 times slower

This was the result I least expected. The checkpoint includes a multi-token prediction head, and I expected speculative decoding to improve single-stream speed. On vLLM 0.25.1 with FlashInfer, enabling it reduced decode from 64.9 to 14.1 tokens per second.

The draft acceptance rate was 76.9%, which looked healthy. The serving path was not. Speculative decoding changed CUDA graphs from FULL to PIECEWISE, and the 64K multimodal window no longer fit with the drafter loaded. The accepted drafts did not repay the graph-mode loss.

Six A/B waves required an 8 to 12 minute cold start for every arm. A plausible flag was not enough. It had to boot, survive an ordinary request, and improve the metric it was supposed to improve.

The memory window was 20 MiB wide

Pipeline parallelism of 2 did not boot. Rank 1 received no KV capacity because the 248K-vocabulary embedding and lm_head landed on one stage. Tensor parallelism of 2 was the topology that reliably split the model across each pair.

The published --gpu-memory-utilization 0.90 missed the KV check by 20 MiB. At 0.95, the server booted with about 10 MiB free and then died when an ordinary request needed a 34 MiB activation. 0.92 stayed up.

Restricting max-num-seqs to 1 hid the OOM but cut aggregate throughput by roughly four times. Eight concurrent sequences was the stable operating point. This is why a successful boot is not a stability result.

What the two pairs delivered

Single-stream decode reached 64.9 tokens per second on the 5080 pair and 48.1 on the 4070 Ti Super pair. At saturation, the pairs reached 275 and 220 tokens per second, or 495 combined at 645 W of GPU board power.

The route weights come from a different test. A production-shaped concurrency-eight run measured 244 on the 5080 pair and 177 on the 4070 pair. That 421-token total is not in conflict with the 495-token saturation result. One run selected the traffic split; the other searched for each replica’s ceiling.

Prefill is where the newer path separates most clearly. At a 16K prompt, the 5080 pair reached 2,767 tokens per second and the Ada pair reached 1,823. Prefix caching changed the interactive experience more than another peak decode number: a repeated roughly 16K agent context fell from 6.1 seconds to 0.54 on the 5080s.

Idle power complicates the pooling story. The 5080 pair idles at 21.6 W. The 4070 pair idles at 102.8 W, nearly five times as much, while producing less work at saturation. Together they consume 124.4 W before host power, or about 1.09 MWh per year if left powered continuously.

If Frigate descriptions were the only workload, I would not keep the older pair awake merely to increase the request count. It earns its place when I need the second profile, burst capacity, or a replica that survives maintenance on the other node. Power-aware scale-to-zero is the next problem worth solving.

A 2,048-token limit inside a 65K route

The multimodal failure looked like load because text traffic was healthy, both replicas were healthy, and GPUStack returned 503. The actual vLLM response was a permanent 400.

• Symptom: a normal ten-frame camera request failed while long text requests continued to work.
• Cause: tokenizer.json retained max_length: 2048, matching the checkpoint’s 4-bit calibration length. The Hugging Face processor honored it when image placeholders expanded.
• Why fixtures lied: cached images bypassed enough processor work to stay below the boundary.
• Fix: set the persisted tokenizer truncation to null in the local cache copy, repeat the patch in the seed job, and restart the runner so the processor is rebuilt.
• Validation: sixteen fresh images completed through the application path, and text token counts remained unchanged.

Fixtures lie. Test images the processor has never seen, through the route the application actually uses. The companion Frigate post shows what that repaired path produced over the next 24 hours.

The configuration I kept

These are vLLM arguments, followed by environment and orchestration settings. They are an inventory of the tested deployment, not one copy-and-paste command.

# vLLM arguments
--tensor-parallel-size 2
--max-model-len 65536          # multimodal profile
--gpu-memory-utilization 0.92
--kv-cache-dtype auto
--max-num-seqs 8
--max-num-batched-tokens 8192
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--enable-prefix-caching
--mamba-cache-mode align
--trust-remote-code
--override-generation-config {"temperature":1.0,"top_p":0.95,"top_k":20}

# environment
PYTORCH_ALLOC_CONF=expandable_segments:True
CUDA_DEVICE_ORDER=PCI_BUS_ID

# GPUStack runner behavior
restart_on_error: true

The text-only profile removes the vision tower and raises --max-model-len to 98,304. A 131,072-token attempt did not boot. For general agent traffic I send reasoning_effort: medium and allocate enough completion tokens for the reasoning budget. Frigate disables thinking.

The deployed checkpoint is the abliterated OrcaRouter NVFP4 build. I chose the available 4-bit build that fit a 32 GiB serving unit while retaining vision, tool use, and the MTP head. Removed refusals were not required for camera descriptions. The endpoint is private, and applications still own their own policy and authorization boundaries.

Open questions

I have not rerun the matrix on the newer vLLM path documented by the checkpoint author. That is the first test I would repeat, especially MTP and KV cache dtype.

The second question is operational: can the Ada pair sleep until a route needs it without turning cold starts into the new bottleneck? Its redundancy is valuable. Its idle draw is hard to ignore.

The third is workload scheduling. Static request weights work for equivalent replicas. They know nothing about an image batch versus a 70K agent prompt. The next routing improvement should account for queued work, not just request count.

Appendix: the six A/B waves

Four GPUs did not become one GPU. They became one service, with two independently deployable replicas and enough flexibility to choose capacity, context, or redundancy for the work in front of them.