What Qwen actually writes about my security cameras

Frigate logged a car. Qwen logged a dark blue Ford pickup with a chrome front bumper and cargo in the bed.

Frigate logged a cat. The night clip showed a raccoon, and Qwen said so.

A detector label tells me which events to keep. A description gives me the words I will remember six months later. That is the difference I wanted from a local vision model. I also wanted the camera frames to stay on my network.

This post is the camera half of the project. The companion article, Four mismatched GPUs, two replicas, one Qwen service, covers GPUStack, the 65K multimodal route, the 98K text route, and the vLLM tuning behind it.

The detector still does the detecting

Frigate records the cameras locally and performs object detection and tracking. Qwen is not in the real-time detection loop. At the end of a tracked object’s life, Frigate sends selected frames to an OpenAI-compatible multimodal route and stores the response as event metadata.

I run Frigate 0.18.0, build 344efb6. The 0.18 object-description path sends the uncompressed detect-stream frames collected during the track when use_snapshot is false. That sequence is what lets the model describe movement. A single snapshot is sharper, but it cannot establish direction by itself.

The model is the abliterated OrcaRouter NVFP4 build of Qwen 3.8 27B. I chose that build because its footprint fits each 2×16 GiB serving unit while retaining vision and tools. The surveillance workload did not require removed refusals. The endpoint stays private, and the saved clip remains the evidence.

One truck, then two harder cases

The west-facing Panorama clip is the compact success case. Frigate’s detector supplied car. Qwen supplied the searchable details.

ACTION: driving past
DIRECTION: passing through
ATTRIBUTES: dark blue pickup truck, Ford badge on the grille, chrome front bumper, items in the bed
SUMMARY: A dark blue Ford pickup truck drives past the camera, moving from the left side of the frame to the right. The truck has various items loaded in its bed.

The color, grille badge, bumper, cargo, and motion are visible in the clip. This is the output that changed the project for me. I can search for a Ford pickup with cargo instead of opening every event whose detector label is car.

The same Jeep from two cameras

Two Panorama cameras caught this Jeep a few seconds apart. I reran ten chronological frames from each saved clip through the same Qwen route. The model did not join the events. The timestamps and video make the relationship clear.

ACTION: driving forward
DIRECTION: approaching
ATTRIBUTES: black, four-door SUV, Jeep Wrangler, off-road tires, roof, occupants
SUMMARY: A black four-door Jeep Wrangler SUV drives forward down a residential driveway toward the camera. The vehicle approaches and then exits the frame to the left.

ACTION: driving away
DIRECTION: departing
ATTRIBUTES: dark green, open-top SUV, soft top roof, rear-mounted spare tire, visible occupants, red taillights
SUMMARY: A dark green open-top SUV with a soft top and rear-mounted spare tire drives away down a paved driveway. The vehicle moves from the left foreground toward the background, carrying several occupants.

The views agree on the body style, soft top, occupants, and trajectory. They disagree slightly on color, and both call Panorama Circle a driveway even though it is a street. Cross-camera correlation plus camera-specific scene context could reconcile those observations. Frigate currently stores two independent events with no shared identity graph.

Frigate said cat. Qwen said raccoon.

The object detector labeled this infrared event as a cat. My cat-specific prompt tells the model that the small-animal detector is unreliable at night and asks it to distinguish a cat from a raccoon using only body shape and a visible tail.

ACTION: walking across the driveway
DIRECTION: passing through
ATTRIBUTES: probably a raccoon, medium size, grey coat, ringed tail
SUMMARY: A probable raccoon with a ringed tail walks across the driveway and heads toward a corner.

The ringed tail and stocky body are visible. This is a better description than blindly repeating the detector label, and the word probably preserves the uncertainty appropriate to infrared footage.

A 24-hour production audit

I fixed a 24-hour window ending August 22 at 13:24 UTC. Frigate returned 2,180 events with snapshots across 33 cameras. Of those, 2,178 tracks had ended, 2,012 had saved clips, and 1,959 had stored descriptions.

Those are event rows, not inference calls. Pool 1 and Pool 2 alone produced 1,083 overlapping person tracks during an evening of activity. The denominator includes short tracks, duplicate views of the same activity, events without a retained clip, and rows that had not received a description by the query time.

I then selected 24 completed events with clips no longer than 45 seconds. The sample covered 13 cameras: 8 people, 8 vehicles, 4 dogs, 3 cat-detector events, and 1 package. This is a structured editorial audit, not a statistical estimate for every one of the 2,180 events.

The partial group was not random prose noise. The failures clustered around distant people, inferred delivery context, and motion direction. One package description got the carried box right but reversed whether it was moving toward or away from the camera. Two distant-person descriptions assigned clothing details that the saved clip could not defend.

The one clearly wrong clip

This event was labeled car. Qwen described a dark sedan with its headlights on driving through the driveway. I cannot find a vehicle in the saved video.

ACTION: driving past
DIRECTION: passing through
ATTRIBUTES: dark-colored sedan, headlights on
SUMMARY: A dark-colored sedan with its headlights on drives past the driveway area. The vehicle moves across the frame and exits the view.

This is why I store the description as metadata, not truth. I do not use it to identify people, infer threat, or make an enforcement decision. The video is still the record.

A Gemma-era event, rerun through Qwen

I went back to an August 9 event created while my previous Gemma4 pipeline was running. The exact image bundle sent to Gemma was not retained, so this is not a controlled model-only benchmark. It is a same-event comparison: the saved Gemma-era description versus the current Qwen pipeline on ten chronological frames from the retained clip.

Saved Gemma-era result:

ACTION: passing_through
DIRECTION: passing_through
ATTRIBUTES: type:sedan, color:silver, make:Toyota, model:Avalon, plate_text:[guessed characters], movement:moving
SUMMARY: A silver Toyota Avalon passes through the driveway.

Current Qwen result:

ACTION: driving away
DIRECTION: departing
ATTRIBUTES: silver, sedan, alloy wheels
SUMMARY: A silver sedan drives away down a curved residential street. The vehicle moves from the right foreground into the distance.

Gemma supplied a make, model, and plate string that the saved evidence did not justify, and it called the road a driveway. Qwen gave up the flashy guesses and got the trajectory and setting right. The prompt also changed between the two pipelines, so I treat this as a comparison of the systems I actually operated rather than proof that every Qwen output beats every Gemma output.

The prompt

I ask for observable metadata, not intent. Frigate’s default prompt can ask what an object might be doing or do next. That is not the record I want to search. My vehicle prompt is:

These are security-camera frames of one vehicle from the {camera} camera,
in chronological order.

Recognized identity: {sub_label}. If that value is None, do not name a
company or owner. Otherwise use the confirmed match as the subject.

Reply with exactly these four lines, in this order, and nothing else.

ACTION: two or three plain words for the main thing the vehicle does

DIRECTION: the net movement across the whole sequence relative to the
camera, one of: approaching, departing, stationary, passing through,
circling, unknown

ATTRIBUTES: comma-separated plain-English phrases for what is actually
visible: colour, body style, make and model only when unmistakable,
roof equipment, readable livery, lights, doors, cargo, or occupants.
Leave out anything you cannot see.

SUMMARY: one or two sentences written like a photo caption. Give the
colour, body style and any readable company name first, then what the
vehicle does and which way it goes.

Never state or guess licence plate characters. Never quote or guess
wording unless it is clearly readable.

The person, dog, cat, and package prompts keep the same four fields but change the allowed attributes. The cat prompt explicitly accounts for raccoons in infrared scenes. The person prompt can use a confirmed Frigate sub-label, but it must not invent a name when recognition returns None.

The Frigate configuration

This is the relevant shape of the live 0.18 configuration. The route alias and credential placeholder are deliberately generic. The actual route is the 65,536-token multimodal configuration described in the infrastructure post.

genai:
  camera_descriptions:
    provider: openai
    base_url: https://your-openai-compatible-endpoint.example/v1
    api_key: "{FRIGATE_GENAI_API_KEY}"
    model: your-multimodal-route
    roles:
      - descriptions
    provider_options:
      context_size: 4518
      max_retries: 0
    runtime_options:
      temperature: 0.1

objects:
  genai:
    enabled: true
    use_snapshot: false
    objects:
      - person
      - car
      - motorcycle
      - dog
      - cat
      - package
    send_triggers:
      tracked_object_end: true
    prompt: |-
      [the four-field prompt above]
    object_prompts:
      car: |-
        [the vehicle-specific prompt above]

Thinking is disabled for these requests. The output is short and observational, and spending extra tokens on a reasoning trace adds latency without improving the metadata I need.

The serving bug the cameras found

Text traffic remained healthy while ordinary multi-image descriptions failed. The model route advertised 65,536 tokens, but the checkpoint processor still enforced a max_length of 2,048 that matched its 4-bit calibration setting. Expanded image placeholders crossed that hidden limit.

Cached fixtures made the failure intermittent. Five fresh images passed and entered the cache. Those five plus one new image also passed because the processor saw only one miss. Six entirely fresh images failed. After I cleared the persisted truncation value and restarted the processor, sixteen fresh images completed through the same route Frigate uses.

The detailed failure analysis and the exact vLLM configuration are in Four mismatched GPUs, two replicas, one Qwen service.

Where I landed

Qwen has crossed the threshold I wanted. Most reviewed descriptions add a fact I can search, the schema is stable, and the night raccoon shows that the model can correct a coarse detector label instead of merely decorating it.

It still gets direction wrong, promotes context into certainty, and occasionally describes an object that is not in the retained video. The fix is not to pretend those cases do not exist. Store the clip, keep the language observational, and make the description easy to challenge.

The next Frigate feature I want is cross-camera correlation. The two Jeep clips already contain enough overlapping detail to suggest a match. Today, I supply that relationship by looking at the timestamps. I would rather Frigate build the graph.