eitanlebras/ghosttrap-qwen2vl-2b-invariance

GhostTrap — a prompt-invariant camera-trap VLM

A QLoRA adapter for Qwen2-VL-2B-Instruct trained for one specific property: giving the same answer when you ask the same question in different words.

On the 5 held-out prompt variants — wordings never seen in training — empty-frame false-positive spread falls from 98.37 points (stock) to 12.2 points (SFT+DPO), a 86.17-point reduction.

⚠️ This model does not output calibrated confidence. It was meant to, and it does not: the training targets omitted the confidence field, so the model learned not to emit one (0% coverage after DPO, 19% after SFT, versus 56% for stock). ECE is therefore undefined for the DPO adapter. The invariance result stands on its own; the calibration half of the original objective does not. Details in Results and Limitations.

Two adapters are published. adapter_dpo is the invariance model (held-out FP spread 12.2 pts, worst-case binary accuracy 86.0%). adapter_sft preserves more capability (species accuracy 51.7% vs 36.5%) at a wider spread (46.8 pts). Pick by which you need; this is a real trade-off, not a ranking.

Why this, and not "fewer hallucinations"

An earlier stage of this project tried to reduce the empty-frame false-positive rate — the tendency of VLMs to invent animals in empty camera-trap frames, documented in Real-Wild-VLM (CVPR 2026 Workshops, Deng et al.). That turned out not to be a property worth training, because a prompt edit does it for free.

Holding the model, the images, and the decoding fixed and changing only the wording of the question, stock Qwen2-VL-2B's empty-frame false-positive rate moves across essentially the entire range:

Model	variant set	FP spread ↓	FP range	FN spread	answer-rate spread
Stock Qwen2-VL-2B	all 15	99.59	0.0–99.59%	53.25	100.0
Stock Qwen2-VL-2B	held-out 5	98.37	0.0–98.37%	53.25	100.0
+ SFT	held-out 5	46.75	10.57–57.32%	9.09	80.08
+ SFT then DPO	held-out 5	12.2	5.28–17.48%	24.02	0.0

Both endpoints are ordinary phrasings a real user might type. Neither is adversarial. So "our model has a low false-positive rate" is not a claim about a model — it is a claim about a prompt.

Prompt sensitivity is different: you cannot fix it with a prompt. That is what makes it the property in this project where post-training is necessary rather than merely sufficient.

Results

All numbers: same 400 held-out-location eval images, greedy decoding, max_new_tokens=32, 256 visual tokens per image, identical parser.

Model	variant set	FP spread ↓	FP range	FN spread	answer-rate spread
Stock Qwen2-VL-2B	all 15	99.59	0.0–99.59%	53.25	100.0
Stock Qwen2-VL-2B	held-out 5	98.37	0.0–98.37%	53.25	100.0
+ SFT	held-out 5	46.75	10.57–57.32%	9.09	80.08
+ SFT then DPO	held-out 5	12.2	5.28–17.48%	24.02	0.0

Per-variant false-positive rate

Prompt variant	split	Stock Qwen2-VL-2B FP	+ SFT FP	+ SFT then DPO FP
v1_two_line_brackets	train	0.0%	—	—
v2_plain_question	HELD-OUT	0.8%	10.6%	5.3%
v3_labelled_no_brackets	train	21.9%	—	—
v4_question_then_conf	train	2.9%	—	—
v5_terse	train	2.0%	—	—
v6_polite_request	train	96.8%	—	—
v7_json	HELD-OUT	0.0%	15.8%	6.5%
v8_no_empty_hint	HELD-OUT	98.4%	57.3%	17.5%
v9_ecologist_role	HELD-OUT	14.2%	12.6%	6.9%
v10_checklist	train	4.9%	—	—
v11_binary_first	HELD-OUT	84.5%	17.5%	6.1%
v12_single_word	train	30.5%	—	—
v13_verbose_instruction	train	79.7%	—	—
v14_uncertainty_ok	train	0.4%	—	—
v15_species_word	train	99.6%	—	—

Semantic sensitivity — the anti-cheat check

Paraphrase training has an obvious degenerate solution: ignore the prompt and emit one answer forever. That scores perfectly on invariance while being a strictly worse model. So the training mix includes semantically different questions with different correct answers, and the eval measures whether the model still answers the question it was asked.

Model	species	presence	count	lighting	mean (non-species)	all beat majority?
Stock Qwen2-VL-2B	70.3%	53.0%	59.1%	90.0%	67.3%	NO — see note
+ SFT	78.7%	76.0%	72.8%	90.0%	79.6%	yes
+ SFT then DPO	71.7%	71.7%	72.0%	90.3%	78.0%	NO — see note

Majority-class baselines: species 61% / presence 61% / count 72% / lighting 58%.

Each task is reported against its majority-class baseline, because a collapsed model tends to match the majority label exactly — that is the tell.

Intended use

Screening camera-trap frames for whether an animal is present at all, robustly across however an operator phrases the request. That robustness is the reason to use this over the base model: worst-case binary accuracy across held-out phrasings is 86.0% versus 2.5% for stock.

It does not provide a confidence value to threshold on — see the warning above. If you need a tunable operating point today, threshold on your own downstream signal, not on the model's stated confidence.

Out of scope:

• General wildlife identification outside the Caltech Camera Traps domain (American Southwest, 22 categories).
• Species-level field research decisions. Species accuracy is well below what a purpose-built classifier achieves; use this to triage empty frames, not to identify animals.
• Any setting where a missed animal is costly without a human in the loop.

How to use

from peft import PeftModel
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration

BASE = "Qwen/Qwen2-VL-2B-Instruct"
model = Qwen2VLForConditionalGeneration.from_pretrained(BASE, device_map="cuda")
model = PeftModel.from_pretrained(
    model, "eitanlebras/ghosttrap-qwen2vl-2b-invariance",
    subfolder="adapter_dpo")   # or "adapter_sft"
processor = AutoProcessor.from_pretrained(BASE)

# The visual-token cap MUST match training, or results will not reproduce.
ip = processor.image_processor
ip.min_pixels, ip.max_pixels = 64 * 28 * 28, 256 * 28 * 28
ip.size = {**ip.size, "min_pixels": ip.min_pixels, "max_pixels": ip.max_pixels}

Ask in whatever wording you like — that is the point.

Training

Base	Qwen/Qwen2-VL-2B-Instruct (Apache-2.0)
Method	QLoRA, nf4 on the LLM, vision tower kept in bf16
LoRA	r=16, alpha=32, dropout 0.05, attention + MLP, language model only
Stages	SFT, then DPO on the SFT checkpoint
Data	2900 SFT examples ({'species': 2000, 'count': 300, 'presence': 300, 'lighting': 300}), 1000 DPO pairs from 2171 mined stock failures
Hardware	1× A40 48GB

The vision tower is deliberately excluded from quantization: nf4 on visual silently destroys it (see below).

Data provenance and licence

Derived from Caltech Camera Traps via LILA BC, CDLA-Permissive-1.0, which permits redistribution of derived data with no share-alike obligation. Split by camera location using the official CaltechCameraTrapsSplits_v0.json — never randomly, because consecutive frames from one camera are near-duplicates and a random split leaks eval into train.

Dataset: eitanlebras/ghosttrap-camera-trap-invariance

Beery, S., van Horn, G., Perona, P. "Recognition in Terra Incognita." ECCV 2018.

Adapter weights are a derivative of an Apache-2.0 base model and are released under Apache-2.0.

Limitations

Written worst-first, because the first one is genuinely the biggest.

• Single seed, single run per configuration. No variance estimate over training runs. Confidence intervals given anywhere in this card are binomial over images only and say nothing about run-to-run stability. This is the weakness most likely to change the conclusions.
• 2B parameters, one model family. Nothing here shows the effect generalises to other VLMs.
• 15 prompt variants is not prompt space. The held-out five were assigned by hash before any evaluation, which makes them an honest test, but five points do not characterise the distribution of things users write.
• Daytime-and-night RGB as a proxy for the infrared condition Real-Wild-VLM studied. Night frames here are IR/flash monochrome, which is close but not the same corpus or the same capture hardware.
• No evaluation against the actual Real-Wild-VLM benchmark, which is not publicly downloadable. Nothing in this card is a result on their data.
• Species accuracy is low in absolute terms and is scored after mapping open-vocabulary output onto 22 CCT categories.
• 256 visual tokens per image. Chosen because measured vision quality did not improve at 512 or 1024 on the reference frames, but it does cap what the model can resolve on small or distant animals.
• Semantic sensitivity is measured on the task phrasings seen in training, so it shows the capability survived — not that it transfers to unseen task wordings.
• No confidence output at all (0% coverage after DPO). The training targets omitted the field; the model learned the target distribution it was shown. This is a data-design error, not an inherent limit, and is the single cheapest thing to fix.
• The count training signal was degenerate. A missing bounding-box file meant every count target was "0" for 10% of SFT examples. The count evaluation is valid (majority baseline 72%); the training data for that task was not.
• DPO trades species accuracy for invariance (51.7% → 36.5%). If species identification matters more than robustness, use adapter_sft.

Claim discipline

The defensible claim: addressing the empty-frame bias documented in Real-Wild-VLM, evaluated on Caltech Camera Traps. Not SOTA on their benchmark; not evaluated on their benchmark at all.

Validation evidence — four silent failures this pipeline catches

These are not footnotes. Each passed a structural check and would have produced a confident, wrong result; they are the reason the numbers above should be trusted at all.

1. nf4 quantization can blind the vision tower. Every structural assertion passed — 4-bit modules present, image tokens in input_ids, pixel_values populated — while the model described a daylight bobcat as "a person standing in a field with a blue sky". Now caught by a behavioural guard whose threshold is calibrated against an unquantized bf16 run.
2. top_k=1 in the shipped generation_config silently overrides do_sample=True. Preference mining returned 0 hallucinations against a 9.2% base rate. Now caught by asserting that 20 samples are not identical.
3. Parser bugs invent false positives. "Animal: blank" was once scored as a hallucination, which put correct abstentions into the REJECTED slot of preference pairs — training the model to hallucinate. Now gated by a 50-case hand-labelled regression test that must pass before any pair is written.

A fourth was found in this run: passing min_pixels/max_pixels to AutoProcessor.from_pretrained sets the attributes but does not change the output, because the processor reads its size dict. Now asserted on the realised patch count.

The rule: assert on behaviour, not configuration state. A setting the API accepted is not a setting that took effect.

Code

https://github.com/eitanlebras/GhostTrap — eval harness, guards, frontier sweep, training, and the full decision log.