Depth-Aware Open-Vocabulary Segmentation
of Adverse-Condition Urban Streetscapes

Zero-shot vision–language segmenters work well on clean scenes but fail in the cases that matter most for mobility and safety analysis: nighttime, rain, glare, occlusion. Symptoms are predictable — boundaries blur into shadow, small but safety-critical classes (signs, poles) vanish, and large regions collapse onto a single background label.

Our hypothesis: under adverse conditions texture alone is a weak cue, but a geometric prior — ground plane, vertical surfaces, near/far — survives illumination loss. The challenge is to inject that prior into a frozen open-vocabulary backbone without sacrificing zero-shot text alignment and without the prohibitive cost of full-parameter VLM training.

Why depth? Adverse-condition frames are dominated by low-contrast texture: shadows swallow object edges and rain blurs long-range structure. The depth-VFM produces features that carry coarse geometric structure — near/far, ground plane, vertical surfaces — that survives illumination loss. We compute a delta feature $F_\ell = V_\ell + \phi_\ell(D_\ell)$, where $\phi_\ell$ is a small learnable projection that aligns depth features into the VLM feature space. This is also the exact intervention point of the zero-tensor ablation reported below.

Why GeoText queries + CMPE? So that a query for road does not have to rediscover the ground plane from raw pixels every time the lighting changes. CMPE injects depth-derived position into the attention keys, keeping the road–sidewalk boundary stable even when shadow + headlight glare would otherwise shift it by tens of pixels.

Data & baselines. We evaluate on the adverse-condition splits (night, rain, fog) of the ACDC validation set, with 19 Cityscapes-aligned categories. We compare against FC-CLIP and SED using their official released ConvNeXt-L checkpoints, run via Detectron2 against the same prompt set.

50 adverse-condition frames per model. We split the 19 Cityscapes classes into Seen (trainId 0–9: road, sidewalk, building, wall, fence, pole, traffic light, traffic sign, vegetation, terrain) and Unseen (trainId 10–18: sky, person, rider, car, truck, bus, train, motorcycle, bicycle).

The Seen-class margin is large (+0.185 over FC-CLIP, +0.244 over SED) because ground-plane / wall geometry is exactly where depth helps when contrast collapses. On Unseen classes the margin is small but consistent (+0.018 over FC-CLIP) — depth is a structure prior, not a recognition prior, so it does not push the frontier on objects whose shape (not geometry) is the dominant cue.

Columns: RGB · Ground Truth · FC-CLIP · SED · Ours. All masks use the Cityscapes color palette.

Two patterns are worth calling out. First, in nighttime frames both baselines suffer boundary bleed along the road–sidewalk transition; our model's fused depth keeps the ground plane consistent. Second, in rain/fog frames where long-range structure is washed out, the baselines tend to collapse large regions onto a single background class (sky or vegetation), while our model retains the road–vegetation–building partition because depth disambiguates it.

At inference time we monkey-patch the depth-fusion step to replace $D_\ell$ with torch.zeros_like(depth_features), leaving everything else untouched. The model is forced to produce a mask using only the frozen visual stream plus the CMPE/decoder — the depth signal is surgically severed.

Differences appear at the road/sidewalk boundary, in road markings, and along vertical structures (poles, signs). This is a pseudo-ablation: we do not retrain without depth, so the surviving weights were never adapted to live without it. The degradation is therefore an upper bound on the depth branch's contribution — but it is the cleanest white-box evidence that the model actually uses its geometric prior.

A forward hook on the decoder's cross-attention submodule captures the output activation; we normalize, reshape into an $H\times W$ heatmap, colorize with COLORMAP_JET, and blend onto the RGB input.

Honesty note. What we capture is the output activation of the cross-attention submodule, not the softmax-normalized attention tensor; the reshape is heuristic. The most we can claim is that the decoder's cross-attention produces a spatially non-trivial response that broadly follows scene structure — not per-class grounding.

Most of the engineering pain came not from the model itself but from the surrounding reproducibility scaffolding. We summarize the issues that consumed real time and how we resolved them.

1. Scope pivot from SVI to ACDC. The original proposal targeted multi-view Google Street View Imagery, but acquiring and densely annotating adverse-weather SVI within the project timeline was infeasible. After the mid-term review we re-scoped to the ACDC adverse-condition validation set, which is annotated at Cityscapes-trainId granularity and provides a controlled night/rain/fog split that is closer to publishable practice.
2. Baseline reproducibility. Running FC-CLIP and SED faithfully required their official Detectron2 code paths rather than HuggingFace / MMSegmentation re-implementations. This forced a separate ov_baselines conda environment, compiling FC-CLIP's deformable ops, and (for SED) putting its bundled open_clip/src at the front of sys.path so that encode_image(..., dense=True) resolved to the right module. We ended up shelling out to a subprocess so that each baseline runs in its own isolated environment.
3. Class-id alignment across heterogeneous outputs. FC-CLIP produces panoptic outputs, SED produces dense CLIP-feature maps over an ad-hoc vocabulary, and our model emits per-class logits over the 19 Cityscapes trainIds. We had to write a small adapter that maps each baseline's raw output into the same trainId space before any metric computation, otherwise the mIoU comparison would be silently unfair.
4. Confusion-matrix overflow from invalid predictions. Some baseline predictions occasionally fell outside [0, n_cls) (out-of-vocabulary classes leaking through, or 255 void labels copied from the input). Naively passing those into np.bincount blew up the confusion matrix. We added an explicit mask to drop pixels with predictions outside the valid range before accumulation (presentation_common.compute_confusion_matrix).
5. Cross-attention "visualization" was less rigorous than intended. The original plan was to extract softmax-normalized attention weights and slice them per class. In practice the forward hook captures the output activation of the cross-attention submodule, not the attention matrix itself, and reshaping that tensor to H × W is heuristic. Rather than fake the rigorous version, we kept the heuristic heatmap and added an explicit honesty note in the report and on this page.