Qualitative Results on GTA5→{Cityscapes, BDD100K, Mapillary}
![](./static/images/qualitative-results/city-1-input.png)
![](./static/images/qualitative-results/city-1-rein.png)
![](./static/images/qualitative-results/city-1-tqdm.png)
![](./static/images/qualitative-results/city-1-gt.png)
![](./static/images/qualitative-results/city-2-input.png)
![](./static/images/qualitative-results/city-2-rein.png)
![](./static/images/qualitative-results/city-2-tqdm.png)
![](./static/images/qualitative-results/city-2-gt.png)
![](./static/images/qualitative-results/city-3-input.png)
![](./static/images/qualitative-results/city-3-rein.png)
![](./static/images/qualitative-results/city-3-tqdm.png)
![](./static/images/qualitative-results/city-3-gt.png)
![](./static/images/qualitative-results/bdd-1-input.jpg)
![](./static/images/qualitative-results/bdd-1-rein.jpg)
![](./static/images/qualitative-results/bdd-1-tqdm.jpg)
![](./static/images/qualitative-results/bdd-1-gt.png)
![](./static/images/qualitative-results/map-1-input.jpg)
![](./static/images/qualitative-results/map-1-rein.jpg)
![](./static/images/qualitative-results/map-1-tqdm.jpg)
![](./static/images/qualitative-results/map-1-gt.png)
![](./static/images/qualitative-results/map-3-input.jpg)
![](./static/images/qualitative-results/map-3-rein.jpg)
![](./static/images/qualitative-results/map-3-tqdm.jpg)
![](./static/images/qualitative-results/map-3-gt.png)
![](./static/images/qualitative-results/pallete.png)
Qualitative Results under Extreme Domain Shifts (trained on GTA5)
![](./static/images/qualitative-results/gpt-2-input.png)
![](./static/images/qualitative-results/gpt-2-rein.png)
![](./static/images/qualitative-results/gpt-2-tqdm.png)
![](./static/images/qualitative-results/gpt-3-input.jpg)
![](./static/images/qualitative-results/gpt-3-rein.png)
![](./static/images/qualitative-results/gpt-3-tqdm.png)
![](./static/images/qualitative-results/gpt-1-input.png)
![](./static/images/qualitative-results/gpt-1-rein.png)
![](./static/images/qualitative-results/gpt-1-tqdm.png)
![](./static/images/qualitative-results/pallete.png)
In this paper, we introduce a method to tackle Domain Generalized Semantic Segmentation (DGSS) by utilizing domain-invariant semantic knowledge from text embeddings of vision-language models. We employ the text embeddings as object queries within a transformer-based segmentation framework (textual object queries). These queries are regarded as a domain-invariant basis for pixel grouping in DGSS. To leverage the power of textual object queries, we introduce a novel framework named the textual query-driven mask transformer (tqdm). Our tqdm aims to (1) generate textual object queries that maximally encode domain-invariant semantics and (2) enhance the semantic clarity of dense visual features. Additionally, we suggest three regularization losses to improve the efficacy of tqdm by aligning between visual and textual features. By utilizing our method, the model can comprehend inherent semantic information for classes of interest, enabling it to generalize to extreme domains (e.g., sketch style). Our tqdm achieves 68.9 mIoU on GTA5→Cityscapes, outperforming the prior state-of-the-art method by 2.5 mIoU.
@article{pak2024textual,
title = {Textual Query-Driven Mask Transformer for Domain Generalized Segmentation},
author = {Pak, Byeonghyun and Woo, Byeongju and Kim, Sunghwan and Kim, Dae-hwan and Kim, Hoseong},
journal = {arXiv preprint arXiv:2407.09033},
year = {2024}
}