Education, Science, Technology, Innovation and Life
Open Access
Sign In
  1. Home
  2. Journals
  3. Journal of Image Processing Theory and Applications
  4. Vol 9, Issue 1, 2026

Stratified Evaluation of SAM 2 for Zero-Shot Building Segmentation in Aerial Imagery

Download as PDF

DOI: 10.23977/jipta.2026.090104 | Downloads: 2 | Views: 162

Author(s)

Bingning Xiong 1, Mingyu Ou 1

Affiliation(s)

1 Beijing Technology and Business University, Beijing, China

Corresponding Author

Bingning Xiong

ABSTRACT

Building footprint extraction from remote sensing imagery underpins urban planning, population estimation, and disaster damage assessment. Deep learning methods have achieved high accuracy for this task, but their dependence on large-scale pixel-level annotations creates a severe bottleneck: annotating a city-scale dataset demands hundreds of hours of manual labor, limiting rapid deployment to new regions. The Segment Anything Model 2 (SAM 2), a foundation model with zero-shot segmentation capability, offers a potential solution by eliminating the need for task-specific annotations entirely. Yet SAM 2 was trained exclusively on natural scene images and videos, raising a critical question: can it generalize to the fundamentally different visual characteristics of aerial remote sensing imagery? This paper presents the first systematic zero-shot evaluation of SAM 2 for aerial building segmentation. We conduct three groups of experiments: (1) benchmarking four SAM 2 model variants to identify the optimal accuracy-efficiency trade-off; (2) stratified evaluation across dense, sparse, large-scale, and small-scale building morphologies to reveal which architectural characteristics challenge SAM 2 most; and (3) comparison of single-point, multi-point, and bounding-box prompting strategies to derive practical guidelines. Results demonstrate that SAM 2 Base+ achieves an IoU of 0.7738 without any training data, while oracle bounding-box prompting reaches 0.8755. SAM 2 excels on dense and large-scale buildings but struggles with sparse scenes. These findings establish SAM 2 as a viable tool for rapid building mapping while highlighting where domain adaptation remains necessary.

KEYWORDS

SAM 2, Building Segmentation, Zero-Shot Learning, Aerial Imagery

CITE THIS PAPER

Bingning Xiong, Mingyu Ou. Stratified Evaluation of SAM 2 for Zero-Shot Building Segmentation in Aerial Imagery. Journal of Image Processing Theory and Applications (2026) Vol. 9, No.1, 31-40. DOI: http://dx.doi.org/10.23977/jipta.2026.090104.

REFERENCES

[1] S. Ji, S. Wei, and M. Lu, "Fully convolutional networks for multi-source building extraction from an open aerial and satellite imagery dataset," IEEE Trans. Geosci. Remote Sens., vol. 57, no. 1, pp. 574-586, 2018.
[2] T. Esch et al., "Delineation of urban footprints from TerraSAR-X data," IEEE Trans. Geosci. Remote Sens., vol. 48, no. 2, pp. 905-916, 2009.
[3] X. X. Zhu et al., "Deep learning in remote sensing: A comprehensive review," IEEE Geosci. Remote Sens. Mag., vol. 5, no. 4, pp. 8-36, 2017.
[4] G. S. Xia et al., "AID: A benchmark for aerial scene classification," IEEE Trans. Geosci. Remote Sens., vol. 55, no. 7, pp. 3965-3981, 2017.
[5] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmentation," in Proc. MICCAI, 2015, pp. 234-241.
[6] L. Chen et al., "Encoder-decoder with atrous separable convolution," in Proc. ECCV, 2018, pp. 801-818.
[7] J. Wang et al., "Deep high-resolution representation learning," IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 10, pp. 3349-3364, 2020.
[8] N. Ravi et al., "SAM 2: Segment anything in images and videos," arXiv preprint arXiv:2408.00714, 2024.
[9] A. Kirillov et al., "Segment anything," in Proc. ICCV, 2023, pp. 4015-4026.
[10] L. P. Osco et al., "The segment anything model (SAM) for remote sensing applications," Int. J. Appl. Earth Obs. Geoinf., vol. 124, p. 103,540, 2023.
[11] S. Ren et al., "Segment anything, from space?" in Proc. WACV, 2024, pp. 3107-3117.
[12] C. Ryali et al., "Hiera: A hierarchical vision transformer," in Proc. ICML, 2023, pp. 29,461-29,472.
[13] M. Kass, A. Witkin, and D. Terzopoulos, "Snakes: Active contour models," Int. J. Comput. Vis., vol. 1, no. 4, pp. 321-331, 1988.
[14] J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proc. CVPR, 2015, pp. 3431-3440.
[15] A. Dosovitskiy et al., "An image is worth 16x16 words," in Proc. ICLR, 2021.
[16] T. Chen et al., "SAM-Adapter: Adapting segment anything," in Proc. ICCV Workshop, 2023, pp. 3367-3377.
[17] Y. Zhang et al., "Segment anything model-based building footprint extraction," Remote Sens., vol. 16, no. 14, p. 2661, 2024.
[18] F. Panangian and K. Bittner, "Segment anything for satellite imagery," arXiv preprint arXiv:2506.16318, 2025.
[19] M. Illarionova et al., "Segmenting forest canopies with SAM and LiDAR," Int. J. Appl. Earth Obs. Geoinf., 2024.
[20] Meta AI, "SAM 2.1 checkpoints," 2024. [Online]. Available: https://github.com/facebookresearch/segment-anything-2
[21] Google Research, "Google Colaboratory," 2024. [Online]. Available: https://colab.research.google.com
[22] Z. Wang et al., "Deep learning-based interactive segmentation in remote sensing," arXiv preprint arXiv:2308.13174, 2023.
[23] A. Yarroudh, "LiDAR automatic unsupervised segmentation using SAM," GitHub repository, 2023.

Subscription
E-Mail Alert
Most popular papers
Call for Special Issue
Publication Ethics & OA Statement
Plagiarism Policy
Contact Us
Downloads: 3171
Visits: 261344

Sponsors, Associates, and Links

All published work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright © 2016 - 2031 Clausius Scientific Press Inc. All Rights Reserved.