PAIRS Lab

Survey Robotics and Autonomous Systems · Vol. 203 · 2026 · Open access

Semantic Visual SLAM

A survey on state of the art, challenges, and future directions

Thanh Nguyen Canh1,*, Haolan Zhang1, Xiem HoangVan2,*, Nak Young Chong1,3

1Japan Advanced Institute of Science and Technology   2VNU University of Engineering and Technology   3Hanyang University   *Corresponding authors

Paper arXiv Paper list Explore the systems
34
systems compared
4
system families
5
pipeline stages
401
references
6
open questions

01 · Overview

Robots that know where they are and what is around them

Semantic visual SLAM localizes a robot and builds a map that also carries meaning: objects, classes and places. Although the first semantic SLAM systems appeared more than two decades ago, the field has lacked a survey that covers both its recent advances and its persistent problems.

This survey traces how visual SLAM evolved, critically reviews earlier surveys, and proposes a unified problem formulation. It then decomposes the problem into a five-stage modular pipeline, covers deep-learning, foundation-model, continuous-representation and multi-robot approaches, reviews datasets and metrics, and closes with open research questions.

A

Unified formulation

Semantic vSLAM as joint estimation of the robot trajectory, a geometric map and the semantic labels of its elements.

B

Modular pipeline

Five stages, from semantic extraction to loop closure, each compared across methods in its own table.

C

Beyond the pipeline

Learning-based, foundation-model, NeRF/3DGS and multi-robot systems, plus the questions that remain open.

02 · Taxonomy

How tightly are geometry and semantics coupled?

The survey organizes approaches by the coupling level between the geometric and semantic components.

Semantic visual SLAM approaches
Loosely coupledSemantics assist or post-process geometry
Geometric-centricSemantic-assisted
  • Semantic-based loop closure
  • Map labeling fusion
Learning-based componentsPlug-in deep modules
  • DL perception for dynamic environments
  • Semantic feature learning
Tightly coupledSemantics inside the estimation
Joint optimizationGeometry + semantics
  • Semantic residuals in factor graphs
  • Object-level SLAM
End-to-end learningDifferentiable systems
  • Differentiable SLAM networks
  • Neural radiance fields for SLAM

Redrawn from Fig. 1 of the paper.

03 · Pipeline

Five interconnected stages

Information flows forward from perception to optimization, and feedback loops let later stages refine earlier ones.

Semantic vSLAM pipeline: semantic extraction, semantic localization, semantic mapping, data association, and loop closure optimization, with forward data flow and feedback loops.
Fig. 2 of the paper (CC BY 4.0): solid arrows show forward data flow, dashed arrows show feedback.

04 · Evolution

From object maps to foundation models

The 34 representative systems in Table 1 by year and family. Hover or tap a system to see its reference.

05 · Explorer

Explore the systems

Filter Table 1 by family and capability, or search by name, sensor model or map type. Press / to search.

SystemYearFamilySLAM approachPerceptionMapBack-endDyn.Open-voc.LoopLargeCode

06 · Compare

Methods at each stage

Tables 3–7 of the paper, one per pipeline stage.

07 · Open questions

Where the field goes next

08 · Cite

Citation

@article{canh2026semantic,
  title   = {Semantic Visual Simultaneous Localization and Mapping: A survey on state of the art, challenges, and future directions},
  author  = {Canh, Thanh Nguyen and Zhang, Haolan and HoangVan, Xiem and Chong, Nak Young},
  journal = {Robotics and Autonomous Systems},
  volume  = {203},
  pages   = {105535},
  year    = {2026},
  doi     = {10.1016/j.robot.2026.105535}
}