SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation

1. Introduction: The Generalization Gap in Object-Goal Navigation

Object-Goal Navigation (ObjNav) requires embodied agents to autonomously locate and reach specified targets using only egocentric visual observations. As analyzed by Su et al., traditional monolithic methods—primarily those utilizing end-to-end Deep Reinforcement Learning (DRL)—suffer from a significant generalization gap. These systems typically optimize reactive mappings based on immediate visual contexts, lacking explicit scene-level semantic modeling. Consequently, when confronted with unseen environments, these agents struggle with distribution shifts that hinder both plausible target location inference and long-horizon reasoning for out-of-view objects.

To address these failure modes, the paper introduces SAGE-Nav, a hierarchical framework that decouples macroscopic semantic reasoning from high-frequency reactive control. By integrating Large Language Models (LLMs) with dynamic Hierarchical Scene Graphs (HSGs), SAGE-Nav attempts to bridge the gap between abstract planning and atomic execution. This report evaluates the methodology, performance, and robustness boundaries of the SAGE-Nav system.

2. Architectural Decoupling: The Global-Local Hierarchy

The authors adopt a strategy of functional and temporal decoupling to manage the complexity of indoor navigation. This hierarchy is organized into three specialized modules: the Hierarchical Global Planner (H-GP), the Hierarchical Scene Graph Encoder (HSGE), and the Goal-aware Alignment-Fusion Network (GAFN).

The following table contrasts the two distinct layers of the SAGE-Nav architecture:

Feature	High-Level Planning (LLM/H-GP)	Low-Level Control (DRL/GAFN)
Frequency	Asynchronous (Triggered by subgoal completion or deadlocks)	High-frequency reactive loop
Function	Decomposes abstract instructions into semantic waypoints; unseen object reasoning.	Executes atomic actions (e.g., MoveAhead, Rotate) based on fused state representations.
Grounding	Retrieval-Augmented Generation (RAG) over Hierarchical Scene Graphs; commonsense priors.	Real-time egocentric visual observations (CLIP features) aligned with structural priors.

3. The Semantic Blueprint: LLM-Guided Global Planning

The core of the framework’s reasoning capability is the Hierarchical Scene Graph (HSG), which structures the environment into three levels: object, cluster, and area.

Retrieval-Augmented Subgraph Logic To prevent policy oscillation and minimize LLM latency, the system extracts an instruction-relevant “compact subgraph” ($G_Q$). Su et al. define a context-augmented semantic prior $S(v)$ for each node, which aggregates the priors of its spatial neighborhood $N(v)$: $S(v) = (1 + \beta \cdot \frac{1}{|N(v)|} \sum_{u \in N(v)} s_u) s_v$ This spatial boosting ensures that the LLM receives only the most topologically and semantically relevant context, preventing the agent from becoming overwhelmed by irrelevant environmental data.

Traveling Salesperson Path Problem (TSPP) Navigation is formulated as a TSPP over the subgraph. The agent sequences waypoints by maximizing a Hierarchical Coherence Weight ($W$), defined as: $W(v_i, v_j) = \lambda_{sem} \cdot S_{multi-level}(v_i, v_j) - \lambda_{dist} \cdot D_{geo}(v_i, v_j)$ For safety researchers, this equation reveals a predictable trade-off: $\lambda_{sem}$ prioritizes functional flow (e.g., moving from a “Stove” to a “Sink” in a kitchen context), while $\lambda_{dist}$ penalizes geometric distance. A red-teamer might exploit this by designing environments where low-cost geometric paths are semantically misleading, potentially inducing sub-optimal or circular behaviors.

Unseen Object Reasoning When a target is out-of-view, the system utilizes a semantic compatibility prior $\phi(q, c)$ to infer likely locations (e.g., searching for a “Remote” near a “Sofa” cluster). This allows for target-guided exploration in novel environments based on LLM-derived commonsense co-occurrence.

4. Encoding and Alignment: Bridging Perception and Structure

The transition from abstract plan to local action is managed by two primary neural components:

Hierarchical Scene Graph Encoder (HSGE) The HSGE utilizes a 3-layer Relational Graph Convolutional Network (R-GCN) to produce structure-aware embeddings. Crucially, the authors include residual terms ($W_{res}x_v$) in the message-passing layers. This design choice is vital for robustness; it mitigates the “over-smoothing” effect common in deep graph networks, preserving the raw multi-modal context of semantic anchors so the agent does not “lose” its specific target identity during deep propagation.

Goal-aware Alignment-Fusion Network (GAFN) The GAFN implements a two-step process to align real-time perception with structural memory:

1. Structure-Infused Cross-Attention: Uses visual features as a query against waypoint embeddings to ground topological guidance in the perceptual space.
2. Alignment-Aware Gated Fusion: Employs an adaptive gating vector $\lambda(\alpha_t)$ based on a real-time alignment score (cosine similarity $\alpha_t$): $\lambda(\alpha_t) = \sigma(MLP(\dots)) \cdot (1 - \eta\sigma(\kappa\alpha_t))$ With hyperparameters $\eta=0.5$ and $\kappa=2.0$, this explicit penalty term $(1 - \eta\sigma(\kappa\alpha_t))$ drives the agent to shift its reliance from raw reactive perception to structural context as it approaches the goal, stabilizing the final approach.

5. Empirical Benchmarks and Zero-Shot Performance

SAGE-Nav was evaluated in the i-THOR and RoboTHOR environments, demonstrating state-of-the-art (SOTA) performance in long-horizon tasks ($L \geq 5$m).

Method	i-THOR (ALL) SR / SPL	i-THOR ($L \geq 5$) SR / SPL
SAGE-Nav (Ours)	82.47% / 42.34%	77.22% / 43.73%
TSOG	80.04% / 41.44%	73.46% / 43.66%
CGI-GAIL	77.59% / 46.25%	69.18% / 46.10%

The authors note that while CGI-GAIL maintains a marginally higher SPL, this is attributed to its reliance on imitation learning from expert demonstrations. SAGE-Nav prioritizes zero-shot capability, achieving superior Success Rates (SR) without expert supervision. In zero-shot tests involving six unseen categories (Bowl, DeskLamp, Laptop, LightSwitch, Plate, and StoveBurner), SAGE-Nav achieved a 75.05% SR, validating the transferability of its LLM-driven semantic priors.

6. The Failure-First Perspective: Analyzing Robustness Gaps

Qualitative analysis by Su et al. identified four critical failure categories that define the system’s operational boundaries:

• Core Failure Modes Identified by Su et al.:
  • Target Visibility Failure: The agent terminates the episode prematurely when the target is technically within distance but occluded or out-of-view. Example: Mistakenly issuing the “Done” command for a plate located on a high shelf.
  • Detection Error: False positives in the object detector between semantically similar objects. Example: Confusing a couch for a chair in a living room.
  • Entrapment: Behavioral deadlocks where the agent cannot navigate out of complex or narrow starting geometries (e.g., narrow corridors).
  • Premature Termination: Miscalculation of spatial proximity, where the agent stops just outside the success threshold ($d_s = 1.5$m).

Future mitigations proposed by the authors include active vision strategies for viewpoint adjustment and 3D volumetric representations to improve obstacle avoidance in constrained spaces.

7. Efficiency and Practical Deployment Implications

SAGE-Nav is designed for real-time deployment on physical hardware, specifically achieving high performance-to-efficiency ratios.

Efficiency Summary: Utilizing the Qwen2.5-VL-7B model on a single NVIDIA RTX 3090 GPU, SAGE-Nav maintains a low control latency of 0.42s per step. By decoupling global reasoning, it requires only 9.1 LLM calls per episode—a significant reduction compared to SG-Nav (32 calls) and CogNav (15 calls).

8. Conclusion: Key Takeaways for AI Safety and Robotics

The SAGE-Nav framework illustrates that hierarchical decoupling is a robust pathway for overcoming the generalization gap in embodied AI. By isolating semantic reasoning from reactive execution, the system maintains high signal integrity in novel environments.

Practitioner Takeaways:

1. Semantic Priors as Anchors: LLM commonsense allows agents to handle out-of-distribution objects by inferring logical search locations, significantly improving zero-shot reliability.
2. Adaptive Gating for Stability: The use of an alignment score and explicit penalty terms provides a tunable mechanism for shifting agent behavior from exploration to goal-directed precision.
3. Hardware-Grounded Latency: Deployment of LLMs in robotics requires asynchronous planning to keep control loops under the 0.5s threshold required for safe physical movement.

---

Attribution: This report is based on the paper “SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation” by Su et al., accepted for IROS 2026.

---

Read the full paper on arXiv · PDF