SpaceMind: A Modular and Self-Evolving Embodied Vision-Language Agent Framework for Autonomous On-orbit Servicing

The low Earth orbit (LEO) environment is approaching a critical threshold. As of 2023, there are over 20,000 cataloged objects—ranging from aging satellites to hazardous debris—threatening global orbital infrastructure. Addressing this crisis requires a paradigm shift toward On-Orbit Servicing, Assembly, and Manufacturing (OSAM). However, current autonomous systems are largely tethered to pre-programmed sequences designed for cooperative, well-known targets. These legacy architectures fail in the unpredictable, high-stakes environment of space, where agents must navigate 6-degree-of-freedom (6-DoF) physical space over long horizons.

Traditional AI agents struggle in orbit because monolithic prompting strategies are unmaintainable for complex visuomotor control and unpredictable scenarios. To bridge the gap between static simulation and real-world autonomy, we have developed SpaceMind: a modular, self-evolving Vision-Language Model (VLM) framework designed as a decision-control hub for the next generation of autonomous space robotics.

---

The Blueprint: A Four-Layered Approach to Modularity

In safety-critical robotics, modularity is not just a design preference; it is a requirement for scalability. SpaceMind moves away from unmanageable “monolithic prompts” by decomposing agent knowledge, tools, and reasoning into a structured hierarchy. This architecture, validated through 192 total closed-loop runs, ensures that the system remains auditable and extensible.

The SpaceMind framework is built on four distinct design layers:

• Skill Layer: Decomposes knowledge into a three-tier taxonomy—Core (safety and conventions), Task (mission objectives like rendezvous), and Helper (strategies like target-loss recovery). An LLM-based Skill Gateway performs dynamic routing to inject only the relevant modules into the active prompt.
• VLM Decision Core: The “brain” of the system, hosting switchable reasoning modes. This allows the system to adjust its cognitive depth—from direct action selection to deliberative planning—based on mission difficulty.
• MCP Tool Layer: Leveraging the Model Context Protocol (MCP), SpaceMind standardizes tool calls for perception, control, and domain knowledge. This abstraction allows for controlled ablation studies; for example, we can declaratively remove LiDAR range sensing to test vision-only robustness without changing a single line of core logic.
• Interface Layer: A Redis-based message bus serves as an environment-agnostic bridge. This decoupling facilitates zero-code-modification transfer, allowing the same agent loop to transition seamlessly between high-fidelity UE5 simulations and physical laboratory hardware.

---

Cognitive Depth: Matching Reasoning to the Mission

SpaceMind allows operators to inject specific reasoning “modes” to match the operational context. Our systematic evaluation across five satellite models (CAPSTONE, IBEX, BioSentinel, New Horizons, and Huygens) demonstrates that no single reasoning strategy is universal.

Reasoning Mode	Operational Mechanism	Best Use Case
Standard	Direct decision-making; direct mapping of observations to actions.	Nominal rendezvous with visible, cooperative targets.
ReAct	Iterative “Thought–Action–Observation” loops.	Close-range inspection requiring step-by-step verification.
Prospective	Deliberative “Plan–Score–Select” mechanism.	Search-and-approach under degraded visual conditions (C2/C3).

Actionable Findings from 135 Evaluation Runs

• Standard Mode is the most efficient for nominal rendezvous, achieving 100% success when targets are visible. However, its search success drops to zero under lateral offsets or lighting degradation.
• Prospective Mode is uniquely robust in search-and-approach tasks. By evaluating multiple movement hypotheses before committing, it avoids the “blind exploration” that causes other modes to time out.
• ReAct Mode is highly effective at preventing hallucinations during inspection under poor visibility. It utilizes a “rapid termination strategy” (limiting internal loops to 1–3 steps) to prevent the accumulation of visual errors. However, ReAct carries an oscillation risk; it can become trapped in repetitive loops when a target is far from its expected position.

---

Skill Self-Evolution: Learning from Failure without Retraining

The most significant barrier to on-orbit autonomy is the inability of agents to learn from experience without costly gradient updates. SpaceMind’s Skill Self-Evolution “Outer Loop” allows the agent to reflect on its own operational history and improve autonomously.

The Post-Episode Self-Evolution pipeline follows four rigorous steps:

1. Episode Summarization: Recording the full trajectory, tool-call logs, and outcomes.
2. Experience Reflection: A VLM analyzes the logs to determine why a failure occurred or how a success was achieved.
3. The Quality Gate: Proposals must pass fingerprint deduplication (to prevent redundant knowledge) and task-scope binding (to ensure a skill learned for one satellite doesn’t contaminate the knowledge base of an unrelated mission).
4. Persistent Skill Generation: New knowledge is materialized as structured .md skill files, which are automatically loaded into the Skill Layer for future missions.

This mechanism has demonstrated the ability to recover from complete failure. In our New Horizons test group, the agent transitioned from 0% to 100% success after a single failed episode reflection by autonomously discovering a distance-dependent step-size reduction strategy. Additionally, inspection scores saw a massive jump, moving from 12 to 59 out of 100 (reaching as high as 68 in cumulative assessments), proving that self-evolution can refine perception as effectively as navigation.

---

From UE5 to the Lab: Real-World Validation

To demonstrate the framework’s portability, SpaceMind was deployed on a myAGV Pro mobile robot in a laboratory environment featuring 3D-printed mockups of CAPSTONE and Artemis.

While the UE5 simulation operates in full 6-DoF, the physical laboratory operates in a 3-DoF planar workspace. Despite this difference, the architecture’s modularity enabled:

• Zero-Code-Modification Transfer: The identical agent loop, skill definitions, and MCP tool signatures used in simulation were deployed to the physical robot without modification.
• Emergent Scale Awareness: Without manual re-tuning, the agent autonomously adapted its movement scale—transitioning from taking meter-scale steps in simulation to centimeter-scale steps in the lab by inferring the environment’s scale from its visual sensors.
• Reliable Performance: The agent achieved a 100% success rate for rendezvous tasks in the physical testbed, confirming that the abstraction layers effectively handle the transition from digital to physical environments.

---

Conclusion: Implications for AI Safety and Future Autonomy

SpaceMind provides a blueprint for deploying foundation models in safety-critical, embodied domains. For the AI research and space communities, three takeaways are critical:

1. Modularity is a Requirement for Scalability: Decoupling reasoning modes from tool protocols allows for safer ablation and targeted updates in environments where monolithic systems are unmanageable.
2. Self-Evolution as a Safety-Critical Recovery Mechanism: By turning failures into auditable, persistent skill files, we move toward systems that can recover from non-nominal conditions without human intervention.
3. The Importance of Stress-Testing: Reliability can only be measured under degraded conditions. SpaceMind’s success was only proven by testing against overexposure, underexposure, and significant positional offsets.

As we look toward complex On-orbit Servicing, Assembly, and Manufacturing (OSAM), the ability for an agent to serve as a self-improving decision-control hub will be the difference between mission success and orbital catastrophe. SpaceMind proves that with the right modular architecture, AI can finally evolve to meet the demands of the final frontier.

Read the full paper on arXiv · PDF