Teach and Grow: An Agent-Centered Architecture for General Robot Learning

1. Introduction: The Physical Retraining Tax in Embodied AI

In generalist robotics, Vision-Language-Action (VLA) models and World-Action Models (WAM) map visual observations and natural language instructions directly to physical low-level actions. While scaling neural network parameters and pretraining datasets has extended basic robot execution, system reliability remains strictly bounded by validated physical coverage. When a deployed robot encounters out-of-distribution (OOD) conditions—such as an unfamiliar object geometry, novel camera angle, altered material property, or unseen contact dynamics—the policy’s execution frequently departs from its intended goal.

In conventional end-to-end VLA and WAM paradigms, resolving these physical execution failures requires a fresh iteration of data collection, policy fine-tuning, and reinforcement learning or behavior cloning optimization. In their paper “Teach and Grow: An Agent-Centered Architecture for General Robot Learning”, authors Chang Nie, Zhe Liu, and Hesheng Wang term this recurring burden the “retraining tax.”

As defined by Nie et al., the retraining tax manifests because capability updates are stored in the same global, shared neural network parameters as prior behaviors. Consequently, fixing a localized physical failure (e.g., handling a specific novel bowl geometry) requires:

1. Collecting targeted physical robot interaction data or demonstrations.
2. Re-optimizing global policy parameters via gradient updates.
3. Conducting extensive regression testing across all previously mastered tasks to prevent performance degradation.

Because local gaps force global parameter modifications, every update spreads changes unpredictably across the shared network weights. To eliminate this bottleneck, Nie et al. propose Teach-and-Grow Learning (TGL)—a training-free framework that enables robots to acquire, adapt, and retain complex manipulation behaviors without gradient updates, fine-tuning, or model weight modifications.

---

2. The Core Dilemma: End-to-End VLA vs. Agent-Centered Architecture

The fundamental obstacle to scaling end-to-end VLA architectures lies in the structural differences between web-based text/code data and physical robotic data. Large language models achieved broad generalization because web data (text, code, books) was pre-existing, vast, and ready-made. In contrast, physical robotics lacks a ready-made corpus: paired observations, continuous actions, and physical feedback exist only when physical hardware or simulators are explicitly run.

Furthermore, physical interaction conditions multiply exponentially rather than add linearly. Factors such as object pose, camera geometry, environmental clutter, material friction, and robot embodiment interact dynamically. Broad pretraining allows models to interpret instructions or recognize visual elements, but it does not inherently supply the specific geometric mappings, contact feedback, and continuous control necessary for execution in dynamic environments.

Teach-and-Grow Learning (TGL) addresses this core dilemma by shifting task acquisition from parameter fine-tuning to explicit skill abstraction. TGL separates what a task accomplishes semantically from how it is realized physically:

• Semantic Strategy ($\rho_i$): What state changes occur, in what sequence, under what conditions, and what defines success. This remains invariant across different environments.
• Physical Grounding ($\gamma_i$): The precise object poses, grasp points, collision-free paths, and motor commands required for a specific scene. These are computed dynamically from current physical observations ($\mathcal{O}$).

To illustrate this separation, consider placing a novel plush toy into a bowl (as investigated in the paper’s physical manipulation tasks). The high-level semantic strategy ($\rho_i$)—locating the target object, establishing a secure grasp, lifting, aligning above the container, and releasing—remains identical regardless of the specific toy or bowl geometry. However, the physical grounding function ($\gamma_i$) evaluates fresh sensor observations ($\mathcal{O}$) to dynamically extract precise 3D bounding boxes, compute feasible grasp vectors for deformable surfaces, generate collision-free trajectory paths, and execute low-level motor commands.

By decoupling these layers, TGL avoids altering base network parameters when introducing new capabilities. Furthermore, TGL distinguishes between fixed foundation parameters ($\theta$) and separate student policy parameters ($\phi$). While foundation model parameters ($\theta$) remain strictly frozen during task acquisition ($\theta_{n+1} = \theta_n = \theta$), verified execution trajectories generated by the TGL framework can be compiled offline to distill fast feed-forward student policies ($\phi$). This design enables immediate training-free acquisition of novel tasks while preserving a path to fast, feed-forward execution for familiar routines.

Metric / Aspect	End-to-End VLA / WAM Architectures	Teach-and-Grow Learning (TGL)
Model Weights during Adaptation	Retrained / fine-tuned ($\theta$ updated)	Fixed ($\theta$ remains frozen; separate student parameters $\phi$ distilled offline)
Data Burden for New Tasks	Policy optimization data (large-scale physical collection & re-optimization)	Few-shot demonstration inputs (sparse demonstrations used as reasoning inputs)
Failure Isolation	Global weight diffusion (changes spread across shared parameters, requiring full regression testing)	Explicit memory diagnosis/repair (failures logged as structured text context without parameter corruption)
Execution Velocity vs. Task Acquisition	Fast feed-forward execution, but high task acquisition latency due to retraining cycles	Agentic deliberation for novel acquisition with a fast-student distillation path ($\phi$) for familiar execution

---

3. Dissecting the TGL Framework: Abstraction, Memory, and Execution

The TGL architecture keeps pretrained foundation model parameters ($\theta$) entirely fixed during task acquisition. Rather than adjusting neural parameters, task acquisition expands explicit knowledge states through two persistent stores: a Skill Library ($\mathcal{B}$) and an Experience Memory ($\mathcal{M}$).

                             +-------------------------------+
                             |    Pretrained Model Weights   |
                             |          (\theta Fixed)       |
                             +---------------+---------------+
                                             |
                                             v
+-----------------------+        +-----------+-----------+        +-----------------------+
|  Few-Shot Demos (D_n) | -----> |  AI Agent Core (TGL)  | <----> |  Robot Tools & System |
+-----------------------+        +-----------+-----------+        +-----------------------+
                                             |
                                             v
                             +---------------+---------------+
                             |   Persistent State Expansion  |
                             |  (\mathcal{B}, \mathcal{M})   |
                             +---------------+---------------+
                                             |
                     +-----------------------+-----------------------+
                     |                                               |
                     v                                               v
       +-------------+-------------+                   +-------------+-------------+
       |     Skill Library (B)     |                   |    Experience Memory (M)    |
       | (Closed-Loop Skill Blocks)|                   | (Context, Diagnosis, Repair)|
       +---------------------------+                   +---------------------------+

3.1 Closed-Loop Skill Blocks

TGL structures behaviors into modular, reusable components called Skill Blocks. Rather than storing static motor trajectories or joint scripts, a Skill Block is defined as an explicit contract:

$b_i := \langle g_i, S_i, \rho_i, \gamma_i, \Pi_i, v_i, R_i \rangle$

Where:

• $g_i$: The goal specification (the intended physical state change).
• $S_i$: The applicability scope (situations where the block can be safely invoked).
• $\rho_i$: The reusable strategy (the high-level semantic plan synthesized across demonstrations).
• $\gamma_i : \mathcal{O} \rightarrow \mathcal{Z}_i$: Grounding function mapping raw camera/sensor observations ($\mathcal{O}$) to scene-specific physical variables ($\mathcal{Z}_i$, such as object bounding boxes, poses, and grasps).
• $\Pi_i$: A set of compatible robot execution backends (e.g., classical motion planners, geometric grasp solvers, tactile controllers, or specialized neural sub-policies).
• $v_i$: An outcome verifier function mapping pre- and post-action observations to outcome states: $\text{pass}$, $\text{fail}$, or $\text{inconclusive}$.
• $R_i$: A set of bounded recovery choices for handling localized execution failures.

Cross-Demonstration Abstraction Mechanics

Nie et al. formalize how the agent extracts candidate Skill Blocks from sparse human demonstrations. Given a set of $M$ demonstrations for a new task:

$\mathcal{D}_n = \{d^{(1)}, d^{(2)}, \dots, d^{(M)}\}$

Each individual demonstration $d^{(j)}$ is recorded as an ordered sequence of physical state observations:

$d^{(j)} = (s_1^{(j)}, s_2^{(j)}, \dots, s_{K_j}^{(j)})$

The multimodal agent analyzes these sequences to identify discrete semantic state transitions (e.g., object contact, container alignment, latch release). Demonstration trajectories are segmented at these semantic boundaries. Segments across different demonstrations that achieve the same underlying physical effect are grouped into segment clusters $A_k$:

$A_k = \{ \text{segment } k \text{ from } d^{(1)}, \text{ segment } k \text{ from } d^{(2)}, \dots \}$

The agent then synthesizes a unified semantic strategy $\rho_k$ across the aligned cluster:

$\rho_k = \text{Synthesize}(A_k)$

By abstracting common subgoals while discarding scene-specific trajectory variations, $\rho_k$ captures an invariant semantic approach that can be instantiated across diverse physical environments.

3.2 Dual Persistent Stores (Library vs. Memory)

TGL manages lifelong learning through two complementary, persistent text-based data stores:

1. Skill Library ($\mathcal{B}$): Stores validated, closed-loop Skill Blocks ($b_i$). A block is admitted to $\mathcal{B}$ only after passing rigorous physical validation checks outside of its original demonstration scenes.
2. Experience Memory ($\mathcal{M}$): Records specific contextual execution histories, failure logs, and successful repairs. Each record is structured as: $\mu_j = \langle \text{task}, \text{context}, \text{blocks}, \text{outcome}, \text{diagnosis}, \text{repair}, \text{evidence} \rangle$

Nie et al. establish an asymmetric learning rule between these stores:

• Successful Demonstrations teach generalizable strategies ($\rho_i$), expanding executable capability within the Skill Library ($\mathcal{B}$).
• Execution Failures teach bounded conditions ($\mu_j$), updating Experience Memory ($\mathcal{M}$) to prevent similar failures in future contexts.

This asymmetric design prevents catastrophic policy degradation. In traditional VLA models, correcting a localized failure requires modifying shared policy weights, which risks corrupting unrelated behaviors. In TGL, successful executions add or refine reusable behavioral contracts in $\mathcal{B}$, while failures log contextual text filters and repair procedures in $\mathcal{M}$. Because base model parameters ($\theta$) and validated Skill Block contracts remain untouched when logging a failure, error handling remains tightly localized to specific environmental triggers without altering baseline capabilities.

3.3 Agentic Semantic Loop and Dual-Rate Control Dynamics

The runtime implementation of TGL relies on an AI agent using OpenAI GPT-6 Astra for multimodal vision-language reasoning and OpenAI Codex to trigger callable robot tools (perception modules, collision-aware path planners, grasp detectors, and motor controllers).

TGL operates via a dual-rate control architecture:

• Semantic Planning Level ($t$): The high-level LLM agent operates at discrete semantic boundaries ($t$), making deliberative decisions regarding plan synthesis, tool selection, outcome evaluation, and error recovery.
• Continuous Execution Level: Continuous motor control runs natively within selected execution backends ($\pi_{i_t} \in \Pi_i$) at high control frequencies (e.g., 20 Hz – 100 Hz), executing trajectories until a semantic subgoal boundary is reached.

+-------------------------------------------------------------------+
|               AGENTIC SEMANTIC LOOP (GPT-6 Astra)                 |
|  Operates at semantic step boundaries (t)                         |
|  - Retrieves B & M                                                |
|  - Synthesizes plan \tau_t = (b_{i_{t,1}}, ..., b_{i_{t,L_t}})     |
|  - Evaluates verifier feedback v_i(o_{t+1}, \varepsilon_t)         |
|  - Selects routing branch (Continue, Replan, Repair, etc.)        |
+---------------------------------+---------------------------------+
                                  |
              Semantic Subgoal    |    Physical Evidence \varepsilon_t &
              Command (b_{i_t})   |    Observation o_{t+1}
                                  v
+-------------------------------------------------------------------+
|              CONTINUOUS MOTOR EXECUTION ENGINE                    |
|  Operates natively at high control frequency                      |
|  - Selected Tool Engine (\pi_{i_t} \in \Pi_i)                     |
|  - Classical Planners / Grasp Solvers / VLA Controllers           |
+-------------------------------------------------------------------+

When tasked with an instruction, the TGL agent executes the following loop:

1. Plan Generation: The agent retrieves relevant skills from $\mathcal{B}$ and contextual repairs from $\mathcal{M}$ to synthesize an ordered working plan: $\tau_t = (b_{i_{t,1}}, b_{i_{t,2}}, \dots, b_{i_{t,L_t}})$ where $b_{i_{t,1}}$ represents the active lead block in remaining plan $\tau_t$.
2. Scene Grounding & Dual-Rate Execution: The agent grounds the active block ($b_{i_{t,1}}$) in current observation space $\mathcal{O}$, selects an executor ($\pi_{i_t} \in \Pi_i$), and dispatches execution to the low-level controller. At the completion of the execution segment, the backend returns physical evidence $\varepsilon_t$ and updated observations $o_{t+1}$: $(o_{t+1}, \varepsilon_t) = \text{Run}(b_{i_{t,1}}, \pi_{i_t}, t, o_t)$
3. Verification & Decision: The verifier $v_i$ evaluates executor evidence $\varepsilon_t$ and physical observations $o_{t+1}$ at semantic boundary $t$ to return an outcome state ($\text{pass}$, $\text{fail}$, or $\text{inconclusive}$).
4. Adaptive Routing: Based on physical evidence $\varepsilon_t$, the high-level agent selects the next action from six distinct branches:
   • Continue: Proceed to the next block in working plan $\tau_t$.
   • Reobserve: Gather additional sensor readings to resolve ambiguous physical states.
   • Switch tool: Substitute the execution engine $\pi_{i_t}$ within the block.
   • Replan: Modify the remaining plan sequence $\tau_t$.
   • Ask teaching: Request a targeted demonstration from a human teacher if unresolvable gaps remain.
   • Stop: Terminate execution upon reaching the goal or exhausting recovery budgets.

---

4. Empirical Evaluation and the Scaling-Law Hypothesis

To evaluate TGL, Nie et al. conducted extensive experiments using fixed foundation model weights across standard robot manipulation benchmarks.

Benchmark Performance

• LIBERO Benchmark Suites: TGL achieved a 99.9% mean success rate across four benchmark suites (LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-100) without any task-specific weight updates.
• LIBERO-Plus Perturbation Categories: Under severe physical perturbations (including camera shifts, object layout changes, background variations, and clutter), TGL achieved a 92.4% mean success rate across seven perturbation categories.

The Teach-and-Grow Scaling-Law Hypothesis

Beyond empirical benchmark evaluations, Nie et al. introduce a theoretical scaling framework for agent-centered robotics. The authors define effective reusable experience ($X_n$) at checkpoint $n$ as:

$X_n = \sum_{h \in \mathcal{H}_n} \omega(h; \mathcal{B}_n, \mathcal{M}_n)$

Where $\mathcal{H}_n$ is the cumulative interaction history, and $\omega(h; \mathcal{B}_n, \mathcal{M}_n) \in [0, 1]$ is a dimensionless scoring function evaluating evidence reliability, coverage, retrievability, grounding validity, and block compatibility.

Nie et al. establish two formal mathematical constraints on $X_n$:

1. Positive-Weight Condition: The interaction history must contain at least one verified experience at checkpoint $n$ such that $X_n > 0$.
2. Temporal Causality Constraint: The scoring function $\omega$ evaluates history using strictly evidence available at checkpoint $n$. Future task outcomes or non-causal information are explicitly prevented from leaking into historical score evaluations.

Nie et al. formulate the Teach-and-Grow scaling-law hypothesis linking effective experience $X$ to system performance:

$E(X) = E_\infty + A X^{-\alpha}$

$D_{\text{teach}}(X) = D_\infty + B X^{-\beta}$

Where:

• $E(X)$ is the expected execution error rate on related future tasks.
• $D_{\text{teach}}(X)$ is the marginal human teaching effort required to acquire a new task. Operatively, $D_{\text{teach}}$ is defined as human teacher intervention time measured under a standardized protocol until a preregistered task success criterion is met, or capped at a preregistered budget ceiling if the criterion is not reached.
• $E_\infty, D_\infty \ge 0$ represent baseline task-dependent error and teaching floors.
• $A, B, \alpha, \beta > 0$ are constants and power-law scaling exponents.

 Performance Rate / Teaching Effort
  ^
  |  E(X)  [Future Task Error Rate]          = E_\infty + A * X^{-\alpha}
  |  D_teach(X) [Marginal Teaching Demand]  = D_\infty + B * X^{-\beta}
  |
  |---\
  |    \
  |     \
  |      \---.
  |           \-------------------------- (Floors: E_\infty, D_\infty)
  +--------------------------------------------------------->
  0        Effective Reusable Experience (X)

This formulation indicates that as a deployed robot accumulates validated, retrievable skills and memory records ($X$), both task error and human teaching requirements follow power-law reductions. Unlike standard neural network scaling laws—which depend on increasing parameter counts or massive static offline datasets—TGL scaling is driven by structured, reusable interaction experience gathered during task deployment.

---

5. Key Takeaways and Implications for Embodied AI

1. Training-Free Adaptation: Acquiring physical subgoals without gradient updates or parameter fine-tuning avoids catastrophic forgetting, eliminates retraining costs, and prevents policy degradation across prior tasks.
2. Explicit Failure Allocation: Storing execution context, failure diagnoses, and bounded repairs within a structured textual Experience Memory isolates error handling without risking global parameter corruption.
3. Teaching as an Accelerator: Few-shot human demonstrations accelerate task acquisition reliability and lower physical exploration risks, but they serve as efficiency boosters rather than strict prerequisites for agentic execution.
4. Experience-Driven Scaling: Lifelong robotic intelligence can scale predictably through the accumulation of explicit, retrievable, and groundable experience ($X$), providing an alternative to relying solely on scaling model parameter counts or static offline datasets.

Read the full paper on arXiv · PDF