Migrate EmbodiAgent #82

yangchen73 · 2026-01-21T03:14:26Z

Description

This PR migrates EmbodiAgent, a LLM-based agent system for autonomous robot task execution in EmbodiChain. The system enables robots to perceive, plan, code, execute, and validate complex manipulation tasks through a closed-loop control cycle, which is developed by https://github.com/Jasonxu1225 .

Key Features

Three specialized agents working in coordination:

TaskAgent: High-level reasoning and task decomposition from visual observations
CodeAgent: Translates natural language plans into executable Python code using atomic actions
ValidationAgent: Validates task execution and provides feedback for refinement

Key Related Files

Core agent implementation: embodichain/agents/hierarchy/
Agent environments: embodichain/lab/gym/envs/tasks/tableware/
Toolkit interfaces: embodichain/toolkits/interfaces.py
Motion generation: embodichain/lab/gym/motion_generation/
Documentation: embodichain/agents/README.md

Type of change

New feature (non-breaking change which adds functionality)

Screenshots

Pour Water

pour_waterr.mp4

Rearrangement

rearrangement.mp4

Dual Arm Pour Water

pour_water_dual.mp4

Checklist

I have run the black . command to format the code base.
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
Dependencies have been updated, if applicable.

Copilot

Pull request overview

This PR migrates EmbodiAgent, an LLM-based hierarchical agent system for autonomous robot manipulation. The system consists of three coordinated agents (TaskAgent, CodeAgent, ValidationAgent) that enable robots to perceive, plan, generate executable code, and validate task execution through closed-loop control.

Changes:

Added agent system with hierarchy-based task decomposition and code generation
Integrated LangChain dependencies (0.2.14, 0.1.22) and ZeroMQ for agent communication
Implemented toolkit interfaces for robot action primitives (grasp, place, move, etc.)
Added mesh processing utilities and online data generation engine
Created new task environments for pour water and rearrangement with agent support

Reviewed changes

Copilot reviewed 74 out of 75 changed files in this pull request and generated 134 comments.

Show a summary per file

File	Description
pyproject.toml	Added langchain, langchain-openai, zmq, and pycocotools dependencies
embodichain/toolkits/interfaces.py	Atomic robot action functions and drive coordination
embodichain/toolkits/processor/	Mesh processing components and entity management
embodichain/lab/gym/envs/tasks/tableware/	New agent-enabled task environments
embodichain/lab/scripts/run_agent.py	Main script for agent-based task execution
embodichain/data/data_engine/	Online data generation engine with ZeroMQ communication
embodichain/database/agent_prompt/	Prompt templates and examples for LLM agents

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-21T04:49:39Z

embodichain/toolkits/processor/entity/meshobject.py

+                    self.remove_componenet(AxisAlignedBoundingBox)
+                if self.has_component(OrientedBoundingBox):
+                    self.remove_componenet(OrientedBoundingBox)


The method name "remove_componenet" contains a typo. It should be "remove_component" (missing the second 't'). This inconsistency may lead to confusion when the correct spelling "remove_component" is defined at line 143, creating two different method names for what appears to be the same operation.

Copilot · 2026-01-21T04:49:40Z

embodichain/toolkits/processor/function/mesh_processor/processor.py

+            elif self.method == "svd":
+                is_success, mesh_o3dt = cad_standardlize_obb(
+                    mesh_o3d,
+                    is_use_mesh_clean=False,
+                    is_cad_eliminate_symmetry=True,
+                    symmetry_axis=self.symmetry_axis,
+                    is_larger_positive=self.is_larger_positive,
+                )


In the SVD case (line 151), the code calls "cad_standardlize_obb" instead of "cad_standardlize_svd". This appears to be a copy-paste error where both the "obb" and "svd" method options call the same function, making the method parameter ineffective. The SVD case should likely call a different function or this conditional block should be refactored.

Copilot · 2026-01-21T04:49:40Z

embodichain/lab/gym/envs/tasks/tableware/rearrangement_v3.py

+        return (
+            abs(spoon_y - spoon_place_target_y) <= tolerance
+            or abs(fork_y - fork_place_target_y) <= tolerance
+        )


The validation logic appears inverted. The condition "abs(spoon_y - spoon_place_target_y) <= tolerance or abs(fork_y - fork_place_target_y) <= tolerance" uses OR, meaning the task is considered successful if EITHER object is correctly placed. However, based on the task description, both objects should be correctly positioned for success. The condition should use AND instead of OR.

Copilot · 2026-01-21T04:49:40Z

embodichain/lab/gym/motion_generation/action/action.py

+    # def reset(self, env_ids: Sequence[int] | None = None) -> None:
+    #     r"""Resets the manager term.
+
+    #     Args:
+    #         env_ids: The environment ids. Defaults to None, in which case
+    #             all environments are considered.
+    #     """
+    #     pass


This comment appears to contain commented-out code.

Copilot · 2026-01-21T04:49:40Z

embodichain/lab/gym/motion_generation/action/action.py

+    # def process_actions(self, actions: torch.Tensor):
+    #     r"""Processes the actions sent to the environment.
+
+    #     Note:
+    #         This function is called once per environment step by the manager.
+
+    #     Args:
+    #         actions: The actions to process.
+    #     """
+    #     raise NotImplementedError


This comment appears to contain commented-out code.

Copilot · 2026-01-21T04:50:10Z

embodichain/toolkits/processor/component/component.py

+
+
+@dataclass(eq=False)
+class SpatializationComponenet(EntityComponent):


The class 'SpatializationComponenet' does not override 'eq', but adds the new attribute location.
The class 'SpatializationComponenet' does not override 'eq', but adds the new attribute rotation.

Copilot · 2026-01-21T04:50:10Z

embodichain/data/data_engine/online/online_generator.py

+                self.queue.remove(shm_name["name"])
+            try:
+                shm = shared_memory.SharedMemory(shm_name["name"])
+            except:


Except block directly handles BaseException.

Copilot · 2026-01-21T04:50:11Z

embodichain/lab/gym/motion_generation/action/arm_action.py

+                rot_matrix = start_pose[:3, :3].T @ end_pose[:3, :3]
+                angle = rotation_matrix_to_angle(rot_matrix)
+                total_angle += angle
+            except Exception:


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-01-21T04:50:11Z

embodichain/data/data_engine/datasets/sim_real_unified_dict_dataset.py

+                        data = CompressedVideoHDF5(file_path, chunks=None).safe_filter(
+                            data, step_id
+                        )
+                except Exception:


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-01-21T04:50:11Z

embodichain/data/data_engine/data_dict_extractor.py

+            state_max = np.max(states, axis=0)
+            state_min = np.min(states, axis=0)
+        else:
+            state_max = np.maximum(state_max, np.max(states, axis=0))


This statement is unreachable.

yuecideng · 2026-01-21T05:18:07Z

embodichain/data/data_engine/__init__.py

@@ -0,0 +1,5 @@
+# ----------------------------------------------------------------------------


Use the correct license

yuecideng · 2026-01-21T05:20:19Z

embodichain/data/data_engine/__init__.py

@@ -0,0 +1,5 @@
+# ----------------------------------------------------------------------------
+# Copyright (c) 2021-2025 DexForce Technology Co., Ltd.


Use lerobot format for now. And remove the hdf5 data recordor.

yangchen73 added 30 commits January 21, 2026 10:37

Migrate the config of tasks

3cbbc21

Migrate execute script

f4dbddf

Migrate base_agent_env

8286998

Migrate prompt template

9af2785

Migrate the core code of agent

60c168c

Migrate prompt file

ee5a568

Update pyproject and gitignore

b1212af

Migrate generate video script

52a2b45

Migrate data engine

0b65baa

Fix: can not process 'func'

8bfeb7f

Migrate indices and mapping

8dba0da

Migrate environments: pour water and rearrangement

4195d30

Migrate API interfaces

96bb2b9

Migrate object

74eee5e

Migrate object

3421022

Migrate necessory files in toolkits

fa8f0fe

Migrate motion generation part

302608a

migrate lab.sim.utility

5baed9b

Migrate necessory files in toolkits

8411a35

Config LLM

64642ca

Extend enum

c3898b6

Add direction of database

2839548

Update enum

18cfc03

Migrate end effector

2ede9a0

Migrate robot

983fd9f

Update direction of database

974ee5f

Migrate robot interface

40957f1

Migrate two get control information function in action_bank utils

d4a3ae5

Migrate misc

c4eb89f

Fix: change data configs to functor format

189965d

yangchen73 added 19 commits January 21, 2026 10:39

Fix: use get_wrapper_attr for wrapped env methods in run_agent.py

4a2d0a9

Fix: missing task plan

c2d0e00

Fix: module-level drive() calls

7bbcda3

Fix: import direction of data_dict_extractor

d95188d

Fix: to_dataset function of base_agent_env

a2392bf

Fix: argument of mul_linear_expand should be 2

038e35b

Migrate articulation entity

91ec67d

Remove correction functions of agent

50ef839

Add generated code and plan

0fc7d8f

Remove the function of correction

f9bbc14

Add script of running feedback agent

11fc818

Keep components in sim the same

4092d31

Fix: use environment variable to add api key

6ded707

Add Readme for EmbodiAgent

a25bf61

Remove generated prompt

0d85e00

Remove scripts with feedback

7542965

Reformat files

e0fac71

Fix: docs build without API keys

037c97b

Reformat files

ba889e3

yuecideng requested review from Jasonxu1225, Copilot and yuecideng January 21, 2026 04:41

Copilot started reviewing on behalf of yuecideng January 21, 2026 04:42 View session

yuecideng requested a review from yhnsu January 21, 2026 04:43

Copilot AI reviewed Jan 21, 2026

View reviewed changes

yuecideng added gym robot learning env and its related features agent Features related to agentic system labels Jan 21, 2026

yuecideng requested changes Jan 21, 2026

View reviewed changes



		@dataclass(eq=False)
		class SpatializationComponenet(EntityComponent):

		@@ -0,0 +1,5 @@
		# ----------------------------------------------------------------------------

		@@ -0,0 +1,5 @@
		# ----------------------------------------------------------------------------
		# Copyright (c) 2021-2025 DexForce Technology Co., Ltd.

Migrate EmbodiAgent #82

Are you sure you want to change the base?

Migrate EmbodiAgent #82

Conversation

yangchen73 commented Jan 21, 2026

Description

Key Features

Key Related Files

Type of change

Screenshots

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

yuecideng Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

yuecideng Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants