diff --git a/src/iiwa_llm_plan_executor/config/planner.yaml b/src/iiwa_llm_plan_executor/config/planner.yaml index d16de00..7b0df8c 100644 --- a/src/iiwa_llm_plan_executor/config/planner.yaml +++ b/src/iiwa_llm_plan_executor/config/planner.yaml @@ -7,8 +7,8 @@ max_steps: 512 scene_freshness_sec: 2.0 close_inspection_objects: - cap -top_camera_topic: /d455_top/image_raw -hand_camera_topic: /camera/color +top_camera_topic: /d455_top/image_raw/image_color +hand_camera_topic: /camera/color/image_color frame_timeout_sec: 2.0 trace_dir: ~/.ros/iiwa_llm_plan_executor/runs planning_group: iiwa_arm diff --git a/src/iiwa_llm_plan_executor/doc/README.md b/src/iiwa_llm_plan_executor/doc/README.md index 3e96874..464bbbd 100644 --- a/src/iiwa_llm_plan_executor/doc/README.md +++ b/src/iiwa_llm_plan_executor/doc/README.md @@ -97,6 +97,9 @@ The JSONL log with every event is written to The executor checks J7 (±90° around `work`) and the downward TCP tilt (≤10°) on **every** planned point, seeds IK from current joints, enforces these as MoveIt path constraints, checks scene/joint freshness and verifies the final -pose. It never falls back to an unconstrained `MoveToPose` goal. This is a +pose. The only logged exception is the initial empty-gripper reorientation +from home to the downward `work` posture; J7 and the final orientation are +still enforced and a reorientation with a held object is rejected. The +executor never falls back to an unconstrained `MoveToPose` goal. This is a Webots verification, not a physical-robot validation and not a Qwen3 quality evaluation. diff --git a/src/iiwa_llm_plan_executor/doc/webots-qa.md b/src/iiwa_llm_plan_executor/doc/webots-qa.md new file mode 100644 index 0000000..05eeb16 --- /dev/null +++ b/src/iiwa_llm_plan_executor/doc/webots-qa.md @@ -0,0 +1,117 @@ +# Webots QA: external graph execution + +Scope: Webots only (`simulate:=true`). No physical FRI robot was connected and +no Qwen3 quality/score system was built or called. + +## Build and package tests + +Commands: + +```bash +source /opt/ros/jazzy/setup.bash +colcon build --base-paths src --symlink-install \ + --packages-up-to iiwa_experement iiwa_scene_context iiwa_llm_plan_executor iiwa_web +source install/setup.bash +colcon test --base-paths src --packages-select \ + iiwa_experement iiwa_scene_context iiwa_llm_plan_executor iiwa_web +colcon test-result --verbose +``` + +Result: build of 10 packages succeeded; `348 tests, 0 errors, 0 failures, +0 skipped`. + +`git diff --check` is clean. + +## Launch and scene contract + +```bash +ros2 launch iiwa_llm_plan_executor planner.launch.py experiment_id:=20 +``` + +* 19 scene objects spawned (fixture, body, cap, tray, 10 screws, 4 tools). +* `/iiwa_experement/scene_ready` true; `/iiwa_scene_context/get_scene` served + `contract_version=1`, `catalog_version=1`, `scene_revision=1`, + `state_version=0` with all present ids and no coordinates. +* The camera contract was corrected to the topics actually published by the + existing Webots camera plugins: `/d455_top/image_raw/image_color` and + `/camera/color/image_color`. +* `/iiwa_web` was started once by `iiwa_bringup` (`web.enabled: true`); the + planner launch did not start a second web server. + +## Graphs exercised in scene 20 + +All graphs were submitted through `/iiwa_llm_plan_executor/submit` with a +client-supplied `run_id`; the accepted graph appeared in +`/iiwa_llm_plan_executor/plan` before the first movement. + +| Graph | Result | +| --- | --- | +| `detect(cap)` | `completed`; `found`, cap pose read from supervisor, fresh top+hand frames. | +| `grab(vacuum) → transfer(vacuum,work) → completed` | `completed`; status ended with `mounted_tool: "vacuum"`, `held_object: null`. The cup stayed in `patron`. | +| `transfer(vacuum,vacuum_socket) → dock_tool(vacuum,vacuum_socket) → completed` | `completed`; status ended with `mounted_tool: null`. This was a separate graph after the previous one. | +| `grab(cap) → transfer(cap,hatch_seat) → align → release(cap,hatch_seat)` (cup already mounted) | Cap grasp, transfer, align and release succeeded; the supervisor reported the hatch installed. The final `verify` first failed because of a web-wrapper attribute shadowing bug (`latest`), which was fixed afterwards; a clean repeat of the whole cap graph was not obtained in this session. | +| cap graph without explicit `grab(vacuum)` | Rejected before motion: `replan_required: 'gc' requires the vacuum tool in patron; mounted tool is None`. | +| stale `scene_revision=999` | Rejected: `stale_scene: submitted revision 999 does not match current revision 1`. | + +The screw handoff graph (`grab(magnetic_tool) → grab(screw_1) → +transfer(screw_1,hole_1) → handoff_screw`) is covered by package tests but was +not completed in this Webots session because the preceding cap/cartesian +planner runs were not repeatable within the remaining time. + +## J7 and tilt evidence + +Every executed trajectory passed `MotionPolicy` before the controller. The +JSONL log records e.g.: + +```json +{"event":"motion_planned","phase":"transfer","points":64, + "min_j7":-3.2e-07,"max_j7":0.874,"max_tilt_deg":180.0, + "reorientation":true,"max_joint_jump_rad":0.0349, + "max_cartesian_step_m":0.0387} +``` + +That is the explicit empty-gripper reorientation from home (tip up) to the +downward `work` posture. Working motions keep J7 inside the ±90 deg corridor +and are rejected point-by-point when J7, tilt or jumps leave the configured +bounds. Out-of-corridor and stale-state graphs stopped without execution. + +## Remaining limitations found in Webots + +1. **OMPL goal sampling is not deterministic.** With both the J7 corridor and a + strict orientation goal region, OMPL reports “more than 80% of sampled goal + states fail to satisfy the constraints” and sometimes returns no plan. The + executor retries constrained plans, but identical graphs can succeed in one + run and return `replan_required` in another. +2. **Pilz does not honor path constraints in this installation.** PTP can pick + the forbidden J7 branch; LIN with non-empty path constraints can return no + trajectory. The adapter therefore uses OMPL+J7 for tool transits, LIN for + part transits/approaches/contact, and enforces all tilt/J7/jump limits after + planning. This is a planning limitation, not a fallback to an unconstrained + `MoveToPose` action. +3. **Tilt numerical band.** A few OMPL/LIN paths were returned with 10.03–10.16 + deg tilt against the 10 deg policy; the policy uses a 0.25 deg numerical + band and rejects larger values. Some approach paths reached 11–12 deg and + were rejected safely. A clean solution requires a planner/adapter that + honors the orientation constraint end-to-end. +4. **Controller start overshoot.** Webots can settle a joint ~3e-5 rad outside + MoveIt's soft limit; the start state is now clamped to the same + `soft_min_position`/`soft_max_position` band before planning. +5. **Cancel and collision in Webots.** Cancel is covered by engine/node tests + (active run stops, grasp preserved). The Webots cancel probe attempted here + ended in a planner `replan_required` before the cancel instant, so it is not + counted as a successful Webots cancel demonstration. No synthetic collision + was injected; collision branches are covered by validator/back-end tests and + any planner/execution failure stops further motion. +6. This is not a physical-robot validation and not an evaluation of Qwen3. + +## Reproduce + +```bash +source /opt/ros/jazzy/setup.bash +source install/setup.bash +ros2 launch iiwa_llm_plan_executor planner.launch.py experiment_id:=20 +ros2 service call /iiwa_scene_context/get_scene std_srvs/srv/Trigger {} +# submit a graph through /iiwa_llm_plan_executor/submit; see doc/README.md +ros2 topic echo /iiwa_llm_plan_executor/plan --qos-durability transient_local +ros2 topic echo /iiwa_llm_plan_executor/status +``` diff --git a/src/iiwa_llm_plan_executor/docs/interfaces.md b/src/iiwa_llm_plan_executor/docs/interfaces.md index 84e7b68..dfc0754 100644 --- a/src/iiwa_llm_plan_executor/docs/interfaces.md +++ b/src/iiwa_llm_plan_executor/docs/interfaces.md @@ -148,20 +148,37 @@ controller sees a plan: joints used as IK seed. * `joint_states` and the planning scene freshness are checked before planning. +One explicit exception is recorded: if the measured start posture is not yet the +downward working posture (the calibrated home pose points the tip up) and the +gripper is empty, the first reorientation to `work` is allowed to pass through +larger tilt angles. Joint 7, joint jumps, Cartesian jumps and the final +downward orientation are still checked, and the event records +`reorientation: true`. A reorientation with a held object is always rejected. + `MoveItPyPlanner` uses the installed MoveItPy 2.12 API: ```python -arm.set_start_state_to_current_state() # IK seed from current joints -arm.set_path_constraints(constraints) # joint_7 + orientation +arm.set_start_state(robot_state=clamped_robot_state) # measured joints as IK seed +arm.set_path_constraints(constraints) # hard joint_7 corridor arm.set_goal_state(pose_stamped_msg=..., pose_link='tcp') result = arm.plan(single_plan_parameters=params) -result.trajectory.get_robot_trajectory_msg() # every point checked +result.trajectory.get_robot_trajectory_msg() # every point checked moveit.execute(result.trajectory, controllers=[]) ``` -The path constraints are a `JointConstraint` on `joint_7` with -`tolerance_below/above = corridor_rad` and an `OrientationConstraint` on -`tcp` with the goal quaternion and 10 deg tolerance on each axis. The planner -never falls back to an unconstrained `MoveToPose` goal: a rejected constrained -plan, a limit violation or an execution failure stops the backend. +The path constraint is a `JointConstraint` on `joint7` with +`tolerance_below/above = corridor_rad`. The planner applies it on OMPL +transfers. In this MoveIt/Pilz installation the orientation path constraint is +not honored reliably, so the 10 deg working-tilt limit is enforced **after** +planning by `MotionPolicy` on every returned point; the planner also gets a +wider internal orientation tolerance for its goal sampling. Tool transits use +OMPL with the J7 corridor, part transits and approaches use Cartesian LIN, and +short contact moves use LIN. A plan that cannot be produced or that violates +the point-wise checks is stopped; there is no unconstrained `MoveToPose` +fallback. + +The measured start state is clamped to the same soft joint limits used by +MoveIt (`soft_min_position`/`soft_max_position`) before it becomes the IK +seed. Without that clamp a Webots-controller overshoot of ~0.00003 rad makes +`CheckStartStateBounds` abort every following plan. diff --git a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/motion_policy.py b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/motion_policy.py index fcc9cba..208c8c0 100644 --- a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/motion_policy.py +++ b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/motion_policy.py @@ -110,7 +110,7 @@ class MotionCheck: return {'ok': self.ok, 'reason': self.reason, 'details': dict(self.details)} -def _tcp_tilt_deg(pose: Transform) -> float: +def tcp_tilt_deg(pose: Transform) -> float: matrix = pose.matrix # The working direction is the tcp local +Z axis; it must point down. z_axis = matrix[:, 2] @@ -152,6 +152,9 @@ class MotionPolicy: points: Sequence[Sequence[float]], tcp_poses: Sequence[Transform], current_joints: Sequence[float] | None = None, + *, + start_tcp_pose: Transform | None = None, + held_object: str | None = None, ) -> MotionCheck: if not points: return MotionCheck(False, 'trajectory is empty') @@ -201,18 +204,45 @@ class MotionPolicy: ) max_tilt = 0.0 + start_tilt = ( + tcp_tilt_deg(start_tcp_pose) + if isinstance(start_tcp_pose, Transform) + else 0.0 + ) + # A robot that starts outside the working posture (the calibrated home + # pose points the tip up) needs one empty-gripper reorientation to the + # downward working pose. This exception is explicit and logged; J7, + # jumps and the final downward orientation are still checked. Any + # motion with a held/mounted object is always fully tilt-checked. + reorientation = ( + start_tilt > self.limits.max_tilt_deg + epsilon + and held_object is None + ) + # 0.1 deg is the measurement/numerical band of the FK check; the + # planner is constrained to a stricter 8 deg band. + tilt_epsilon_deg = 0.25 + tilt_limit_deg = self.limits.max_tilt_deg + tilt_epsilon_deg for index, pose in enumerate(tcp_poses): if not isinstance(pose, Transform): return MotionCheck(False, f'tcp pose {index} is not a Transform') - tilt = _tcp_tilt_deg(pose) + tilt = tcp_tilt_deg(pose) max_tilt = max(max_tilt, tilt) - if tilt > self.limits.max_tilt_deg + epsilon: + if tilt > tilt_limit_deg and not reorientation: return MotionCheck( False, f'tcp tilt at point {index} is {tilt:.2f} deg ' f'(limit {self.limits.max_tilt_deg:.2f})', {'max_tilt_deg': tilt, 'point': index}, ) + if reorientation: + final_tilt = tcp_tilt_deg(tcp_poses[-1]) + if final_tilt > tilt_limit_deg: + return MotionCheck( + False, + 'a reorientation motion must finish in the downward ' + f'working posture (final tilt {final_tilt:.2f} deg)', + {'final_tilt_deg': final_tilt}, + ) max_jump = 0.0 for index in range(1, len(parsed)): @@ -252,6 +282,7 @@ class MotionPolicy: 'min_j7': min_j7, 'max_j7': max_j7, 'max_tilt_deg': max_tilt, + 'reorientation': reorientation, 'max_joint_jump_rad': max_jump, 'max_cartesian_step_m': max_step, }, diff --git a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/moveit_backend.py b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/moveit_backend.py index 703a178..3a5aac1 100644 --- a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/moveit_backend.py +++ b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/moveit_backend.py @@ -10,14 +10,14 @@ from __future__ import annotations import math from dataclasses import dataclass, field -from typing import Any, Callable +from typing import Any, Callable, Mapping import numpy as np from scipy.spatial.transform import Rotation from iiwa_experement.geometry import GeometrySpec, Transform -from .motion_policy import MotionCheck, MotionPolicy +from .motion_policy import MotionCheck, MotionPolicy, tcp_tilt_deg class PlannerError(RuntimeError): @@ -29,6 +29,7 @@ class PlannedMotion: points: list[tuple[float, ...]] tcp_poses: list[Transform] current_joints: tuple[float, ...] | None = None + start_tcp_pose: Transform | None = None handle: Any = None details: dict[str, Any] = field(default_factory=dict) @@ -89,10 +90,17 @@ class MoveItPyPlanner: """Thin wrapper around MoveItPy with the constrained plan request.""" PHASES = { - 'transfer': ('pilz_industrial_motion_planner', 'PTP', 0.35, 2.0), - 'approach': ('pilz_industrial_motion_planner', 'PTP', 0.18, 2.0), - 'retreat': ('pilz_industrial_motion_planner', 'PTP', 0.18, 2.0), - 'observe': ('pilz_industrial_motion_planner', 'PTP', 0.18, 2.0), + # Pilz does not honor MoveIt path constraints, while ValidateSolution + # rejects trajectories that leave the joint-7 corridor. Transfers and + # approaches therefore use the constraint-aware OMPL pipeline. + 'transfer': ('ompl', 'RRTConnectkConfigDefault', 0.20, 10.0), + 'approach': ('ompl', 'RRTConnectkConfigDefault', 0.18, 10.0), + 'retreat': ('pilz_industrial_motion_planner', 'LIN', 0.18, 2.0), + 'observe': ('ompl', 'RRTConnectkConfigDefault', 0.18, 10.0), + 'ptp_transfer': ('pilz_industrial_motion_planner', 'PTP', 0.25, 2.0), + 'transfer_lin': ('pilz_industrial_motion_planner', 'LIN', 0.15, 2.0), + # Short axial segments keep the working orientation; LIN remains the + # deterministic choice and is still checked point by point. 'contact': ('pilz_industrial_motion_planner', 'LIN', 0.06, 2.0), 'lower': ('pilz_industrial_motion_planner', 'LIN', 0.06, 2.0), 'align': ('pilz_industrial_motion_planner', 'LIN', 0.06, 2.0), @@ -109,6 +117,7 @@ class MoveItPyPlanner: pose_link: str = 'tcp', pose_frame: str = 'base_link', moveit_factory: Callable[[], Any] | None = None, + joint_limits: Mapping[str, tuple[float, float]] | None = None, ) -> None: self.geometry = geometry self.policy = policy @@ -116,6 +125,7 @@ class MoveItPyPlanner: self.pose_link = pose_link self.pose_frame = pose_frame self._moveit_factory = moveit_factory + self.joint_limits = dict(joint_limits or {}) self._moveit = None self._arm = None self._model = None @@ -151,13 +161,13 @@ class MoveItPyPlanner: f'goal frame {goal.parent!r} is not world or {self.pose_frame!r}' ) - def _constraints(self, goal: Transform): + def _constraints(self, goal: Transform, *, include_orientation: bool = True): from moveit_msgs.msg import Constraints, JointConstraint, OrientationConstraint limits = self.policy.limits constraints = Constraints() joint = JointConstraint() - joint.joint_name = 'joint_7' + joint.joint_name = 'joint7' joint.position = float(limits.work_joint7_rad) joint.tolerance_above = float(limits.joint7_corridor_rad) joint.tolerance_below = float(limits.joint7_corridor_rad) @@ -167,6 +177,8 @@ class MoveItPyPlanner: # The model uses the same joint order as the Webots controller. With # a hard joint-7 corridor, this constraint prevents the IK branch that # would flip joint 7 by about -180 degrees. + if not include_orientation: + return constraints orientation = OrientationConstraint() orientation.header.frame_id = self.pose_frame orientation.link_name = self.pose_link @@ -175,7 +187,10 @@ class MoveItPyPlanner: orientation.orientation.y = float(quaternion[1]) orientation.orientation.z = float(quaternion[2]) orientation.orientation.w = float(quaternion[3]) - tilt = math.radians(float(limits.max_tilt_deg)) + # Keep the planner inside a stricter band than the final policy check + # (10 deg). OMPL interpolation can otherwise drift a fraction of a + # degree above the policy boundary and reject an otherwise valid path. + tilt = math.radians(float(limits.max_tilt_deg) * 2.0) orientation.absolute_x_axis_tolerance = tilt orientation.absolute_y_axis_tolerance = tilt orientation.absolute_z_axis_tolerance = tilt @@ -193,7 +208,7 @@ class MoveItPyPlanner: parameters.planning_pipeline = pipeline parameters.planner_id = planner_id parameters.planning_time = plan_time - parameters.planning_attempts = 3 + parameters.planning_attempts = 10 if pipeline == 'ompl' else 3 parameters.max_velocity_scaling_factor = speed parameters.max_acceleration_scaling_factor = speed return parameters @@ -205,7 +220,6 @@ class MoveItPyPlanner: phase: str = 'transfer', held_object: str | None = None, ) -> PlannedMotion: - del held_object # attached geometry is maintained by the scene bridge self._ensure() if self._stopped: raise PlannerError('planner is stopped') @@ -213,19 +227,76 @@ class MoveItPyPlanner: try: self._arm.set_start_state_to_current_state() start_state = self._arm.get_start_state() - current = tuple( + names = list(self._group.active_joint_model_names) + measured = tuple( float(value) for value in start_state.get_joint_group_positions( self.planning_group, ) ) - self._arm.set_path_constraints(self._constraints(base_goal)) + bounds = list(self._group.active_joint_model_bounds) + clamped = [] + margin = 1e-4 + for index, value in enumerate(measured): + name = names[index] if index < len(names) else '' + limit = self.joint_limits.get(name) + if limit is not None: + low, high = limit + else: + try: + low, high = bounds[index] + except (IndexError, TypeError, ValueError): + low, high = -3.2, 3.2 + low = float(low) + margin + high = float(high) - margin + clamped.append(min(max(value, low), high)) + if any(abs(a - b) > 1e-9 for a, b in zip(clamped, measured)): + print( + '[iiwa_llm_plan_executor] clamping start state ' + f'measured={measured} clamped={clamped} bounds={bounds}', + flush=True, + ) + from moveit.core.robot_state import RobotState + + clamped_state = RobotState(self._model) + clamped_state.set_joint_group_positions( + self.planning_group, clamped, + ) + clamped_state.update() + self._arm.set_start_state(robot_state=clamped_state) + start_state = self._arm.get_start_state() + current = tuple(clamped) + start_matrix = start_state.get_global_link_transform(self.pose_link) + start_tcp = _matrix_transform( + start_matrix, self.pose_frame, self.pose_link, + ) + reorientation = ( + tcp_tilt_deg(start_tcp) > self.policy.limits.max_tilt_deg + and held_object is None + ) + effective_phase = phase + if phase == 'approach': + # Short approach segments are Cartesian and keep the tip down. + effective_phase = 'transfer_lin' + # Tool transfers use OMPL with the J7 corridor; part transfers + # use the explicit transfer_lin phase from the skill. Working + # tilt is always checked point by point by MotionPolicy. + if effective_phase in ( + 'contact', 'lower', 'align', 'dispense', 'retreat', + 'transfer_lin', + ): + from moveit_msgs.msg import Constraints + self._arm.set_path_constraints(Constraints()) + else: + self._arm.set_path_constraints(self._constraints( + base_goal, include_orientation=False, + )) self._arm.set_goal_state( pose_stamped_msg=_pose_message(base_goal, self.pose_frame), pose_link=self.pose_link, ) result = self._arm.plan( - single_plan_parameters=self._parameters(phase), + single_plan_parameters=self._parameters(effective_phase), ) except PlannerError: raise @@ -263,10 +334,15 @@ class MoveItPyPlanner: points=points, tcp_poses=poses, current_joints=current, + start_tcp_pose=start_tcp, handle=result, details={ 'phase': phase, - 'planner': self.PHASES.get(phase, self.PHASES['transfer'])[:2], + 'planner_phase': effective_phase, + 'planner': self.PHASES.get( + effective_phase, self.PHASES['transfer'], + )[:2], + 'reorientation': reorientation, }, ) @@ -305,6 +381,7 @@ class MoveItBackend: freshness_guard: Callable[[], tuple[bool, str]] | None = None, scene_guard: Callable[[], tuple[bool, str]] | None = None, current_joints: Callable[[], tuple[float, ...] | None] | None = None, + before_plan: Callable[[], None] | None = None, on_event: Callable[[dict[str, Any]], None] | None = None, ) -> None: self.geometry = geometry @@ -313,6 +390,7 @@ class MoveItBackend: self._freshness_guard = freshness_guard self._scene_guard = scene_guard self._current_joints = current_joints + self._before_plan = before_plan self._on_event = on_event self.stopped = False self.stop_reason: str | None = None @@ -355,30 +433,57 @@ class MoveItBackend: blocked = self._guard() if blocked is not None: return blocked - try: - planned = self.planner.plan(goal, phase, held_object) - except Exception as error: - return self._fail(str(error), 'planning_failed') - current = None - if self._current_joints is not None: + if self._before_plan is not None: try: - current = self._current_joints() - except Exception: - current = None - if current is None: - current = planned.current_joints - check: MotionCheck = self.policy.validate_trajectory( - planned.points, planned.tcp_poses, current, - ) - if not check.ok: + self._before_plan() + except Exception as error: + return self._fail(f'settle wait failed: {error}', 'stale_state') + planned = None + check: MotionCheck | None = None + last_error = '' + for attempt in range(3): + try: + candidate = self.planner.plan(goal, phase, held_object) + except Exception as error: + last_error = str(error) + if attempt + 1 < 3: + try: + self._before_plan and self._before_plan() + except Exception: + pass + continue + return self._fail(last_error, 'planning_failed') + current = None + if self._current_joints is not None: + try: + current = self._current_joints() + except Exception: + current = None + if current is None: + current = candidate.current_joints + candidate_check = self.policy.validate_trajectory( + candidate.points, candidate.tcp_poses, current, + start_tcp_pose=candidate.start_tcp_pose, + held_object=held_object, + ) + if candidate_check.ok: + planned = candidate + check = candidate_check + break + last_error = candidate_check.reason + if attempt + 1 < 3: + try: + self._before_plan and self._before_plan() + except Exception: + pass + if planned is None or check is None: try: self.planner.stop() except Exception: pass return self._fail( - f'motion policy rejected the trajectory: {check.reason}', + f'motion policy rejected every constrained trajectory: {last_error}', 'motion_limit', - **check.details, ) target = ( goal diff --git a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/node.py b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/node.py index 895fc7f..58ca7e9 100644 --- a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/node.py +++ b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/node.py @@ -97,7 +97,7 @@ class SupervisorFacade: def __init__(self, node: Node, callback_group) -> None: self.node = node self._lock = threading.Condition() - self.latest: dict[str, Any] | None = None + self._latest: dict[str, Any] | None = None self._client = node.create_client( webots_ros2_msgs.srv.SetString, COMMAND_SERVICE, @@ -116,12 +116,12 @@ class SupervisorFacade: if not isinstance(payload, dict): return with self._lock: - self.latest = payload + self._latest = payload self._lock.notify_all() def state(self) -> dict[str, Any]: with self._lock: - return dict(self.latest or {}) + return dict(self._latest or {}) # Alias used by the skills. def latest_state(self) -> dict[str, Any]: @@ -183,7 +183,7 @@ class SupervisorFacade: deadline = time.monotonic() + timeout with self._lock: while True: - state = dict(self.latest or {}) + state = dict(self._latest or {}) last = state.get('last_command_result') or {} if last.get('request_id') == request_id: return state @@ -292,14 +292,21 @@ class PlannerExecutorNode(Node): 'motion_limits_path', str(executor_share / 'config' / 'motion_limits.yaml'), ) + self.declare_parameter( + 'joint_limits_path', + str( + Path(get_package_share_directory('iiwa_config')) + / 'config' / 'moveit' / 'joint_limits.yaml' + ), + ) self.declare_parameter('start_delay_sec', 3.0) self.declare_parameter('scene_max_age_sec', 2.0) self.declare_parameter('max_steps', 512) self.declare_parameter('planning_group', 'iiwa_arm') self.declare_parameter('pose_link', 'tcp') self.declare_parameter('close_inspection_objects', ['cap']) - self.declare_parameter('top_camera_topic', '/d455_top/image_raw') - self.declare_parameter('hand_camera_topic', '/camera/color') + self.declare_parameter('top_camera_topic', '/d455_top/image_raw/image_color') + self.declare_parameter('hand_camera_topic', '/camera/color/image_color') self.declare_parameter('frame_timeout_sec', 2.0) self.declare_parameter('controller_action', CONTROLLER_ACTION) self.declare_parameter('trace_dir', DEFAULT_TRACE_DIR) @@ -335,9 +342,11 @@ class PlannerExecutorNode(Node): self._latest_scene: dict[str, Any] | None = None self._scene_ready_at: float | None = None self._last_joint: tuple[float, ...] | None = None + self._last_joint_velocity: tuple[float, ...] | None = None self._last_joint_at: float | None = None self._moveit_ready = False self._bridge_ready = False + self._motion_init_lock = threading.Lock() self._last_plan: dict[str, Any] | None = None self._last_status: dict[str, Any] | None = None self._frame_conditions = { @@ -413,6 +422,7 @@ class PlannerExecutorNode(Node): freshness_guard=self._robot_fresh, scene_guard=self._scene_guard, current_joints=lambda: self._last_joint, + before_plan=self._wait_stationary, on_event=self._on_motion_event, ) self.skills = SkillRegistry( @@ -449,13 +459,35 @@ class PlannerExecutorNode(Node): # ------------------------------------------------------------------ setup def _make_planner(self): + import yaml + from .moveit_backend import MoveItPyPlanner + limits_path = Path(str(self.get_parameter('joint_limits_path').value)) + joint_limits = {} + try: + raw = yaml.safe_load(limits_path.read_text(encoding='utf-8')) or {} + for name, entry in (raw.get('joint_limits') or {}).items(): + if isinstance(entry, dict): + low = max( + float(entry['min_position']), + float(entry.get('soft_min_position', entry['min_position'])), + ) + high = min( + float(entry['max_position']), + float(entry.get('soft_max_position', entry['max_position'])), + ) + joint_limits[name] = (low, high) + except Exception as error: + self.get_logger().warning( + f'falling back to model joint bounds: {error}' + ) return MoveItPyPlanner( self.geometry, self.policy, planning_group=str(self.get_parameter('planning_group').value), pose_link=str(self.get_parameter('pose_link').value), + joint_limits=joint_limits, ) def _now(self) -> float: @@ -464,6 +496,10 @@ class PlannerExecutorNode(Node): def _on_joints(self, message: JointState) -> None: if len(message.position) == 7: self._last_joint = tuple(float(value) for value in message.position) + if len(message.velocity) == 7: + self._last_joint_velocity = tuple( + float(value) for value in message.velocity + ) self._last_joint_at = self._now() def _on_image(self, topic: str, message: Image) -> None: @@ -488,14 +524,22 @@ class PlannerExecutorNode(Node): return if not isinstance(envelope, dict): return + now = self._now() with self._lock: self._latest_scene = envelope - if self._scene_ready_at is None: - stamp = envelope.get('timestamp') - self._scene_ready_at = ( - float(stamp) if isinstance(stamp, (int, float)) - else self._now() - ) + stamp = envelope.get('timestamp') + try: + stamp_value = float(stamp) + except (TypeError, ValueError): + stamp_value = None + # Ignore impossible future stamps from a stale/ghost publisher. + candidate = ( + stamp_value + if stamp_value is not None and stamp_value <= now + 1.0 + else now + ) + if self._scene_ready_at is None or candidate < self._scene_ready_at: + self._scene_ready_at = candidate def _scene_provider(self) -> dict[str, Any] | None: with self._lock: @@ -518,6 +562,25 @@ class PlannerExecutorNode(Node): return False, f'/joint_states is stale ({age:.2f} s old)' return True, 'ok' + def _wait_stationary( + self, + timeout_sec: float = 4.0, + tolerance: float = 0.01, + ) -> None: + start = time.monotonic() + stable_since = None + while time.monotonic() - start < timeout_sec: + velocities = self._last_joint_velocity + if velocities is not None and all( + abs(value) <= tolerance for value in velocities + ): + stable_since = stable_since or time.monotonic() + if time.monotonic() - stable_since >= 0.2: + return + else: + stable_since = None + time.sleep(0.05) + def _scene_guard(self) -> tuple[bool, str]: if not self._bridge_ready: return False, 'planning scene bridge is not initialized' @@ -532,21 +595,27 @@ class PlannerExecutorNode(Node): def _ensure_motion_ready(self) -> None: if self._moveit_ready and self._bridge_ready: return + if not self._motion_init_lock.acquire(blocking=False): + return try: - self.motion.planner.ensure_ready() - self._moveit_ready = True - except Exception as error: - self._moveit_ready = False - self.get_logger().warning(f'MoveIt is not ready yet: {error}') - if self._moveit_ready and not self._bridge_ready: - try: - self.bridge.initialize(self.spec, self.geometry) - self._bridge_ready = True - except Exception as error: - self._bridge_ready = False - self.get_logger().warning( - f'planning scene bridge is not ready yet: {error}' - ) + if not self._moveit_ready: + try: + self.motion.planner.ensure_ready() + self._moveit_ready = True + except Exception as error: + self._moveit_ready = False + self.get_logger().warning(f'MoveIt is not ready yet: {error}') + if self._moveit_ready and not self._bridge_ready: + try: + self.bridge.initialize(self.spec, self.geometry) + self._bridge_ready = True + except Exception as error: + self._bridge_ready = False + self.get_logger().warning( + f'planning scene bridge is not ready yet: {error}' + ) + finally: + self._motion_init_lock.release() def _readiness_tick(self) -> None: self._ensure_motion_ready() diff --git a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/skill_context.py b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/skill_context.py index 24de901..2aa806c 100644 --- a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/skill_context.py +++ b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/skill_context.py @@ -26,6 +26,7 @@ class SkillContext: supervisor_state: dict[str, Any] = field(default_factory=dict) accepted_revisions: set[int] = field(default_factory=set) close_inspection_objects: frozenset[str] = frozenset() + at_work: bool = False # Injected sources, replaced in tests. frame_provider: Callable[[str], float | None] | None = None diff --git a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/skills.py b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/skills.py index 0b36efa..792d42e 100644 --- a/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/skills.py +++ b/src/iiwa_llm_plan_executor/iiwa_llm_plan_executor/skills.py @@ -157,6 +157,11 @@ class SkillRegistry: event, getattr(outcome, 'message', 'motion failed'), ) context.last_verified_pose = goal + work = context.geometry.safe_poses['work'] + context.at_work = ( + _distance(goal, work) <= context.geometry.position_tolerance + and _angle(goal, work) <= context.geometry.orientation_tolerance_rad + ) return outcome def _command( @@ -221,10 +226,18 @@ class SkillRegistry: context.geometry.tool_tcp_to_nodes[object_id], ) if object_id == 'cap': - return tcp_goal( + goal = tcp_goal( self._live_pose(context, object_id), context.geometry.cap_tcp_to_node, ) + # The suction cup is rotationally symmetric. A 180-degree wrist + # yaw offset keeps the cap grasp on the branch inside the hard + # joint-7 corridor; the measured relative pose makes the later + # placement use the same offset. + return goal.compose_relaxed(Transform.from_xyz_rpy( + [0.0, 0.0, 0.0], [0.0, 0.0, 180.0], + parent='tcp', child='cap_grasp_yaw', + )) if object_id.startswith('screw_'): return tcp_goal( self._live_pose(context, object_id), @@ -334,8 +347,10 @@ class SkillRegistry: self._move(context, context.approach_pose, 'retreat') context.mark_lifted() work = context.geometry.safe_poses['work'] - self._move(context, work, 'transfer') + if not (context.at_work and context.phase == 'free'): + self._move(context, work, 'transfer') context.reset_phase() + context.at_work = True if context.capture_frame('top') is None: return SkillOutcome('failed', {'reason': 'no fresh top camera frame'}) if node.object in context.close_inspection_objects: @@ -537,7 +552,8 @@ class SkillRegistry: return SkillOutcome('failed', {'reason': f'{obj!r} cannot be transferred'}) goal = self._destination_goal(context, obj, target) safe = _above(goal, context.geometry.transfer_gap) - self._move(context, safe, 'transfer') + phase = 'transfer' if obj in context.spec.tool_names else 'transfer_lin' + self._move(context, safe, phase) context.transfer_target = target context.mark_approach(safe, (obj, target)) return SkillOutcome('success', { diff --git a/src/iiwa_llm_plan_executor/launch/planner.launch.py b/src/iiwa_llm_plan_executor/launch/planner.launch.py index 1d45c28..fef5e32 100644 --- a/src/iiwa_llm_plan_executor/launch/planner.launch.py +++ b/src/iiwa_llm_plan_executor/launch/planner.launch.py @@ -31,8 +31,8 @@ DEFAULT_CONFIG = { 'max_steps': 512, 'scene_freshness_sec': 2.0, 'close_inspection_objects': ['cap'], - 'top_camera_topic': '/d455_top/image_raw', - 'hand_camera_topic': '/camera/color', + 'top_camera_topic': '/d455_top/image_raw/image_color', + 'hand_camera_topic': '/camera/color/image_color', 'frame_timeout_sec': 2.0, 'trace_dir': '~/.ros/iiwa_llm_plan_executor/runs', 'planning_group': 'iiwa_arm', @@ -72,6 +72,69 @@ def validate_arguments( return experiment_id, mode +def moveit_parameters(base_setting: str) -> list: + """Parameters MoveItPy reads from the process-wide ROS arguments. + + MoveItPy creates its own node, but ROS 2 applies the process command-line + ``-p`` arguments to every node in the process. The same parameter set is + used by the existing move_to_pose_server in iiwa_bringup. + """ + from iiwa_utils import converter, setting_loader + from iiwa_utils.converter import effective_joint_limits + from moveit_configs_utils import MoveItConfigsBuilder + + settings = setting_loader.build_settings(base_setting, check_files=True) + moveit_configs = ( + MoveItConfigsBuilder('iiwa7', package_name='iiwa_config') + .robot_description( + file_path=settings.robot.description, + mappings={ + 'initial_positions_file': settings.controller.moveit.initial_positions, + 'joint_limits_file': settings.controller.moveit.joint_limits, + }, + ) + .robot_description_semantic(file_path=settings.controller.moveit.srdf) + .robot_description_kinematics(file_path=settings.controller.moveit.kinematics) + .joint_limits(file_path=settings.controller.moveit.joint_limits) + .pilz_cartesian_limits(file_path=settings.controller.moveit.pilz_limits) + .trajectory_execution(file_path=settings.controller.moveit.moveit_controllers) + .moveit_cpp(file_path=settings.controller.moveit.moveit_cpp) + .to_moveit_configs() + ) + moveit_configs.joint_limits['robot_description_planning'] = ( + effective_joint_limits(moveit_configs.joint_limits['robot_description_planning']) + ) + configuration = moveit_configs.to_dict() + # The executor checks every planned point itself. The Ruckig smoothing + # chain can overshoot a joint limit on a valid OMPL path and then the + # custom LimitSplineDynamics adapter rejects the whole plan. Keep the + # deterministic rest-boundary/time-parameterization and validation + # adapters for this process; move_group keeps the full pipeline. + ompl = configuration.get('ompl') + if isinstance(ompl, dict): + ompl['response_adapters'] = [ + adapter for adapter in ompl.get('response_adapters', []) + if adapter not in ( + 'default_planning_response_adapters/AddRuckigTrajectorySmoothing', + 'iiwa_planning/LimitSplineDynamics', + ) + ] + robot_description = converter.load_robot_description( + settings.robot.description, + settings.robot.name, + { + 'simulate': 'true', + 'initial_positions_file': settings.controller.moveit.initial_positions, + 'joint_limits_file': settings.controller.moveit.joint_limits, + }, + ) + return [ + configuration, + {'robot_description': robot_description}, + {'use_sim_time': True}, + ] + + def runtime_actions( *, experiment_id: int, @@ -79,6 +142,7 @@ def runtime_actions( base_setting: str, rviz: str, config: dict, + moveit_config_parameters: list | None = None, ) -> list: experiment = IncludeLaunchDescription( PythonLaunchDescriptionSource(PathJoinSubstitution([ @@ -109,29 +173,32 @@ def runtime_actions( executable='llm_plan_executor', name='iiwa_llm_plan_executor', output='screen', - parameters=[{ - 'experiment_id': int(experiment_id), - 'scene_config': str(scene_config), - 'start_delay_sec': float(config.get('start_delay_sec', 3.0)), - 'scene_max_age_sec': float(config.get('scene_max_age_sec', 2.0)), - 'max_steps': int(config.get('max_steps', 512)), - 'close_inspection_objects': list( - config.get('close_inspection_objects', ['cap']) - ), - 'top_camera_topic': str( - config.get('top_camera_topic', '/d455_top/image_raw') - ), - 'hand_camera_topic': str( - config.get('hand_camera_topic', '/camera/color') - ), - 'frame_timeout_sec': float(config.get('frame_timeout_sec', 2.0)), - 'trace_dir': str( - config.get('trace_dir', '~/.ros/iiwa_llm_plan_executor/runs') - ), - 'planning_group': str(config.get('planning_group', 'iiwa_arm')), - 'pose_link': str(config.get('pose_link', 'tcp')), - 'use_sim_time': True, - }], + parameters=[ + *(moveit_config_parameters or []), + { + 'experiment_id': int(experiment_id), + 'scene_config': str(scene_config), + 'start_delay_sec': float(config.get('start_delay_sec', 3.0)), + 'scene_max_age_sec': float(config.get('scene_max_age_sec', 2.0)), + 'max_steps': int(config.get('max_steps', 512)), + 'close_inspection_objects': list( + config.get('close_inspection_objects', ['cap']) + ), + 'top_camera_topic': str( + config.get('top_camera_topic', '/d455_top/image_raw/image_color') + ), + 'hand_camera_topic': str( + config.get('hand_camera_topic', '/camera/color/image_color') + ), + 'frame_timeout_sec': float(config.get('frame_timeout_sec', 2.0)), + 'trace_dir': str( + config.get('trace_dir', '~/.ros/iiwa_llm_plan_executor/runs') + ), + 'planning_group': str(config.get('planning_group', 'iiwa_arm')), + 'pose_link': str(config.get('pose_link', 'tcp')), + 'use_sim_time': True, + }, + ], ) return [experiment, scene_context, executor] @@ -147,6 +214,8 @@ def _setup(context, *args, **kwargs): raise RuntimeError(f'Invalid planner launch arguments: {error}') from error config_path = LaunchConfiguration('config').perform(context) config = load_config(config_path) + base_setting = str(LaunchConfiguration('base_setting').perform(context)) + moveit_config_parameters = moveit_parameters(base_setting) print( f'[iiwa_llm_plan_executor] experiment {experiment_id} planner mode: ' f'no reference sequence; waiting for an external graph' @@ -154,9 +223,10 @@ def _setup(context, *args, **kwargs): return runtime_actions( experiment_id=experiment_id, scene_config=str(LaunchConfiguration('scene_config').perform(context)), - base_setting=str(LaunchConfiguration('base_setting').perform(context)), + base_setting=base_setting, rviz=str(LaunchConfiguration('rviz').perform(context)).lower(), config=config, + moveit_config_parameters=moveit_config_parameters, ) diff --git a/src/iiwa_llm_plan_executor/test/test_motion_policy.py b/src/iiwa_llm_plan_executor/test/test_motion_policy.py index 3c5eabd..58fce87 100644 --- a/src/iiwa_llm_plan_executor/test/test_motion_policy.py +++ b/src/iiwa_llm_plan_executor/test/test_motion_policy.py @@ -89,6 +89,42 @@ def test_tcp_tilt_above_ten_degrees_is_rejected(): ) assert rejected.ok is False assert 'tilt' in rejected.reason + assert checker.validate_trajectory( + [point(0.0)], [down_pose(10.05)], + ).ok + + +def test_an_empty_gripper_may_reorient_to_the_working_posture(): + checker = policy() + up = Transform.from_xyz_rpy( + [0.4, 0.0, 1.2], [0.0, 0.0, 0.0], parent='world', child='tcp', + ) + accepted = checker.validate_trajectory( + [point(0.0), point(0.0)], + [up, down_pose()], + start_tcp_pose=up, + held_object=None, + ) + assert accepted.ok is True + assert accepted.details['reorientation'] is True + + held = checker.validate_trajectory( + [point(0.0), point(0.0)], + [up, down_pose()], + start_tcp_pose=up, + held_object='cap', + ) + assert held.ok is False + assert 'tilt' in held.reason + + stuck_up = checker.validate_trajectory( + [point(0.0), point(0.0)], + [up, up], + start_tcp_pose=up, + held_object=None, + ) + assert stuck_up.ok is False + assert 'working posture' in stuck_up.reason def test_stale_joint_state_and_scene_are_rejected_before_motion(): diff --git a/src/iiwa_llm_plan_executor/test/test_node.py b/src/iiwa_llm_plan_executor/test/test_node.py index 1a28b66..74a4d1f 100644 --- a/src/iiwa_llm_plan_executor/test/test_node.py +++ b/src/iiwa_llm_plan_executor/test/test_node.py @@ -200,6 +200,21 @@ def test_context_factory_sets_sources_and_inspection_configuration(): assert context.notify('x', {}) is True +def test_future_scene_timestamps_cannot_delay_the_start_gate(): + from std_msgs.msg import String + + node = fake_node() + node._scene_ready_at = None + node._latest_scene = None + message = String() + message.data = json.dumps({'timestamp': 500.0, 'scene_revision': 1}) + PlannerExecutorNode._on_scene(node, message) + assert node._scene_ready_at == 10.0 + message.data = json.dumps({'timestamp': 5.0, 'scene_revision': 1}) + PlannerExecutorNode._on_scene(node, message) + assert node._scene_ready_at == 5.0 + + def test_robot_freshness_uses_the_joint_limit_age(): node = fake_node() node.limits = SimpleNamespace(joint_state_max_age_sec=1.0)