Skip to main content

Chapter 5.4: Final Demonstration and Evaluation

We have arrived at the final chapter of our journey. We have designed, simulated, and deployed a complex, AI-driven robotics system. The final step is to demonstrate its capabilities, evaluate its performance, and reflect on the path forward.

This chapter provides a guide for conducting a final demonstration of the capstone project and discusses key metrics for evaluating its success, bringing together all the concepts covered in this book.

Structuring the Demonstration​

A successful demonstration tells a story. It should clearly show the problem, the solution, and the result. For our VLA system, a compelling demonstration sequence would be:

  1. The Setup:

    • Show the simulation environment (or physical room) with the robot at its starting position.
    • Show the objects placed in the environment, for instance, a red can and a blue box on a table.
    • Start all the necessary software components by running the main launch file: ros2 launch vla_system main_launch.py.
    • Bring up RViz, configured to show the robot's camera feed, the map, the global and local costmaps, and the planned path. This provides a "look inside the robot's brain" for the audience.

  2. The Command:

    • Clearly state the voice command into the microphone. For example: "Robot, please go to the table and find the red can."

  3. The "Thinking" Phase:

    • Show the terminal output of the Voice Commander node as it transcribes the speech.
    • Show the output of the LLM Planner node as it receives the text and generates the JSON action plan. Explain that this is the robot "thinking" about how to achieve the goal.

  4. The Execution Phase:

    • Point to the RViz window. The audience should see Nav2 generate a global plan (a blue line) to the table.
    • Watch the robot (in simulation or reality) as it follows the path, avoiding any obstacles. The local costmap should be visible, showing how the robot perceives its immediate surroundings.
    • Once the robot reaches the table, show the output of the Action Executor as it calls the vision service.
    • The Vision Service node can be configured to display the camera image with the final bounding box it selects, showing that it has successfully identified the "red can."

  5. The Result:

    • The robot should make a final maneuver to position itself in front of the can.
    • Finally, the robot should use a text-to-speech (TTS) system (a simple extension of the respond function) to announce: "I have found the red can."

This sequence demonstrates every part of the VLA pipeline, from understanding the user's intent to perceiving the world and acting upon that understanding.

Evaluating Performance​

How do we know if our system is "good"? We need to evaluate it against measurable metrics.

Task Success Rate​

  • Definition: The percentage of times the robot successfully completes the entire task when given a valid command.
  • How to Measure: Run the demonstration scenario 10 times with slight variations in object placement. If it succeeds 8 out of 10 times, the success rate is 80%.
  • What it tells us: This is the primary "end-to-end" metric. It measures the reliability and robustness of the entire integrated system.

Component-Level Metrics​

If the end-to-end task fails, we need to dig deeper to see which component is the weak link.

  1. ASR (Whisper) Accuracy:

    • Metric: Word Error Rate (WER).
    • How to Measure: Compare Whisper's transcribed text to the ground-truth text of your spoken commands over 50-100 trials. Many online tools can calculate WER.

  2. Planner (LLM) Accuracy:

    • Metric: Plan Correctness.
    • How to Measure: For a set of 20 different commands, manually inspect the JSON plan generated by the LLM. Is the sequence of functions logical? Are the parameters correct?

  3. Vision (VLM) Accuracy:

    • Metric: Precision and Recall for object detection.
    • How to Measure: In a series of test images, measure:
      • Precision: Of all the objects the robot identified as "red can," how many were actually red cans?
      • Recall: Of all the red cans present in the images, how many did the robot successfully identify?

  4. Navigation Performance:

    • Metric: Path Efficiency and smoothness.
    • How to Measure: Compare the length of the path taken by the robot to the optimal path length. Measure the jerkiness or vibration of the robot's movements.

Conclusion and Future Work​

Congratulations! By completing this capstone, you have built a system that embodies the core principles of modern, AI-driven robotics. You have bridged the gap between high-level human language and low-level robot control, using a modular architecture that leverages state-of-the-art tools for perception, planning, and simulation.

The journey doesn't end here. This project is a foundation. Here are some exciting next steps to explore:

  • Add Manipulation: Implement the pick_up and place_on functions by integrating a motion planning library like MoveIt2. This is the most significant and challenging next step.
  • Improve Conversation: Make the LLM planner stateful. Allow the robot to ask for clarification if a command is ambiguous (e.g., "There are two red cans, which one should I get?").
  • Real-World Fine-Tuning: Collect a small dataset of images from your physical robot's camera and use it to fine-tune the vision model, further closing the sim-to-real gap.
  • Explore Other Behaviors: Use Reinforcement Learning with Isaac Gym to teach the robot new skills, like opening a door or pushing a button, and add these as new primitive functions to your LLM planner.

You now have the tools, the architecture, and the fundamental knowledge to continue exploring the exciting and rapidly evolving field of Physical AI.