How to evaluate a humanoid robot demonstration

A demonstration is a starting point

Good demonstrations make progress visible. For someone considering a robot in a home or workplace, the next step is to understand the conditions behind the result. The following is HHA’s proposed evaluation framework, not a report of trials we have completed.

Seven questions worth asking

1. What counts as finished? Define the task and the acceptable result before watching. Moving an object and completing a customer job are different outcomes.
2. Was the sequence continuous? Look for elapsed time, edits, resets and setup outside the camera frame.
3. Where was the human? Ask about remote control, supervision, safety staff and any intervention. An autonomous segment should not be assumed to represent the entire workflow.
4. How many attempts were made? Ask for successes and failures across repeated trials, not just the selected successful run.
5. What changed between attempts? Different objects, lighting and layouts help reveal how narrow or broad the capability is.
6. How does it recover? A useful system needs a sensible response when it drops an item, loses connectivity or meets an obstacle.
7. What did the complete job cost? Include preparation, operator time, charging, maintenance and any manual work needed to finish.

Ask for a simple record

For each trial, record the task, site, software version, start and finish times, outcome, interventions and reason for any stop. Agree in advance what would constitute a pass. Repeating a defined test provides a much stronger basis for a decision than comparing promotional videos.

A practical example

For a proposed cleaning trial, define the area and required finish. Include getting equipment ready, accessing the work, doing the task, putting tools away and checking the result. Keep observations about cleaning quality separate from observations about movement or object handling.

Our approach is to begin with controlled evaluation and let evidence determine the next step. This checklist does not certify a robot or replace a site-specific safety assessment. It helps structure the questions before a trial is designed.

Have a question or correction?

Email the HHA team about this article. Please avoid personal or sensitive information. Reader messages are not automatically published.