How to verify a robot demo: autonomy and safety checklist

Evaluate a humanoid or mobile-robot demo by recording its control mode, operating conditions, human interventions, repeated task results, recovery behaviour and safety limits.

AIWorldNext editorial approach · Robot demonstration evaluation guide · Checks and limitations are described below.

Treat the demonstration as one test, not universal proof

A successful video or live demonstration shows that a robot completed a particular task under particular conditions. It does not by itself establish that the robot will repeat the task, work in a different site or operate without remote help. Write down the exact task, environment, software version, payload, duration and success criteria before watching. NIST's response-robot programme uses defined test apparatus, procedures and metrics so performance can be repeated and compared instead of inferred from a highlight reel.

Define the operating conditions

Record the floor material, slopes, steps, door widths, lighting, weather, temperature, noise, dust, network connection, GNSS availability, crowd density, object shape and available clearance. Include the speed, payload and battery state. These conditions form the boundary within which the result has meaning. If the intended site differs, ask for a new test there or reproduce the relevant conditions. A vendor should not generalise a result from a smooth, mapped stage to an unknown warehouse, hospital or outdoor site without supporting evidence.

Label the control mode precisely

Ask who chose each action and when. Useful labels include direct manual control, remote teleoperation, waypoint execution, a fixed pre-programmed sequence, supervised autonomy and autonomous task execution. Record whether a human approved plans, corrected perception, selected a grasp, repositioned an object or took control. Avoid treating the word autonomous as a binary product label: autonomy can differ across navigation, manipulation, recovery and safety functions in the same task.

Count every intervention and reset

Keep a simple event log from preparation through shutdown. Count remote inputs, emergency stops, safety stops, retries, prompt or parameter changes, map edits, manual charging, object repositioning, robot resets and technician assistance. Separate planned supervision from unplanned intervention. Report interventions per run or per operating hour and describe their cause. A run that succeeds after three invisible resets is different evidence from one that completes without assistance.

Predefine success, failure and safe stop

Turn the task into observable criteria before the robot starts. For a delivery task, specify the start point, destination, payload, maximum time, permitted route deviation, acceptable placement error and conditions that require a safe stop. Define a partial result, a recoverable failure and a failed run. This prevents an impressive movement from replacing the actual business outcome and stops the evaluator from changing the definition of success after seeing the result.

Repeat the same task and report the distribution

Request enough repetitions to reveal variation and keep all valid runs, including failures. Report completed runs over attempted runs, median and slowest completion time, interventions, safe stops and unsuccessful recoveries. Do not publish only the fastest take. NIST describes repeated testing as a way to establish confidence in a system's capabilities, and its test approach controls variables before introducing them incrementally. The appropriate number of runs depends on the consequence of failure and the decision being made; a short sales demo cannot prove long-term reliability.

Change one important condition at a time

After a baseline is repeatable, vary the conditions that matter at the intended site: lighting, surface, obstacle position, object pose, payload, network quality or pedestrian movement. Change one factor first so its effect is visible, then test realistic combinations. Record the exact setup and preserve failed runs. This mirrors the NIST principle of controlling variables and adding complexity progressively, rather than moving directly from a staged success to an unstructured environment.

Separate subsystem evidence from the end-to-end result

A task can fail because of perception, localisation, planning, mobility, manipulation, communication, power, the end effector or a safety system. Test critical functions separately and then together. NIST's performance framework describes task-driven measures for areas such as perception, mobility, dexterity and safety, while its response-robot standards cover mobility, manipulation, sensing, energy, communications, human interaction, logistics and safety. A strong grasp test does not prove reliable navigation, and a navigation result does not prove the complete application.

Test recovery and degraded conditions

Deliberately create foreseeable problems: move an object, partially block a path, interrupt a network connection, introduce a localisation error or start with a low but permitted battery state. Observe whether the robot detects the problem, stops safely, asks for help, retries within a limit and resumes without losing the task state. Record the time and human effort required. Recovery evidence is often more useful for operations planning than a flawless nominal run because real deployments include interruptions.

Evaluate safety at robot and application level

A robot component and an integrated application have different safety responsibilities. ISO 10218-1:2025 addresses industrial robots at the robot level, while ISO 10218-2:2025 addresses industrial robot applications and cells. The full system includes the robot, end effector, controls, power, sensors, software, surrounding equipment and people. Ask for the risk assessment, safeguarding concept, emergency-stop behaviour, operating modes, training and maintenance plan appropriate to the actual application. A demo is not a safety certification, and the applicable standards vary by robot type, industry and country.

Use current standards for the robot category

Confirm which standards and legal requirements apply to the exact deployment. A3 lists ANSI/A3 R15.06-2025 for industrial robots and robot systems and ANSI/RIA R15.08-2-2023 for industrial mobile robot systems and applications. ISO's robotics catalogue also includes service-robot performance methods, including ISO 18646-5:2026 for locomotion performance of legged robots. Standards can be revised, and access to a standard does not prove conformity. Ask a competent integrator or safety professional to identify the current requirements rather than copying a checklist from another application.

Request an evidence pack

Ask for the test protocol, raw run log, full-length video, system and software versions, configuration, map or calibration assumptions, payload details, operator actions, failure notes and maintenance state. Record who conducted the test and whether the vendor, buyer or an independent laboratory witnessed it. Documentation should make the result reproducible without exposing security-sensitive material. A polished edit can support communication, but the decision record should rely on complete runs and traceable measurements.

Run a buyer-witnessed acceptance test

Convert the intended workflow into a site acceptance protocol with named pass criteria, number of attempts, allowed assistance, safety prerequisites and evidence to retain. Use representative staff, objects, routes and operating hours. Agree what happens after a failure: correction and retest, revised scope, service credit or rejection. Keep quotation, pilot and acceptance decisions separate. The test should establish whether this configured system meets the agreed use case, not whether the robot can perform an unrelated showcase routine.

Fictional example: a mobile manipulator moving cartons

A fictional buyer needs a mobile manipulator to move sealed cartons between two marked stations. The baseline protocol fixes carton mass, pickup pose, route, lighting and a ten-minute limit. Ten runs are attempted; the log records completion time, placement error, interventions and safe stops. The evaluator then changes carton pose, blocks part of the route and reduces network quality within the proposed operating limits. A remote grasp correction is labelled as an intervention rather than an autonomous success. The buyer proceeds only if the repeated result and recovery behaviour meet the written threshold. This example is illustrative and does not claim that AIWorldNext tested a robot.

Use a go, revise or stop decision

Proceed to a limited pilot when the control mode is disclosed, operating boundary is relevant, repeated results meet the predefined threshold, failures are understood, recovery is workable and the application-level safety review has an owner. Revise the task or integration when evidence identifies a fixable gap. Stop when human assistance is hidden, only selected runs are available, the intended conditions cannot be tested, safety responsibilities are unclear or the vendor will not agree to measurable acceptance criteria.

Limitations and source dates

This guide is a procurement and evaluation aid, not a robotics safety assessment, certification or legal opinion. It does not set a universal pass rate, sample size or autonomy level. Those choices depend on the task, system, jurisdiction and consequences of failure. NIST pages used here were checked on 10 November 2026; its Performance Assessment Framework and response-robot standards pages show updates on 26 March 2025. ISO 10218 Parts 1 and 2 were published in February 2025, and ISO 18646-5 was published in 2026. The OSHA technical manual provides useful system-safety context but cites some older standard editions, so consult the current standards and qualified specialists before deployment.

Sources and further reading

Documentation checked 10 November 2026; product options can change.

Robot demo evidence worksheet

Complete one row for the exact task, robot configuration and site. Attach a test record or named owner to every ready decision.

Evaluation area	Evidence to record	Ready condition
Task	Start state, end state, payload, route, time limit and required output.	The business task and observable result are unambiguous.
Operating conditions	Surface, space, lighting, weather, network, people, objects and battery state.	The test represents the intended operating boundary.
Control mode	Manual, teleoperated, scripted, supervised or autonomous actions by function.	Every human and machine decision is labelled.
Success criteria	Completion, time, accuracy, quality, route and safe-stop thresholds.	Criteria were fixed before the first run.
Run set	Attempt number, result, duration and reason for each failure.	All valid attempts are included in the summary.
Interventions	Remote input, approval, reset, map edit, repositioning and technician action.	Assistance is counted and its cause is known.
Variation	One changed condition, expected range and observed effect.	Important site variation has been tested progressively.
Subsystems	Perception, mobility, manipulation, communication, power and safety results.	Critical component evidence supports the full task result.
Recovery	Fault introduced, detection, stop, retry, escalation and resume time.	Foreseeable failures lead to a controlled response.
Safety	Risk assessment, safeguards, modes, emergency stops, training and responsible parties.	A competent person has reviewed the integrated application.
Evidence pack	Protocol, raw log, complete video, versions, configuration and maintenance state.	Another evaluator can understand and reproduce the test.
Acceptance	Site test, witness, pass threshold, allowed assistance and retest terms.	Buyer and supplier agree how acceptance will be decided.
Operations	Operator skill, support hours, maintenance, spares, charging and escalation.	The buyer can operate the system beyond the demonstration.
Decision	Go, revise or stop rationale, owner, conditions and next review date.	The decision follows recorded evidence and named limits.

All guides