NVIDIA / NVIDIA/IsaacTeleop

Examples have no systematic test coverage; GPU example tests must be hand-registered

Open
#880 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
385
Forks
88
Avg merge
3d 23h
Merged PRs (30d)
82

Description

Problem

There is no systematic way to test an example, and no convention for where an example test belongs. Examples are separate uv projects, so nothing in a normal build imports or runs them — an example can drift away from the library beside it and only fail when a user runs it. That is not hypothetical: mcap_record_replay and deviceio_live_view shipped pinned to isaacteleop==1.3.9rc1 and every script died on ImportError: cannot import name 'BodyJointIndex' before it started.

Survey of examples/ (15 examples)

State Examples
Tests that run teleop_ros2 (C++ integration tests + topic verifier in the ROS2 CI job — the only real end-to-end coverage), camera_viz (3 pytest files via ctest, but behind if(BUILD_VIZ)), oxr (3 python + 2 C++ via run_tests_with_cloudxr.sh), native_openxr (xdev_list, same script)
No tests at all haptic_feedback, retargeting, teleop, teleop_session_manager, schemaio, cloudxr_mujoco_teleop, lerobot, noitom, oglo_tactile
Not built by CMake at all teleop, cloudxr_mujoco_teleop, lerobot, noitom, oglo_tactile — nothing compiles or imports them in CI

Orphaned tests. examples/oxr/python/ holds nine test_*.py; only three are listed in CXR_PYTHON_GPU_TESTS. These six are copied into the test container by Dockerfile.test and then never invoked:

  • test_controller_tracker.py
  • test_full_body_tracker.py
  • test_hand_inactive_on_plugin_stop.py
  • test_oak_camera.py
  • test_session_sharing.py
  • test_synthetic_hands.py

Why it keeps happening

  1. GPU/runtime tests are hand-registered in a bash array in scripts/run_tests_with_cloudxr.sh. Nothing discovers them, so adding a file next to nine others silently does nothing — which is how six accumulated.
  2. The container only sees one example. deps/cloudxr/Dockerfile.test hardcodes COPY examples/oxr/python/ /app/tests/, so no other example's tests can reach the CloudXR runtime without editing the Dockerfile.
  3. No home for non-GPU example tests. camera_viz puts them in tests/ with its own pyproject; oxr puts them beside the scripts. Neither is documented as the convention.

What would fix it

  • Discover GPU example tests instead of listing them, and generalise the COPY so any example can contribute tests to the CloudXR container.
  • Pick one convention for headless example tests (ctest entry + uv run --no-project, isaacteleop from the build tree on PYTHONPATH) and document it.
  • Decide the six orphans' fate: wire them up or delete them.
  • Give the five unbuilt examples at least an import/lint check so they cannot silently rot.

Starting point for the headless half: PR #879 discussion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/run_tests_with_cloudxr.sh, deps/cloudxr/Dockerfile.test, the CMake example test entries, and the PR #879 discussion. Trace how CXR_PYTHON_GPU_TESTS and the hardcoded examples/oxr/python/ copy work, then compare camera_viz and oxr conventions. Done means example tests are discoverable or explicitly handled, unbuilt examples receive checks, and the convention is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, cmake, docker, dockerfile, python
Domain
build-system, devops, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.