NVIDIA-NeMo / NVIDIA-NeMo/Gym

bug: test_verify in my_weather_tool fails due to incorrect input/output schema

Open
#601 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

Describe the bug

The test_verify test in resources_servers/my_weather_tool/tests/test_app.py fails with 51 Pydantic validation errors due to incorrect schema format for NeMoGymResponseCreateParamsNonStreaming and NeMoGymResponse objects. The input should use role and content keys, and the response output needs the NeMoGymResponseOutputMessage structure.

If it's designed to succeed, there's a bug in the tutorial. If the test is intentionally broken (tutorial teaches debugging), we should add documentation explaining: Why the test fails (incorrect schema format) and the corrected code.

Steps/Code to reproduce bug

Follow this tutorial to create resource server. Copy and paste the example code into app.py and test_app.py.

Expected behavior

Configs
ng_dump_config
policy_base_url: https://api.openai.com/v1
policy_api_key:
policy_model_name: gpt-4.1-2025-04-14
head_server:
host: 127.0.0.1
port: 11000
disallowed_ports:

  • 11000
    head_server_deps:
  • ray[default]==2.50.1
  • openai==2.6.1
    python_version: 3.12.9

Environment details

  • OS version: Ubuntu 22.04.5 LTS
  • Python version: Python 3.12.9
  • uv pip list output
    Package Version Editable project
    location

    accessible-pygments 0.0.5
    accumulation-tree 0.6.4
    aiofiles 24.1.0
    aiohappyeyeballs 2.6.1
    aiohttp 3.12.15
    aiohttp-cors 0.8.1
    aiosignal 1.4.0
    alabaster 1.0.0
    alembic 1.16.5
    annotated-types 0.7.0
    antlr4-python3-runtime 4.9.3
    anyio 4.10.0
    astroid 3.3.11
    asttokens 2.4.1
    attrs 25.3.0
    babel 2.17.0
    beautifulsoup4 4.13.5
    blinker 1.9.0
    brotli 1.1.0
    cachetools 5.5.2
    certifi 2025.8.3
    cffi 1.17.1
    cfgv 3.4.0
    charset-normalizer 3.4.3
    click 8.2.1
    cloudpickle 3.1.1
    colorama 0.4.6
    colorful 0.5.7
    contourpy 1.3.3
    coverage 7.10.5
    cryptography 45.0.7
    cycler 0.12.1
    databricks-sdk 0.65.0
    datasets 4.4.1
    devtools 0.12.2
    dill 0.4.0
    distlib 0.4.0
    distro 1.9.0
    docker 7.1.0
    docutils 0.21.2
    execnet 2.1.1
    executing 2.2.1
    fastapi 0.116.1
    ffmpy 0.6.1
    filelock 3.19.1
    flask 3.1.2
    fonttools 4.59.2
    frozenlist 1.7.0
    fsspec 2025.9.0
    gitdb 4.0.12
    gitpython 3.1.45
    google-api-core 2.27.0
    google-auth 2.40.3
    googleapis-common-protos 1.71.0
    gradio 5.44.1
    gradio-client 1.12.1
    graphene 3.4.3
    graphql-core 3.2.6
    graphql-relay 3.2.0
    greenlet 3.2.4
    groovy 0.1.2
    grpcio 1.76.0
    gunicorn 23.0.0
    h11 0.16.0
    hf-xet 1.1.9
    httpcore 1.0.9
    httpx 0.28.1
    huggingface-hub 0.34.4
    hydra-core 1.3.2
    identify 2.6.13
    idna 3.10
    imagesize 1.4.1
    importlib-metadata 8.7.0
    iniconfig 2.1.0
    itsdangerous 2.2.0
    jinja2 3.1.6
    jiter 0.10.0
    joblib 1.5.2
    jsonschema 4.25.1
    jsonschema-specifications 2025.9.1
    kiwisolver 1.4.9
    mako 1.3.10
    markdown-it-py 3.0.0
    markupsafe 3.0.2
    matplotlib 3.10.6
    mdit-py-plugins 0.5.0
    mdurl 0.1.2
    mlflow 3.3.2
    mlflow-skinny 3.3.2
    mlflow-tracing 3.3.2
    msgpack 1.1.2
    multidict 6.6.4
    multiprocess 0.70.18
    mypy 1.17.1
    mypy-extensions 1.1.0
    myst-parser 4.0.1
    nemo-gym 0.2.0rc0 /home/nvidia/Gym
    nodeenv 1.9.1
    numpy 2.3.2
    nvidia-sphinx-theme 0.0.8
    omegaconf 2.3.0
    openai 2.6.1
    opencensus 0.11.4
    opencensus-context 0.1.3
    opentelemetry-api 1.36.0
    opentelemetry-exporter-prometheus 0.57b0
    opentelemetry-proto 1.38.0
    opentelemetry-sdk 1.36.0
    opentelemetry-semantic-conventions 0.57b0
    orjson 3.11.3
    packaging 25.0
    pandas 2.3.2
    pathspec 0.12.1
    pillow 11.3.0
    platformdirs 4.4.0
    pluggy 1.6.0
    pre-commit 4.3.0
    prometheus-client 0.23.1
    propcache 0.3.2
    proto-plus 1.26.1
    protobuf 6.32.0
    psutil 7.1.3
    py-spy 0.4.1
    pyarrow 21.0.0
    pyasn1 0.6.1
    pyasn1-modules 0.4.2
    pycparser 2.22
    pydantic 2.11.7
    pydantic-core 2.33.2
    pydata-sphinx-theme 0.16.1
    pydub 0.25.1
    pygments 2.19.2
    pyparsing 3.2.3
    pytest 8.4.1
    pytest-asyncio 1.1.0
    pytest-cov 6.2.1
    pytest-xdist 3.8.0
    python-dateutil 2.9.0.post0
    python-multipart 0.0.20
    pytz 2025.2
    pyudorandom 1.0.0
    pyyaml 6.0.2
    ray 2.50.1
    referencing 0.37.0
    requests 2.32.5
    requests-mock 1.12.1
    rich 14.1.0
    roman-numerals-py 3.1.0
    rpds-py 0.27.1
    rsa 4.9.1
    ruff 0.12.11
    safehttpx 0.1.6
    scikit-learn 1.7.1
    scipy 1.16.1
    semantic-version 2.10.0
    shellingham 1.5.4
    six 1.17.0
    smart-open 7.4.1
    smmap 5.0.2
    sniffio 1.3.1
    snowballstemmer 3.0.1
    soupsieve 2.8
    sphinx 8.2.3
    sphinx-autobuild 2025.8.25
    sphinx-autodoc2 0.5.0
    sphinx-copybutton 0.5.2
    sphinx-design 0.6.1
    sphinx-reredirects 1.0.0
    sphinxcontrib-applehelp 2.0.0
    sphinxcontrib-devhelp 2.0.0
    sphinxcontrib-htmlhelp 2.1.0
    sphinxcontrib-jsmath 1.0.1
    sphinxcontrib-mermaid 1.0.0
    sphinxcontrib-qthelp 2.0.0
    sphinxcontrib-serializinghtml 2.0.0
    sqlalchemy 2.0.43
    sqlparse 0.5.3
    starlette 0.47.3
    swagger-plugin-for-sphinx 6.0.0
    tdigest 0.5.2.2
    threadpoolctl 3.6.0
    tomlkit 0.13.3
    tqdm 4.67.1
    typer 0.17.3
    typing-extensions 4.15.0
    typing-inspection 0.4.1
    tzdata 2025.2
    urllib3 2.5.0
    uvicorn 0.35.0
    uvloop 0.21.0
    virtualenv 20.34.0
    watchfiles 1.1.0
    websockets 15.0.1
    werkzeug 3.1.3
    wrapt 2.0.0
    xxhash 3.6.0
    yappi 1.6.10
    yarl 1.20.1
    zipp 3.23.0
    Additional context

Add any other context about the problem here.
Example: GPU model

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with resources_servers/my_weather_tool/tests/test_app.py and the linked creating-resource-server tutorial, then run test_verify to inspect the reported validation errors. Compare the example inputs and outputs with the named NeMoGymResponse schemas; done means the tutorial example and test agree, or the tutorial clearly documents the intentional failure and corrected code.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.