NVIDIA / NVIDIA/NeMo-Retriever

[BUG]: Base python client example fails (paddle model error)

Open
#371 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Version

24.10.1

Which installation method(s) does this occur on?

Docker

Describe the bug.

Deployed nvingest with the docker compose from main branch, on H100's.

Tried to run the base python client example, with the default document in the notebook.

Immediately got an error in the handling of paddle NIM output.
Error logs extracted from nvingest runtime container.

Minimum reproducible example
https://github.com/NVIDIA/nv-ingest/blob/main/client/client_examples/examples/python_client_usage.ipynb
Relevant log output
INFO:     172.18.0.7:38424 - "GET /v1/health/ready HTTP/1.1" 200 OK
2025-01-22 11:04:55,866 - ERROR - Inference failed for model paddle: 'int' object has no attribute 'decode'
Unhandled error calling image inference model: Inference failed for model paddle: 'int' object has no attribute 'decode'
Traceback (most recent call last):
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in _call_image_inference_grpc_client
    return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in <listcomp>
    return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
AttributeError: 'int' object has no attribute 'decode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 78, in _update_metadata
    paddle_result = call_image_inference_model(paddle_client, "paddle", image_array, trace_info=trace_info)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/tracing/tagging.py", line 185, in wrapper_inject_trace_info
    result = func(*args, **kwargs)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 103, in call_image_inference_model
    response = _call_image_inference_grpc_client(client, model_name, image_data)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 124, in _call_image_inference_grpc_client
    raise RuntimeError(err_msg)
RuntimeError: Inference failed for model paddle: 'int' object has no attribute 'decode'
2025-01-22 11:04:55,877 - ERROR - Future result failure - Inference failed for model paddle: 'int' object has no attribute 'decode'

Error occurred while extracting table data.
Traceback (most recent call last):
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in _call_image_inference_grpc_client
    return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in <listcomp>
    return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
AttributeError: 'int' object has no attribute 'decode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 133, in _extract_table_data
    df["metadata"] = df.apply(_update_metadata, axis=1, args=(paddle_client, paddle_version, trace_info))
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/frame.py", line 9568, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 764, in apply
    return self.apply_standard()
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 891, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 907, in apply_series_generator
    results[i] = self.f(v)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 142, in f
    return func(x, *args, **kwargs)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 78, in _update_metadata
    paddle_result = call_image_inference_model(paddle_client, "paddle", image_array, trace_info=trace_info)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/tracing/tagging.py", line 185, in wrapper_inject_trace_info
    result = func(*args, **kwargs)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 103, in call_image_inference_model
    response = _call_image_inference_grpc_client(client, model_name, image_data)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 124, in _call_image_inference_grpc_client
    raise RuntimeError(err_msg)
RuntimeError: Inference failed for model paddle: 'int' object has no attribute 'decode'
Other/Misc.

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the traceback locations in nv_ingest/util/nim/helpers.py and nv_ingest/stages/nim/table_extraction.py, then reproduce the failure using client/client_examples/examples/python_client_usage.ipynb and the reported Docker setup. Check the Paddle inference output handling and table extraction path. Done means the base Python client example completes without the reported decode error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, jupyter-notebook, python
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.