NVIDIA / NVIDIA/NeMo-Retriever
[BUG]: Base python client example fails (paddle model error)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Version
24.10.1
Which installation method(s) does this occur on?
Docker
Describe the bug.
Deployed nvingest with the docker compose from main branch, on H100's.
Tried to run the base python client example, with the default document in the notebook.
Immediately got an error in the handling of paddle NIM output.
Error logs extracted from nvingest runtime container.
Minimum reproducible example
https://github.com/NVIDIA/nv-ingest/blob/main/client/client_examples/examples/python_client_usage.ipynb
Relevant log output
INFO: 172.18.0.7:38424 - "GET /v1/health/ready HTTP/1.1" 200 OK
2025-01-22 11:04:55,866 - ERROR - Inference failed for model paddle: 'int' object has no attribute 'decode'
Unhandled error calling image inference model: Inference failed for model paddle: 'int' object has no attribute 'decode'
Traceback (most recent call last):
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in _call_image_inference_grpc_client
return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in <listcomp>
return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
AttributeError: 'int' object has no attribute 'decode'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 78, in _update_metadata
paddle_result = call_image_inference_model(paddle_client, "paddle", image_array, trace_info=trace_info)
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/tracing/tagging.py", line 185, in wrapper_inject_trace_info
result = func(*args, **kwargs)
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 103, in call_image_inference_model
response = _call_image_inference_grpc_client(client, model_name, image_data)
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 124, in _call_image_inference_grpc_client
raise RuntimeError(err_msg)
RuntimeError: Inference failed for model paddle: 'int' object has no attribute 'decode'
2025-01-22 11:04:55,877 - ERROR - Future result failure - Inference failed for model paddle: 'int' object has no attribute 'decode'
Error occurred while extracting table data.
Traceback (most recent call last):
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in _call_image_inference_grpc_client
return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in <listcomp>
return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
AttributeError: 'int' object has no attribute 'decode'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 133, in _extract_table_data
df["metadata"] = df.apply(_update_metadata, axis=1, args=(paddle_client, paddle_version, trace_info))
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/frame.py", line 9568, in apply
return op.apply().__finalize__(self, method="apply")
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 764, in apply
return self.apply_standard()
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 891, in apply_standard
results, res_index = self.apply_series_generator()
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 907, in apply_series_generator
results[i] = self.f(v)
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 142, in f
return func(x, *args, **kwargs)
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 78, in _update_metadata
paddle_result = call_image_inference_model(paddle_client, "paddle", image_array, trace_info=trace_info)
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/tracing/tagging.py", line 185, in wrapper_inject_trace_info
result = func(*args, **kwargs)
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 103, in call_image_inference_model
response = _call_image_inference_grpc_client(client, model_name, image_data)
File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 124, in _call_image_inference_grpc_client
raise RuntimeError(err_msg)
RuntimeError: Inference failed for model paddle: 'int' object has no attribute 'decode'
Other/Misc.
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the traceback locations in nv_ingest/util/nim/helpers.py and nv_ingest/stages/nim/table_extraction.py, then reproduce the failure using client/client_examples/examples/python_client_usage.ipynb and the reported Docker setup. Check the Paddle inference output handling and table extraction path. Done means the base Python client example completes without the reported decode error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, jupyter-notebook, python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100