ROCm / ROCm/AMDMIGraphX

libmigraphx_c unconditionally loads the TensorFlow parser and causes duplicate protobuf descriptor registration

Open
#5,089 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
333
Forks
150
Avg merge
4d 19h
Merged PRs (30d)
54

Description

Description

Loading the ONNX Runtime MIGraphX Execution Provider in a process that
already contains OpenCV DNN aborts during dlopen():

File already exists in database: tensor_shape.proto
Check failed: GeneratedDatabase()->Add(encoded_file_descriptor, size)

The crash occurs before model execution, while loading the MIGraphX
provider.

Environment
  • MIGraphX: 7.2.3
  • ROCm: 7.2.3
  • ONNX Runtime: reproduced with 1.26.0 and 1.27.1
  • protobuf: 35.1
  • OpenCV: 4.13.0
  • Application: darktable 5.6.0 on NixOS
  • Provider: MIGraphX
  • CPU provider works correctly
Stack trace
google::protobuf::DescriptorPool::InternalAddGeneratedFile()
google::protobuf::internal::AddDescriptors()
call_init
_dl_init
dlopen
onnxruntime::PosixEnv::LoadDynamicLibrary()
onnxruntime::ProviderLibrary::Load()
OrtSessionOptionsAppendExecutionProvider_MIGraphX
Analysis

darktable loads G'MIC, which brings OpenCV DNN into the process.
OpenCV DNN registers TensorFlow-related protobuf descriptors.

MIGraphX's stable C API library links both parsers unconditionally:

https://github.com/ROCm/AMDMIGraphX/blob/rocm-7.2.3/src/api/CMakeLists.txt#L40

target_link_libraries(migraphx_c PRIVATE migraphx migraphx_tf migraphx_onnx)

Consequently, loading libmigraphx_c.so.3 for ONNX inference also loads
libmigraphx_tf, although the TensorFlow parser is not used. This causes
tensor_shape.proto to be registered for a second time and protobuf
aborts.

Removing the TensorFlow API implementation from src/api/api.cpp and
removing migraphx_tf from the migraphx_c link dependencies fixes the
crash. The ONNX Runtime MIGraphX provider then loads successfully after
OpenCV DNN.

Disabling G'MIC/OpenCV in darktable also avoids the crash, but this is an
application-level workaround.

Expected behavior

Using the MIGraphX C API for ONNX inference should not load the TensorFlow
parser or register its protobuf descriptors.

Would it be possible to split the TensorFlow parser API from
libmigraphx_c, or otherwise make parser dependencies optional?

Minimal reproduction

The crash can be reproduced without darktable or ONNX Runtime by loading
OpenCV DNN followed by the unmodified MIGraphX C API:

import ctypes

ctypes.CDLL(
    "/path/to/libopencv_dnn.so",
    mode=ctypes.RTLD_GLOBAL,
)
ctypes.CDLL("/path/to/libmigraphx_c.so.3")

Result:

File already exists in database: tensor_shape.proto
Check failed: GeneratedDatabase()->Add(encoded_file_descriptor, size)
Aborted (exit status 134)

Tested with:

  • OpenCV 4.13.0
  • MIGraphX 7.2.3
  • protobuf 35.1

After removing migraphx_tf from the libmigraphx_c link dependencies
and removing the TensorFlow C API implementation from src/api/api.cpp,
the same reproducer succeeds with exit status 0.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/api/CMakeLists.txt around the migraphx_c target and src/api/api.cpp, then review how the TensorFlow and ONNX parser APIs are exposed and linked. Run the provided ctypes reproducer with OpenCV DNN loaded first; done means libmigraphx_c loads without duplicate protobuf registration while ONNX Runtime MIGraphX loading still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
backend, build-system, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.