NVIDIA / NVIDIA/TensorRT-Edge-LLM
[Bug] gen_cubins.py: XQA cubin generation writes to the module-level default path instead of --output_dir when workers do not inherit __main__ state
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
Platform: Jetson Orin Nano 8GB (Seeed reComputer J3011), JetPack 7.2 / L4T
R39.2.0, custom Yocto-built minimal OS (not stock JetPack), CUDA 13.2. Native
aarch64 cmake build.
Build command:
cmake .. -DCMAKE_BUILD_TYPE=Release -DTRT_PACKAGE_DIR=/usr \
-DEMBEDDED_TARGET=jetson-orin -DCUDA_CTK_VERSION=13.2 -DENABLE_CUTE_DSL=ALL
make
Symptom
The build fails during XQA cubin generation:
ptxas fatal : Output file '../../cpp/kernels/decodeAttentionKernels/cubin/xqa_kernel_....cubin' could not be opened
Note the path: it is the relative module-level default, not the absolute
--output_dir CMake passes in.
What I think is happening
Line numbers are against main as of 2026-08-04 (kernelSrcs/xqa/gen_cubins.py).
cubin_dir is defined at module scope with a relative default:
# line 81
cubin_dir = "../../cpp/kernels/decodeAttentionKernels/cubin/"
It is rebound to the resolved output directory, but only inside the __main__
block:
# line 705
if __name__ == "__main__":
...
# line 709
cubin_dir = os.path.abspath(args.output_dir)
The directory that gets created is that resolved path:
# lines 829-831
if os.path.exists(cubin_dir):
shutil.rmtree(cubin_dir)
os.makedirs(cubin_dir)
But the value the workers actually use is read at module scope, inside
build_commands() (line 394, for the nvcc -o argument) and
save_cubin_cpp_file() (line 414), both reached from run_cubin_gen(), which is
dispatched to a pool:
# lines 835-836
with multiprocessing.Pool(processes=thread_count) as pool:
name_size_list = pool.map(run_cubin_gen, arch_macro_lists)
Under the fork start method this is fine — children inherit the parent's memory
after line 709 has run, so they see the resolved path. Under a start method that
re-imports the module in the child (spawn, forkserver), __name__ is not
"__main__" there, line 709 never executes, and the workers fall back to the
line 81 relative default — which is not the directory os.makedirs created, and
generally will not exist relative to the worker's cwd. That matches the failure
exactly: ptxas reporting the relative path as unopenable.
What I could not confirm
I have not reproduced this under fork, and I don't believe it can occur
there. Checking after the fact, the two Python installs I still have access to
(3.12.3 on the device, 3.13.13 on my host) both default to fork, where I would
expect the current code to work. I no longer have the exact build container from
the failing run and did not record its Python version, so I can't state which
start method was in play — I'm inferring it from the relative path in the error.
Since Python 3.14 changes the default start method on Linux to forkserver, this
would begin affecting fork-based setups on newer interpreters even where it
works today, which is the main reason I'm reporting it despite the incomplete
reproduction.
Questions
- Is the module-level
cubin_dirat line 81 intended as a real fallback, or is
it vestigial and only ever meant to be overridden by--output_dir? - If the latter, would you accept a change that makes the resolved path explicit
to the workers — passing it as part of thepool.mappayload, or setting it
via aPool(initializer=...)— so the behaviour no longer depends on the
start method?
Happy to test a patch on this hardware, or to provide the full cmake/make logs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read kernelSrcs/xqa/gen_cubins.py, starting with the module-level cubin_dir and the main block, then trace build_commands(), save_cubin_cpp_file(), and run_cubin_gen() through the Pool.map call. Verify that workers launched with spawn or forkserver use the resolved --output_dir, and confirm cubin generation writes successfully there.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, python
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100