llvm / llvm/llvm-project

[opt] Incorrect global ID (gid) computation via hipExtModuleLaunchKernel for non-divisible global sizes when using -fno-offload-uniform-block

Open
#173,588 1 comment 0 reactions 0 assignees View on GitHub
new issue
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

**Environment**
- Backend: AMDGPU
- Compiler: Clang / LLVM 17
- Runtime: HIP (ROCm)
- Kernel launch API: hipExtModuleLaunchKernel
- Global size not divisible by block size (tail workgroup exists)
- Add the -fno-offload-uniform-block option during compilation.

**Problem Description**
When launching a kernel via hipExtModuleLaunchKernel with:
`totalThreads % blockSize != 0`
the kernel exhibits incorrect global id (gid) computation:
- Some lower gid values are executed twice
- This only happens when a tail workgroup is present(i.e., global size is not divisible by block size)
- When the global size is divisible by block size, the problem disappears.

**demo**
- Reproducing the problem
`hipcc --offload-device-only kernel.hip -o kernel.co -fno-offload-uniform-block #-mcode-object-version=5`
`hipcc main.cpp -o main_test`
`./main_test`

- kernel.hip
```
#include
#include

extern "C" __global__
void write_gid_kernel(uint64_t* out)
{
uint64_t gid = blockIdx.x * blockDim.x + threadIdx.x;
printf("gid = %lu\n", gid);
out[gid] = gid;
}
```

- main.cpp

```
#include
#include
#include
#include
#include

int main()
{
hipModule_t module;
hipFunction_t kernel;

hipModuleLoad(&module, "kernel.co");
hipModuleGetFunction(&kernel, module, "write_gid_kernel");

// const int block_size = 256;
// const int total_threads = 1000;

const int block_size = 512;
const int total_threads = 822528; // 822528 % 512 = 256

printf("logical threads = %d\n", total_threads);
printf("block size = %d\n", block_size);

uint64_t* d_out;
hipMalloc(&d_out, total_threads * sizeof(uint64_t));

void* args[] = {
(void*)&d_out
};


hipExtModuleLaunchKernel(
kernel,
total_threads, 1, 1,
block_size, 1, 1,
0,
nullptr,
args,
nullptr
);

hipDeviceSynchronize();

std::vector h_out(total_threads, 0);
hipMemcpy(
h_out.data(), d_out,
total_threads * sizeof(uint64_t),
hipMemcpyDeviceToHost);

hipFree(d_out);
hipModuleUnload(module);
return 0;
}
```

**Observed Behavior**
When running the above demo,
- gid in the range:[411136, 411391] is printed twice
- gid in the final tail region: [822272, totalThreads) is not printed at all
- 822528/512=1606; 822528 % 512 = 256; 1606*256=411136; 411136+256-1=411391

- It is suspected that, for the final remainder workgroup(256 threads), gid computation uses 256 * blockIdx + lane_id, leading to incorrect global IDs.

**Expected Behavior**
Each gid in [0, totalThreads) should be printed exactly once

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with kernel.hip and main.cpp using the listed hipcc commands, comparing divisible and non-divisible launch sizes with -fno-offload-uniform-block. Trace the generated kernel and hipExtModuleLaunchKernel path to determine where the tail workgroup's gid is computed. Done means every gid in [0, totalThreads) is printed exactly once for the non-divisible case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.