AcademySoftwareFoundation / AcademySoftwareFoundation/OpenShadingLanguage
Null Pointer Exception possibly due to Invalid Cast from `llvm::Value*` to `llvm::Function*`
- Dominant language
- C++
- Stars
- 2.3k
- Forks
- 414
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 10
Description
## Problem
I have the suspicion that the following cast [in `llvm_util.cpp` line 4353](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blame/73e420999eccc1de0d8576f09e10edb5b71fee09/src/liboslexec/llvm_util.cpp#L4353 "https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blame/73e420999eccc1de0d8576f09e10edb5b71fee09/src/liboslexec/llvm_util.cpp#L4353") ist not correct, which results in a null value, which in turn results in a null pointer exception occurring.
```C++
static_cast(func)->getFunctionType()
```
** **
The cast is done in the following function, where a null pointer exception occurs when calling `builder().CreateCall(...)` (*here in line 10*):
```C++
llvm::Value*
LLVM_Util::call_function(llvm::Value* func, cspan args)
{
OSL_DASSERT(func);
#if 0
...
#endif
//llvm_gen_debug_printf (std::string("start ") + std::string(name));
llvm::Value* r = builder().CreateCall(
static_cast(func)->getFunctionType(), func,
llvm::ArrayRef(args.data(), args.size()));
//llvm_gen_debug_printf (std::string(" end ") + std::string(name));
return r;
}
```
to get `llvm::Value* func`'s `FunctionType` for `CreateCall`'s `FunctionType *FTy` argument:
```C++
CallInst *CreateCall(FunctionType *FTy, Value *Callee, ArrayRef Args,
ArrayRef OpBundles,
const Twine &Name = "", MDNode *FPMathTag = nullptr) {...}
```
(From `IRBuilder.h` in LLVM's `llvm/IR`.)
** **
`llvm::Value* func_ptr` -- which is passed to `LLVM_Util::call_function`'s `llvm::Value* func` argument -- is created [in `llvm_gen.cpp` line 3879](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/73e420999eccc1de0d8576f09e10edb5b71fee09/src/liboslexec/llvm_gen.cpp#L3879 "https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/73e420999eccc1de0d8576f09e10edb5b71fee09/src/liboslexec/llvm_gen.cpp#L3879") ff. if the closure entry (`clentry`) has a (non-`nullptr`) ''prepare'' method (*here in line 6 ff.*):
```C++
// If the closure has a "prepare" method, call
// prepare(renderer, id, memptr). If there is no prepare method, just
// zero out the closure parameter memory.
if (clentry->prepare) {
// Call clentry->prepare(renderservices *, int id, void *mem)
llvm::Value* funct_ptr
= rop.ll.constant_ptr((void*)clentry->prepare,
rop.llvm_type_prepare_closure_func());
llvm::Value* args[] = { render_ptr, id_int, mem_void_ptr };
rop.ll.call_function(funct_ptr, args);
} else {
rop.ll.op_memset(mem_void_ptr, 0, clentry->struct_size, 4 /*align*/);
}
```
** **
Now from my testing `static_cast(func)` does not cast `llvm::Value* func` to a correct `llvm::Function*`, hence calling members of `llvm::Function` (such as `getFunctionType()`) result in invalid results.
## What Works
An older, **deprecated** (`getPointerElementType()` is deprectaed) way works to get `llvm::FunctionType*` from `llvm::Value* func`:
```C++
llvm::cast(func->getType()->getPointerElementType())
```
This way `llvm::Value* func`'s type is gotten and then cast to `llvm::FunctionType`.
As opposed to first casting `llvm::Value` to `llvm::Function` and then getting its (function) type.
Hence, from what I can tell, I think `llvm::Value* func`'s type should be gotten (like in the deprecated way) and then cast to `llvm::FunctionType`. I could however not figure out how to do this in a way that aligns with the new opaque pointer dogma of LLVM.
### and What also Doesn't Work
Using `llvm::cast` instead of `static_cast` resulted in the same error for me.
Using only `llvm::cast(func->getType())` also didn't work, resulting at a type mismatch at compile time.
## Reproduction
Registering a closure using
```C++
void OSL::ShadingSystem::register_closure(string_view name, int id, const ClosureParam* params,
PrepareClosureFunc prepare, SetupClosureFunc setup);
```
with `PrepareClosureFunc prepare` being non-`nullptr` should cause this issue, as it will then fullfil the ''if the closure has a \"prepare\" method'' condition from above.
** **
Unfortunately I'm not sure how I would go about creating a *minimal reproducible example* for this.
I've run into this problem working on Applessed; its closures can be seen [in this file in the Appleseed repository](https://github.com/appleseedhq/appleseed/blob/1e60a023d4a77965c867dbd296a71e061238acba/src/appleseed/renderer/kernel/shading/closures.cpp "https://github.com/appleseedhq/appleseed/blob/1e60a023d4a77965c867dbd296a71e061238acba/src/appleseed/renderer/kernel/shading/closures.cpp").
## Conclusion
Is my suspicion correct that the cast here is invalid? If so, how would be the correct (non-deprecated) way to get the `llvm::FunctionType`?
Or am I perhaps missing something and this *should* work and not result in a null pointer exception? If so, where might my mistake lie?
I'm thankful for any and all support.
Kind regards,
Alexander
## My Versions
OSL: Release 1.13.11.0
OS: Ubuntu 22.04.5 LTS
C++: Compiler: GNU 11.4.0 / Clang 18.1.8 (issue is the same between both)
LLVM: 14.0.0 (via `build_llvm.bash` build script)
OIIO: Release 2.5.16.0
Contributor guide
Research direction
Start in src/liboslexec/llvm_util.cpp around line 4353 and trace the llvm::Value* passed from src/liboslexec/llvm_gen.cpp around line 3879. Reproduce the problem by registering a closure with a non-null prepare method, then verify the call obtains the correct llvm::FunctionType without an invalid cast or null-pointer failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100