AcademySoftwareFoundation / AcademySoftwareFoundation/OpenShadingLanguage

Null Pointer Exception possibly due to Invalid Cast from `llvm::Value*` to `llvm::Function*`

Open
#1,957 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.3k
Forks
414
Avg merge
3d 1h
Merged PRs (30d)
10

Description

## Problem

I have the suspicion that the following cast [in `llvm_util.cpp` line 4353](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blame/73e420999eccc1de0d8576f09e10edb5b71fee09/src/liboslexec/llvm_util.cpp#L4353 "https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blame/73e420999eccc1de0d8576f09e10edb5b71fee09/src/liboslexec/llvm_util.cpp#L4353") ist not correct, which results in a null value, which in turn results in a null pointer exception occurring.

```C++
static_cast(func)->getFunctionType()
```

** **

The cast is done in the following function, where a null pointer exception occurs when calling `builder().CreateCall(...)` (*here in line 10*):

```C++
llvm::Value*
LLVM_Util::call_function(llvm::Value* func, cspan args)
{
OSL_DASSERT(func);
#if 0
...
#endif
//llvm_gen_debug_printf (std::string("start ") + std::string(name));
llvm::Value* r = builder().CreateCall(
static_cast(func)->getFunctionType(), func,
llvm::ArrayRef(args.data(), args.size()));
//llvm_gen_debug_printf (std::string(" end ") + std::string(name));
return r;
}
```

to get `llvm::Value* func`'s `FunctionType` for `CreateCall`'s `FunctionType *FTy` argument:

```C++
CallInst *CreateCall(FunctionType *FTy, Value *Callee, ArrayRef Args,
ArrayRef OpBundles,
const Twine &Name = "", MDNode *FPMathTag = nullptr) {...}
```

(From `IRBuilder.h` in LLVM's `llvm/IR`.)

** **

`llvm::Value* func_ptr` -- which is passed to `LLVM_Util::call_function`'s `llvm::Value* func` argument -- is created [in `llvm_gen.cpp` line 3879](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/73e420999eccc1de0d8576f09e10edb5b71fee09/src/liboslexec/llvm_gen.cpp#L3879 "https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/73e420999eccc1de0d8576f09e10edb5b71fee09/src/liboslexec/llvm_gen.cpp#L3879") ff. if the closure entry (`clentry`) has a (non-`nullptr`) ''prepare'' method (*here in line 6 ff.*):

```C++
// If the closure has a "prepare" method, call
// prepare(renderer, id, memptr). If there is no prepare method, just
// zero out the closure parameter memory.
if (clentry->prepare) {
// Call clentry->prepare(renderservices *, int id, void *mem)
llvm::Value* funct_ptr
= rop.ll.constant_ptr((void*)clentry->prepare,
rop.llvm_type_prepare_closure_func());
llvm::Value* args[] = { render_ptr, id_int, mem_void_ptr };
rop.ll.call_function(funct_ptr, args);
} else {
rop.ll.op_memset(mem_void_ptr, 0, clentry->struct_size, 4 /*align*/);
}
```

** **

Now from my testing `static_cast(func)` does not cast `llvm::Value* func` to a correct `llvm::Function*`, hence calling members of `llvm::Function` (such as `getFunctionType()`) result in invalid results.

## What Works

An older, **deprecated** (`getPointerElementType()` is deprectaed) way works to get `llvm::FunctionType*` from `llvm::Value* func`:

```C++
llvm::cast(func->getType()->getPointerElementType())
```

This way `llvm::Value* func`'s type is gotten and then cast to `llvm::FunctionType`.

As opposed to first casting `llvm::Value` to `llvm::Function` and then getting its (function) type.

Hence, from what I can tell, I think `llvm::Value* func`'s type should be gotten (like in the deprecated way) and then cast to `llvm::FunctionType`. I could however not figure out how to do this in a way that aligns with the new opaque pointer dogma of LLVM.

### and What also Doesn't Work

Using `llvm::cast` instead of `static_cast` resulted in the same error for me.

Using only `llvm::cast(func->getType())` also didn't work, resulting at a type mismatch at compile time.

## Reproduction

Registering a closure using

```C++
void OSL::ShadingSystem::register_closure(string_view name, int id, const ClosureParam* params,
PrepareClosureFunc prepare, SetupClosureFunc setup);
```

with `PrepareClosureFunc prepare` being non-`nullptr` should cause this issue, as it will then fullfil the ''if the closure has a \"prepare\" method'' condition from above.

** **

Unfortunately I'm not sure how I would go about creating a *minimal reproducible example* for this.

I've run into this problem working on Applessed; its closures can be seen [in this file in the Appleseed repository](https://github.com/appleseedhq/appleseed/blob/1e60a023d4a77965c867dbd296a71e061238acba/src/appleseed/renderer/kernel/shading/closures.cpp "https://github.com/appleseedhq/appleseed/blob/1e60a023d4a77965c867dbd296a71e061238acba/src/appleseed/renderer/kernel/shading/closures.cpp").

## Conclusion

Is my suspicion correct that the cast here is invalid? If so, how would be the correct (non-deprecated) way to get the `llvm::FunctionType`?

Or am I perhaps missing something and this *should* work and not result in a null pointer exception? If so, where might my mistake lie?

I'm thankful for any and all support.

Kind regards,

Alexander

## My Versions

OSL: Release 1.13.11.0
OS: Ubuntu 22.04.5 LTS
C++: Compiler: GNU 11.4.0 / Clang 18.1.8 (issue is the same between both)
LLVM: 14.0.0 (via `build_llvm.bash` build script)
OIIO: Release 2.5.16.0

Contributor guide

Open the contributing guide

Research direction

Start in src/liboslexec/llvm_util.cpp around line 4353 and trace the llvm::Value* passed from src/liboslexec/llvm_gen.cpp around line 3879. Reproduce the problem by registering a closure with a non-null prepare method, then verify the call obtains the correct llvm::FunctionType without an invalid cast or null-pointer failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.