AcademySoftwareFoundation / AcademySoftwareFoundation/OpenShadingLanguage

Intermittent crash in LLVM_Util::getPointerToFunction(llvm::Function* func)

Open
#1,712 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.3k
Forks
414
Avg merge
3d 1h
Merged PRs (30d)
10

Description

### Problem

In 3ds max, we have lots of users crashing with a callstack that seems to be caused by this problem.
We have a scene that "reproduces" the problem, but the reproduction is intermittent and seems to a race condition of sorts.
Basically, you load a particular file, you start an interactive render and the material editor at the same time, then start changing parameters in the material many many many many many times. Eventually, we get this crash. Or not. Depending on phase of the moon, the wind direction, humidity, etc.

Crash is reported on this line:

![image](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/assets/102356572/f7e0be85-5d89-4ae1-a870-68bc88c1d613)

...i.e. in the case this function is reached before the shader has been optimized. Somehow, it seems like the call to exec->finalizeObject(); crashes.

The call stack is something like this:

> oslexec.dll!OSL_v1_12::pvt::LLVM_Util::getPointerToFunction(llvm::Function * func) Line 1712 C++
oslexec.dll!OSL_v1_12::pvt::BackendLLVM::run() Line 1674 C++
oslexec.dll!OSL_v1_12::pvt::ShadingSystemImpl::optimize_group(OSL_v1_12::ShaderGroup & group, OSL_v1_12::ShadingContext * ctx, bool do_jit) Line 3595 C++
oslexec.dll!OSL_v1_12::ShadingContext::execute_init(OSL_v1_12::ShaderGroup & sgroup, int shadeindex, OSL_v1_12::ShaderGlobals & ssg, void * userdata_base_ptr, void * output_base_ptr, bool run) Line 91 C++
oslexec.dll!OSL_v1_12::ShadingContext::execute(OSL_v1_12::ShaderGroup & sgroup, int shadeindex, OSL_v1_12::ShaderGlobals & ssg, void * userdata_base_ptr, void * output_base_ptr, bool run) Line 217 C++
oslexec.dll!OSL_v1_12::pvt::ShadingSystemImpl::execute(OSL_v1_12::ShadingContext & ctx, OSL_v1_12::ShaderGroup & group, int index, OSL_v1_12::ShaderGlobals & ssg, void * userdata_base_ptr, void * output_base_ptr, bool run) Line 3264 C++
[Inline Frame] OSLMap.dlt!OSL_v1_12::ShadingSystem::execute(OSL_v1_12::ShadingContext &) Line 688 C++
[Inline Frame] OSLMap.dlt!OSL_v1_12::ShadingSystem::execute(OSL_v1_12::ShadingContext *) Line 695 C++
OSLMap.dlt!OSLTex::EvalColor(ShadeContext & sc, int output, bool bump) Line 3227 C++
OSLMap.dlt!OSLTex::EvalColor(ShadeContext & sc) Line 2936 C++
3dsmax.exe!RenderTexmapRange::__l5::::operator()(const tbb::blocked_range & rng) Line 1951 C++
[Inline Frame] 3dsmax.exe!tbb::interface9::internal::start_for,`RenderTexmapRange'::`5'::,tbb::auto_partitioner const>::run_body(tbb::blocked_range &) Line 115 C++
3dsmax.exe!tbb::interface9::internal::dynamic_grainsize_mode>::work_balance,`RenderTexmapRange'::`5'::,tbb::auto_partitioner const>,tbb::blocked_range>(tbb::interface9::internal::start_for,`RenderTexmapRange'::`5'::,tbb::auto_partitioner const> & start, tbb::blocked_range & range) Line 439 C++
3dsmax.exe!tbb::interface9::internal::start_for,`RenderTexmapRange'::`5'::,tbb::auto_partitioner const>::execute() Line 143 C++
[External Code]
[Inline Frame] 3dsmax.exe!tbb::task::spawn_root_and_wait(tbb::task &) Line 809 C++
[Inline Frame] 3dsmax.exe!tbb::interface9::internal::start_for,`RenderTexmapRange'::`5'::,tbb::auto_partitioner const>::run(const tbb::blocked_range &) Line 95 C++
[Inline Frame] 3dsmax.exe!tbb::parallel_for(const tbb::blocked_range &) Line 201 C++
3dsmax.exe!RenderTexmapRange(HWND__ * hwnd, Texmap * tx, Bitmap * bm, FBox2 * range, float scale3d, int filter, int display, int t, const wchar_t * name, float z, int mono, bool disableBitmapProxies, bool bake) Line 1925 C++
3dsmax.exe!RenderTexmap(HWND__ * hwnd, Texmap * tex, Bitmap * bm, float scale3d, int filter, int display, int t, const wchar_t * name, float z, int mono, bool disableBitmapProxies, bool bake) Line 1877 C++
3dsmax.exe!InterfaceImp::Execute(int cmd, unsigned __int64 arg1, unsigned __int64 arg2, unsigned __int64 arg3, unsigned __int64 arg4, unsigned __int64 arg5, unsigned __int64 arg6) Line 6844 C++
core.dll!Texmap::GetVPDisplayDIB(int t, TexHandleMaker & thmaker, Interval & valid, int mono, int forceW, int forceH) Line 3851 C++

**Expected behavior:**

It not to crash?

**Actual behavior:**

It crash. Sometimes.

### Steps to Reproduce

1. [First Step]
2. [Second Step]
3. [and so on...]

### Versions

* OSL branch/version: Internal Autodesk fork of OSL 1.12.13
* OS: Windows
* C++ compiler: Visual Studio 2019 / 2022
* LLVM version: llvm-14.0.6-3dsmax-001-osl_subset-vc142.7z
* OIIO version: OpenImageIO-2.4.5.0-3dsmax-002-vc142.zip

Contributor guide

Open the contributing guide

Research direction

Start at LLVM_Util::getPointerToFunction(), reported at line 1712, and follow the call from BackendLLVM::run() to exec->finalizeObject(). Use the described 3ds Max scene and concurrent interactive render/material-editor parameter changes with the listed OSL, LLVM, OIIO, Windows, and Visual Studio versions. Done means the intermittent crash is reproduced and resolved or otherwise prevented under this workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, computer-graphics
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.