Autodesk / Autodesk/arnold-usd

Random hangs and crashes with husk renders

Open
#2,259 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
284
Forks
68
Avg merge
2d 19h
Merged PRs (30d)
14

Description

**Describe the bug**
husk renders can randomly crash (Linux) or hang (Windows) when you render scenes with instances and light linking.

**To Reproduce**
Steps to reproduce the behavior:
1. Render the same frame over and over. Usually within 100 retries, husk will crash or hang.

**Additional context**
Here's the backtrace from a Linux crash:

```
>> 0 0x00007f40c0b1dd49 [libai.so ]
void std::__introsort_loop(AtNode**, AtNode**, long, __gnu_cxx::__ops::_Iter_less_iter) ld-temp.o:?
* 1 0x00007f40c0b1d570 [libai.so ]
TypedArray::simplify() ld-temp.o:?
* 2 0x00007f40c072e3c1 [libai.so ]
GObjectUpdate(AtNode*) ld-temp.o:?
* 3 0x00007f40c0b0efa6 [libai.so ]
AtNode::update() ld-temp.o:?
* 4 0x00007f40c0b32a98 [libai.so ]
AtUniverse::UpdateNodes(void*, int) ld-temp.o:?
* 5 0x00007f40c0949928 [libai.so ]
tbb::internal::function_task(void (&)(void*, int), void*)::{lambda()#1}>::execute() ld-temp.o:?
* 6 0x00007f40c12f96c4 [libai.so ]
tbb::internal::custom_scheduler::process_bypass_loop(tbb::internal::context_guard_helper&, tbb::task*, long) ld-temp.o:?
* 7 0x00007f40c12f8d00 [libai.so ]
tbb::internal::custom_scheduler::local_wait_for_all(tbb::task&, tbb::task*) ld-temp.o:?
* 8 0x00007f40c12f1b14 [libai.so ]
tbb::internal::arena::process(tbb::internal::generic_scheduler&) ld-temp.o:?
* 9 0x00007f40c12f1175 [libai.so ]
tbb::internal::market::process(rml::job&) ld-temp.o:?
* 10 0x00007f40c12fb46c [libai.so ]
tbb::internal::rml::private_worker::run() ld-temp.o:?
* 11 0x00007f40c12fb3b5 [libai.so ]
tbb::internal::rml::private_worker::thread_routine(void*) ld-temp.o:?
* 12 0x00007f410e43d1c9 [libpthread.so.0] start_thread
* 13 0x00007f410d751e72 [libc.so.6 ] __GI___clone [:?]
```

And here's the stack from a Windows hang:

```
[0x0] ntdll!RtlDelayExecution+0x43 0x43ff160 0x7ffd91bdc6eb
[0x1] KERNELBASE!SwitchToThread+0x1b 0x43ff190 0x7ffbee2c52c1
[0x2] libpxr_sdf!pxrInternal_v0_24__pxrReserved__::Sdf_PathExpressionEvalBase::_PatternImplBase::_Next+0x3411 0x43ff1c0 0x7ffbee2c9bc9
[0x3] libpxr_sdf!pxrInternal_v0_24__pxrReserved__::Sdf_PrimPathNode::~Sdf_PrimPathNode+0x49 0x43ff210 0x2b53f270
[0x4] libgusd!UT_IntrusiveRefCounter,UT_IntrusiveThreadSafeCounterPolicy>::conditionalAddRef+0x1330 0x43ff250 0x7ffd92143a93
[0x5] ucrtbase!::operator()+0xab 0x43ff280 0x7ffd9211042e
[0x6] ucrtbase!__crt_seh_guarded_call::operator()<, &, >+0x36 0x43ff2d0 0x7ffd9210dddd
[0x7] ucrtbase!execute_onexit_table+0x3d 0x43ff300 0x2b63f97d
[0x8] libgusd!pxrInternal_v0_24__pxrReserved__::GusdXformWrapper::defineForRead+0x3b5d 0x43ff340 0x2b63faa2
[0x9] libgusd!pxrInternal_v0_24__pxrReserved__::GusdXformWrapper::defineForRead+0x3c82 0x43ff380 0x7ffd946d8b8f
[0xa] ntdll!LdrpCallInitRoutine+0x6b 0x43ff3e0 0x7ffd94701596
[0xb] ntdll!LdrShutdownProcess+0x176 0x43ff450 0x7ffd9470118d
[0xc] ntdll!RtlExitUserProcess+0xad 0x43ff560 0x7ffd92847fcb
[0xd] kernel32!ExitProcessImplementation+0xb 0x43ff590 0x7ffd9210bed8
[0xe] ucrtbase!exit_or_terminate_process+0x50 0x43ff5c0 0x7ffd9210c099
[0xf] ucrtbase!common_exit+0x79 0x43ff5f0 0x1dc5c945
[0x10] libUT!UT_Exit::doExit+0x65 0x43ff650 0x1dc5ca2b
[0x11] libUT!UT_Exit::exitWithSpecificCode+0xb 0x43ff680 0x1400c0408
[0x12] husk+0xc0408 0x43ff6b0 0x14024868c
[0x13] husk+0x24868c 0x43ffef0 0x7ffd9284259d
[0x14] kernel32!BaseThreadInitThunk+0x1d 0x43fff30 0x7ffd9470af38
[0x15] ntdll!RtlUserThreadStart+0x28 0x43fff60 0x0
```

`

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue in husk with scenes using instances and light linking, repeatedly rendering the same frame on Linux and Windows. Use the provided stacks as entry points, especially GObjectUpdate and AtUniverse::UpdateNodes in libai, then compare the libpxr_sdf and libgusd hang trace. Done means repeated renders no longer crash or hang.

Written by the indexing model from the issue text.

Assessment

Domain
computer-graphics
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
22/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.