AcademySoftwareFoundation / AcademySoftwareFoundation/OpenCue
Frame dependencies are incorrect with an interleaved range and chunking
- Dominant language
- Python
- Stars
- 957
- Forks
- 259
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 16
Description
**Describe the bug**
I've discovered a case where the frame dependencies aren't set up correctly and it leads to frames running before they should, normally resulting in file not found errors.
It can be reproduced using basic shell commands.
In our real world scenario, we have a render layer which produces images and a denoise layer which denoises the rendered image and produces a second image. The denoise layer has a chunk size of 3.
**To Reproduce**
```
cmd = 'echo "Frame: %{FRAME}, zframe: %{ZFRAME}, range: %{RANGE}"'
layer1 = Shell("mock_render_layer", command=cmd, env={"CUE_ENV": "show"}, range="1001-1200:10", chunk=1, shell=True)
layer2 = Shell("mock_denoise_layer", command=cmd, env={"CUE_ENV": "show"}, range="1001-1200:10", chunk=3, shell=True)
layer2.depend_on(layer1) # frame on frame
# dispatch an outline with those 2 layers added.
```
**Expected behavior**
After the first round of frames on 10s, the frame dependencies on the mock_denoise_layer are incorrect, as viewed in cuetopia.
The output for denoise frame 1046:

The dependencies as shown in cuetopia:

**Second example**
It can be shown better by changing the commands to write and read files:
You'll need to substitute a file path that is accessible by the render hosts.
```
render_cmd = `touch /jobs/pipeline/interleave_test/mock_render.%{FRAME}'
layer1 = Shell("mock_render_layer", command=render_cmd, env={"CUE_ENV": "show"}, range="1001-1200:10", chunk=1, shell=True)
denoise_cmd = 'ls /jobs/pipeline/interleave_test/mock_render.%{FRAME}'
layer2 = Shell("mock_denoise_layer", command=denoise_cmd, env={"CUE_ENV": "show"}, range="1001-1200:10", chunk=3, shell=True)
layer2.depend_on(layer1) # frame on frame
# dispatch an outline with those 2 layers added.
```
This will lead to a lot of denoise frames failing as they attempt to `ls` the file before the render layer has touched it.
**Version Number**
0.21
Please let me know if any more information is needed.
Many thanks,
Michael
Contributor guide
Research direction
Start by running the provided Python reproducer with the two Shell layers, interleaved range 1001-1200:10, and downstream chunk size 3. Trace how frame-on-frame dependencies are generated and inspect the resulting dependencies in cuetopia. Done means denoise frames do not run before their corresponding render frames and the file-based example no longer fails because the input is missing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100