AcademySoftwareFoundation / AcademySoftwareFoundation/OpenCue
Frame retry default/max should be configurable without recompiling Cuebot
- Dominant language
- Python
- Stars
- 957
- Forks
- 259
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 16
Description
## Summary
There is no way to configure the default (or maximum) frame retry count for submitted jobs without editing Java source and recompiling Cuebot. The values are compile-time constants in `JobSpec.java`, and the current defaults silently clamp every job to **1 retry** regardless of what the client requests. This should be configurable via `opencue.properties` (and/or the `CONFIG` DB table) so operators can set a facility-wide default without a custom build.
## Current behavior
Retries at submission time are governed by two compile-time constants in `cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java`:
```java
public static final int FRAME_RETRIES_DEFAULT = 1; // applied only when the spec omits
public static final int FRAME_RETRIES_MAX = 1; // hard cap — clamps whatever the client sends
public static final int FRAME_RETRIES_MIN = 0;
```
Parse path (`JobSpec.java` ~L301, L317-324):
```java
job.maxRetries = FRAME_RETRIES_DEFAULT;
...
if (jobTag.getChildTextTrim("maxretries") != null) {
job.maxRetries = Integer.valueOf(jobTag.getChildTextTrim("maxretries"));
if (job.maxRetries > FRAME_RETRIES_MAX) {
job.maxRetries = FRAME_RETRIES_MAX; // clamps DOWN to 1
} else if (job.maxRetries < FRAME_RETRIES_MIN) {
job.maxRetries = FRAME_RETRIES_MIN;
}
}
```
The clamp is the footgun: **pyoutline always emits a `` tag** (default `2` from `pyoutline/outline/outline.cfg`), so every normal job hits the `> FRAME_RETRIES_MAX` branch and is clamped to **1**. This has been the case since the initial 2018 import (`FRAME_RETRIES_MAX = 1` unchanged).
At runtime, `FrameCompleteHandler` marks a frame `DEAD` when `frame.retries >= job.maxRetries`, so the clamped value directly limits how many times a frame reruns before dying.
### Things that look like config but aren't
- **`job.int_max_retries SMALLINT DEFAULT 3`** (`V1__Initial_schema.sql`) — never applies; the job INSERT always supplies `job.maxRetries` explicitly (`JobDaoJdbc`), overriding the column default.
- **`CONFIG.MAX_FRAME_RETRIES = 16`** (`seed_data.sql`) — only consulted by `JobDaoJdbc.updateMaxFrameRetries()`, i.e. the post-submission `SetMaxRetries` gRPC/API ceiling. It is **not** read during job submission.
- **No `opencue.properties` key** exists for default/max frame retries. (`dispatcher.frame_kill_retry_limit` is unrelated — it governs kill-RPC retries.)
## Why this is a problem
To make all jobs default to, say, 3 retries today, an operator must:
1. Edit `FRAME_RETRIES_DEFAULT = 3` **and** `FRAME_RETRIES_MAX = 3` in `JobSpec.java` (editing only `DEFAULT` does nothing, because the client's `` tag gets clamped by `MAX`),
2. Edit `pyoutline/outline/outline.cfg` `maxretries = 3`,
3. Rebuild and redeploy a custom Cuebot image.
That's a source fork just to change a policy number, which is a poor operator experience for a Dockerized deployment.
## Proposed solution
Make the default and max frame retries runtime-configurable, with the existing constants as fallback defaults. Preferred: `opencue.properties` (consistent with other dispatcher tunables), e.g.:
```properties
# Default frame retries applied when a job spec omits
dispatcher.frame_retries_default=1
# Upper bound clamp applied to job-spec at submission
dispatcher.frame_retries_max=16
```
Wire these into `JobSpec` (via injected config) so `FRAME_RETRIES_DEFAULT` / `FRAME_RETRIES_MAX` read the property with the current constant as the fallback default. Alternatively (or additionally), source the submission-time max from the existing `CONFIG.MAX_FRAME_RETRIES` row so there's a single DB-backed source of truth shared with the `SetMaxRetries` API.
### Acceptance criteria
- Operators can set a facility-wide default retry count via configuration only (no recompile).
- The submission-time clamp is configurable and defaults to a sane value (>= the client default so pyoutline's `maxretries` isn't silently reduced).
- Backward compatible: absent config, behavior matches today's constants.
- Document the new keys in `opencue.properties` (relates to #2393).
## Additional notes
- Consider raising the stock `FRAME_RETRIES_MAX` default — clamping to 1 while pyoutline ships a default of 2 means the two defaults contradict each other out of the box.
- There's also a hardcoded `job.maxRetries = 2` in the PostJobs path (`JobSpec.java` ~L945) that should honor the same config.
Contributor guide
Research direction
Start with cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java, including the submission parse path around the retry constants and the PostJobs path. Compare the existing opencue.properties configuration approach with JobDaoJdbc and the CONFIG.MAX_FRAME_RETRIES references. Done means operators can configure the default and submission clamp without recompiling, absent settings preserve current behavior, and the new keys are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- backend, devops, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100