AcademySoftwareFoundation / AcademySoftwareFoundation/OpenCue

Frame retry default/max should be configurable without recompiling Cuebot

Open
#2,477 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
957
Forks
259
Avg merge
3d 6h
Merged PRs (30d)
16

Description

## Summary

There is no way to configure the default (or maximum) frame retry count for submitted jobs without editing Java source and recompiling Cuebot. The values are compile-time constants in `JobSpec.java`, and the current defaults silently clamp every job to **1 retry** regardless of what the client requests. This should be configurable via `opencue.properties` (and/or the `CONFIG` DB table) so operators can set a facility-wide default without a custom build.

## Current behavior

Retries at submission time are governed by two compile-time constants in `cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java`:

```java
public static final int FRAME_RETRIES_DEFAULT = 1; // applied only when the spec omits
public static final int FRAME_RETRIES_MAX = 1; // hard cap — clamps whatever the client sends
public static final int FRAME_RETRIES_MIN = 0;
```

Parse path (`JobSpec.java` ~L301, L317-324):

```java
job.maxRetries = FRAME_RETRIES_DEFAULT;
...
if (jobTag.getChildTextTrim("maxretries") != null) {
job.maxRetries = Integer.valueOf(jobTag.getChildTextTrim("maxretries"));
if (job.maxRetries > FRAME_RETRIES_MAX) {
job.maxRetries = FRAME_RETRIES_MAX; // clamps DOWN to 1
} else if (job.maxRetries < FRAME_RETRIES_MIN) {
job.maxRetries = FRAME_RETRIES_MIN;
}
}
```

The clamp is the footgun: **pyoutline always emits a `` tag** (default `2` from `pyoutline/outline/outline.cfg`), so every normal job hits the `> FRAME_RETRIES_MAX` branch and is clamped to **1**. This has been the case since the initial 2018 import (`FRAME_RETRIES_MAX = 1` unchanged).

At runtime, `FrameCompleteHandler` marks a frame `DEAD` when `frame.retries >= job.maxRetries`, so the clamped value directly limits how many times a frame reruns before dying.

### Things that look like config but aren't

- **`job.int_max_retries SMALLINT DEFAULT 3`** (`V1__Initial_schema.sql`) — never applies; the job INSERT always supplies `job.maxRetries` explicitly (`JobDaoJdbc`), overriding the column default.
- **`CONFIG.MAX_FRAME_RETRIES = 16`** (`seed_data.sql`) — only consulted by `JobDaoJdbc.updateMaxFrameRetries()`, i.e. the post-submission `SetMaxRetries` gRPC/API ceiling. It is **not** read during job submission.
- **No `opencue.properties` key** exists for default/max frame retries. (`dispatcher.frame_kill_retry_limit` is unrelated — it governs kill-RPC retries.)

## Why this is a problem

To make all jobs default to, say, 3 retries today, an operator must:

1. Edit `FRAME_RETRIES_DEFAULT = 3` **and** `FRAME_RETRIES_MAX = 3` in `JobSpec.java` (editing only `DEFAULT` does nothing, because the client's `` tag gets clamped by `MAX`),
2. Edit `pyoutline/outline/outline.cfg` `maxretries = 3`,
3. Rebuild and redeploy a custom Cuebot image.

That's a source fork just to change a policy number, which is a poor operator experience for a Dockerized deployment.

## Proposed solution

Make the default and max frame retries runtime-configurable, with the existing constants as fallback defaults. Preferred: `opencue.properties` (consistent with other dispatcher tunables), e.g.:

```properties
# Default frame retries applied when a job spec omits
dispatcher.frame_retries_default=1
# Upper bound clamp applied to job-spec at submission
dispatcher.frame_retries_max=16
```

Wire these into `JobSpec` (via injected config) so `FRAME_RETRIES_DEFAULT` / `FRAME_RETRIES_MAX` read the property with the current constant as the fallback default. Alternatively (or additionally), source the submission-time max from the existing `CONFIG.MAX_FRAME_RETRIES` row so there's a single DB-backed source of truth shared with the `SetMaxRetries` API.

### Acceptance criteria

- Operators can set a facility-wide default retry count via configuration only (no recompile).
- The submission-time clamp is configurable and defaults to a sane value (>= the client default so pyoutline's `maxretries` isn't silently reduced).
- Backward compatible: absent config, behavior matches today's constants.
- Document the new keys in `opencue.properties` (relates to #2393).

## Additional notes

- Consider raising the stock `FRAME_RETRIES_MAX` default — clamping to 1 while pyoutline ships a default of 2 means the two defaults contradict each other out of the box.
- There's also a hardcoded `job.maxRetries = 2` in the PostJobs path (`JobSpec.java` ~L945) that should honor the same config.

Contributor guide

Open the contributing guide

Research direction

Start with cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java, including the submission parse path around the retry constants and the PostJobs path. Compare the existing opencue.properties configuration approach with JobDaoJdbc and the CONFIG.MAX_FRAME_RETRIES references. Done means operators can configure the default and submission clamp without recompiling, absent settings preserve current behavior, and the new keys are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
backend, devops, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.