[BUG]: Incorrect behavior with REST client's 'fetch_message' API when a job ID does not exist.

Version

24.10

Which installation method(s) does this occur on?

No response

Describe the bug.

When using the REST Client to fetch a message from FastAPI, if the job ID does not exist, the Rest proxy will immediately return a '404', and the client will immediately re-issue the fetch request. This results in 'spamming' the log outputs, with something similar to the snippet below.

This is incorrect behaviorally: we should engage in some kind of exponential backoff to avoid flooding the user, and recognize that there is a failure if the job ID does not exist after some grace period.

This is also incorrect technically: the 404 isn't a timeout, its an HTTP request failure.

DEBUG:nv_ingest_client.util.processing:Error processing future result: Timeout: No response within 10 seconds for job ID 0
DEBUG:nv_ingest_client.message_clients.rest.rest_client:Invoking fetch_message http endpoint @ 'http://localhost:7670/v1/fetch_job/'
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:7670

Minimum reproducible example

No direct reproducer, this behavior was discovered when diagnosing another issue where a submit_message call to the FastAPI service was failing and silently failing job creation.

Relevant log output

Other/Misc.

No response

Start by tracing the REST client's fetch_message API and its handling of responses from the FastAPI service, focusing on the path for a nonexistent job ID. Reproduce the 404 case if possible, then inspect related timeout and retry behavior; done means 404 failures no longer loop immediately, retries are bounded with backoff, and the failure is surfaced clearly.