ContextualAIParseTool can poll forever when a parse job never reaches a terminal state

Summary

ContextualAIParseTool._run() submits a Contextual AI parse job and then polls its status in an unconditional while True loop:

status_url = f"{base_url}/parse/jobs/{job_id}/status"
while True:
    result = requests.get(status_url, headers=headers, timeout=30)
    parse_response = json.loads(result.text)["status"]

    if parse_response == "completed":
        break
    if parse_response == "failed":
        raise RuntimeError("Document parsing failed")

    sleep(5)

Each individual GET has a 30-second request timeout, but there is no bound on the number of polls or total elapsed time. If the remote job remains in a non-terminal state such as pending/processing indefinitely, the CrewAI tool invocation never returns.

Why this matters

Tool-level HTTP timeouts only bound one request; they do not bound an asynchronous job lifecycle. A stuck upstream job can therefore hold an agent task indefinitely, which is particularly problematic for autonomous crews where the tool call may block subsequent steps.

Expected behavior

The parser should enforce a configurable or documented maximum polling duration / attempt count and return a clear timeout error when that budget is exhausted.

For example, a narrow implementation could use a monotonic deadline:

from time import monotonic, sleep

poll_timeout = 300
started = monotonic()

while True:
    ...
    if monotonic() - started >= poll_timeout:
        raise TimeoutError(
            f"Document parsing did not complete within {poll_timeout}s"
        )
    sleep(5)

A max-attempts approach would also work. Ideally the budget should be configurable on the tool while preserving the current polling interval by default.

Related request-boundary hardening

The same polling path parses response bodies with json.loads(result.text) without calling raise_for_status(). A 4xx/5xx response containing JSON with a different shape can currently surface as KeyError('status') rather than a clear HTTP failure. This can be handled separately if maintainers prefer keeping the polling-deadline change narrowly scoped.

Regression coverage

A deterministic test can mock:

1. the submit request returning a job id;
2. repeated status responses returning processing;
3. a monotonic clock crossing the configured deadline;

and assert that _run() returns/fails with a bounded timeout instead of continuing to poll. A second case can verify that completed still exits normally before the deadline.

I searched the current CrewAI issue and PR trackers for this specific ContextualAI parse polling-deadline problem and did not find an existing report.

_{AI-assisted review disclosure: an AI coding assistant was used to inspect the polling control flow and help prepare this source-backed report.}

Start at ContextualAIParseTool._run() and trace the submit request, status polling loop, and existing response handling. Add or update deterministic regression coverage using mocked submit and repeated processing responses, then verify that polling stops with a clear timeout while a completed response still exits normally.