trinodb / trinodb/trino-python-client

339 regression: HTTP 200 empty-body retry causes severe spooled-result regression and retries valid empty responses

Open
#636 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
439
Forks
206
PR merge metrics
No merged PRs in 30d

Description

Expected behavior

The HTTP 200 empty-response retry added for statement responses in f8b1203 / #596 should apply only to Trino statement-protocol responses.

Binary spooled-result downloads and other HTTP operations which legitimately return an empty response body should not be inspected as text or retried solely because their body is empty.

Actual behavior

Since trino-python-client 0.339.0, the HTTP 200 empty-body retry condition is installed on the shared _get, _post, _delete, and _head request wrappers:

lambda response: getattr(response, "status_code", None) == 200
and not getattr(response, "text", "").strip()

Those wrappers are also used by spooled-result requests.

This causes at least two observable regressions.

1. Binary spooled-result downloads become extremely slow

Accessing response.text on a compressed binary segment causes requests to determine an apparent encoding when no usable charset is present.

For compressed spooled segments this can invoke charset_normalizer over the complete binary response, even though the decoded text is immediately discarded.

Measured timings:

Segment size 0.338.0 0.339.0 Patched
0.25 MiB 1.3 ms 898.6 ms 1.3 ms
1.00 MiB 2.3 ms 3,666.0 ms 2.6 ms
4.00 MiB 5.8 ms 14,797.7 ms 6.3 ms
8.00 MiB 8.6 ms 29,130.0 ms 11.3 ms

The cost is approximately linear with segment size.

Production symptoms included:

DEBUG charset_normalizer: Encoding detection: Unable to determine any suitable charset.

and a query taking approximately 22.2 seconds when its results were distributed over four spooled segments compared with approximately 6.7 seconds for a single segment.

2. Successful spooled-result acknowledgements are retried

Spooled acknowledgement requests return HTTP 200 with an empty body by design.

The retry predicate therefore treats a successful acknowledgement as a transient failure. With the default retry configuration the acknowledgement is sent three times and eventually produces:

INFO trino.client: failed after 3 attempts

despite the first request having succeeded.

This also increases acknowledgement traffic and occupies the acknowledgement worker pool with unnecessary retry backoff.

Additional affected operation

The same empty-response condition is also applied to _head.

A successful HEAD request normally has no response body, so an HTTP 200 heartbeat response can also satisfy this retry condition despite being successful.

Regression

The behaviour was introduced by commit f8b1203 ("Retry on HTTP 200 with empty response body"), which fixed #596.

0.338.0 does not exhibit the spooling regressions.

A local reproduction gives:

Version Acknowledgement sent once 4 MiB fetch #596 empty-statement-response retry
0.338.0 PASS PASS FAIL
0.339.0 FAIL FAIL PASS
0.339.0 + proposed patch PASS PASS PASS

Proposed fix

Scope the HTTP 200 empty-body retry condition to statement-protocol requests rather than the shared HTTP wrappers.

For example:

  • _get, _post, _delete, and _head retain the generic retry conditions for 429/502/503/504.
  • Statement GET and POST use statement-specific wrapped request functions which additionally retry an HTTP 200 empty body.
  • Test emptiness using response.content, rather than response.text, so determining whether the body is empty cannot trigger charset detection.

This preserves the fix for #596 while preventing the statement-specific condition from affecting spooled binary downloads, acknowledgements, HEAD requests, or other transport operations.

Workaround

Pin:

trino==0.338.0

This avoids the regression, at the cost of not including the HTTP 200 empty-statement-response retry introduced for #596.

Trino Python client version

0.339.0

Are you willing to submit PR?

Yes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the shared _get, _post, _delete, and _head request wrappers, then trace the statement GET/POST and spooled-result request paths. Confirm the behavior for empty statement responses, binary spooled segments, acknowledgements, and HEAD requests; done means statement retries remain while other successful empty responses are not retried.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.