galaxyproject / galaxyproject/pulsar
curl remote transfers have no timeout
- Dominant language
- Python
- Stars
- 46
- Forks
- 62
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 15
Description
If I am reading the code correctly, when Pulsar is configured with remote_transfer and `client.transport.curl` is used, this uses the `get_file()` and `post_file()` functions, but not the `PycurlTransport` class, which has a timeout option defined, although [that timeout option probably does not work the way we would want it to](https://curl.se/libcurl/c/CURLOPT_TIMEOUT.html).
I have a Pulsar server which appears to have had a network interruption at some point. AMQP recovered and subsequent jobs are handled fine, but ones that were preprocessing during the hiccup are just stuck and have written 0 bytes to disk in the many hours since. Preprocessing thread stacks are:
```
Thread 101101 (idle): "[manager=vgp_jetstream2]-[action=preprocess]-[job=57400350]"
get_file (pulsar/client/transport/curl.py:119)
write_to_path (pulsar/client/action_mapper.py:482)
(pulsar/managers/staging/pre.py:20)
_retry_over_time (pulsar/managers/util/retry.py:93)
execute (pulsar/managers/util/retry.py:42)
preprocess (pulsar/managers/staging/pre.py:20)
do_preprocess (pulsar/managers/stateful.py:114)
run (threading.py:917)
run (sentry_sdk/integrations/threading.py:70)
_bootstrap_inner (threading.py:980)
_bootstrap (threading.py:937)
```
I would expect TCP to do something more reasonable here so that part is a mystery to me, but we could probably also allow the configuration of the [low speed limit](https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html) and [low speed time](https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html) options to mitigate these sorts of issues.
Contributor guide
Research direction
Start in pulsar/client/transport/curl.py at get_file() and post_file(), then compare their behavior with the timeout option in PycurlTransport. Trace how remote_transfer config reaches these functions and determine how low-speed limit and low-speed time should be configured. Done means interrupted transfers no longer remain stuck indefinitely and the relevant behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100