galaxyproject / galaxyproject/pulsar

curl remote transfers have no timeout

Open
#362 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46
Forks
62
Avg merge
3d 9h
Merged PRs (30d)
15

Description

If I am reading the code correctly, when Pulsar is configured with remote_transfer and `client.transport.curl` is used, this uses the `get_file()` and `post_file()` functions, but not the `PycurlTransport` class, which has a timeout option defined, although [that timeout option probably does not work the way we would want it to](https://curl.se/libcurl/c/CURLOPT_TIMEOUT.html).

I have a Pulsar server which appears to have had a network interruption at some point. AMQP recovered and subsequent jobs are handled fine, but ones that were preprocessing during the hiccup are just stuck and have written 0 bytes to disk in the many hours since. Preprocessing thread stacks are:

```
Thread 101101 (idle): "[manager=vgp_jetstream2]-[action=preprocess]-[job=57400350]"
get_file (pulsar/client/transport/curl.py:119)
write_to_path (pulsar/client/action_mapper.py:482)
(pulsar/managers/staging/pre.py:20)
_retry_over_time (pulsar/managers/util/retry.py:93)
execute (pulsar/managers/util/retry.py:42)
preprocess (pulsar/managers/staging/pre.py:20)
do_preprocess (pulsar/managers/stateful.py:114)
run (threading.py:917)
run (sentry_sdk/integrations/threading.py:70)
_bootstrap_inner (threading.py:980)
_bootstrap (threading.py:937)
```

I would expect TCP to do something more reasonable here so that part is a mystery to me, but we could probably also allow the configuration of the [low speed limit](https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html) and [low speed time](https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html) options to mitigate these sorts of issues.

Contributor guide

Open the contributing guide

Research direction

Start in pulsar/client/transport/curl.py at get_file() and post_file(), then compare their behavior with the timeout option in PycurlTransport. Trace how remote_transfer config reaches these functions and determine how low-speed limit and low-speed time should be configured. Done means interrupted transfers no longer remain stuck indefinitely and the relevant behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.