HDFGroup / HDFGroup/h5pyd

Timeout for file flush

Open
#268 1 comment 0 reactions 1 assignee Claimed by @jreadey View on GitHub
Dominant language
Python
Stars
124
Forks
46
Avg merge
1h 24m
Merged PRs (30d)
6

Description

If connection to HSDS (or the internet) is lost during a file flush, the process hangs indefinitely. There should be some timeout logic, preferably set by a parameter to the flush function, that allows the HTTP request to stop after a certain amount of time.

On a device, like an IoT device, pulling the plug on the network connection mid-flush results in this sample script below hanging indefinitely.

```
import h5pyd

path = '/test/timeout.h5'

with h5pyd.File(path, 'w') as f:
dset = f.create_dataset('data', (1000000000,), dtype='i')
print('Start flush PUT')
f.flush()
print('Successful flush PUT')
```
Here is a full traceback after KeyboardInterrupt:
```
Start flush PUT
^CTraceback (most recent call last):
File "/home/user/repo/test_timeout.py", line 8, in
f.flush()
File "/home/user/.local/lib/python3.10/site-packages/h5pyd/_hl/files.py", line 775, in flush
rsp = self.PUT(req, body=body)
File "/home/user/.local/lib/python3.10/site-packages/h5pyd/_hl/base.py", line 1017, in PUT
rsp = self._id._http_conn.PUT(req, body=body, params=params, format=format)
File "/home/user/.local/lib/python3.10/site-packages/h5pyd/_hl/httpconn.py", line 580, in PUT
rsp = s.put(
File "/home/user/.local/lib/python3.10/site-packages/requests/sessions.py", line 649, in put
return self.request("PUT", url, data=data, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/user/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/home/user/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/home/user/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "/home/user/.local/lib/python3.10/site-packages/urllib3/connection.py", line 565, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.10/ssl.py", line 1274, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.10/ssl.py", line 1130, in read
return self._sslobj.read(len, buffer)
KeyboardInterrupt
```

Since you are using the requests library, which as far as I understand already have a timeout parameter, which you can add in h5pyd/_hl/httpconn.py", line 580, in PUT, rsp = s.put(, you can hopefully add timeout as a parameter in file.flush() and pass it to the put request.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.