ipfs-shipyard / ipfs-shipyard/py-ipfs-http-client
Concurrent fetches are slower than sequential
- Dominant language
- Python
- Stars
- 685
- Forks
- 200
- PR merge metrics
- No merged PRs in 30d
Description
In [IPWB](https://github.com/oduwsdl/ipwb) (an web archiving replay system) we store archived HTTP headers and payloads in two separate objects for better deduplication of payloads. At the time of replay, we fetch the two objects, combine them, make necessary modifications, and return the response to the client. In our initial implementation we used to fetch the two objects sequentially, but there is an opportunity to perform these fetches concurrently and minimize the response time.
Recently, we experimented with Python threads (https://github.com/oduwsdl/ipwb/pull/425), but the response time became worse than before. A simple ApacheBench test took over 17 seconds with threaded fetches while the same took less than 8 seconds with sequential code.
To ensure that we can reproduce the issue in an isolated environment, I have created a separate repository, [IPFS API Concurrency Test](https://github.com/ibnesayeed/ipfsapi-concurrency-test), to test this behavior. While the #59 suggests that the API is not async-friendly, I am not sure how related it is to this issue. I would assume, with concurrent threads it should not take any longer to finish if not quicker. The worst case time should still be bound by the sequential access time. However, my profiling results are showing the opposite. The more concurrent requests are made, the slower it becomes.
```
======= SUMMARY =======
Data Items: 1
Fetch Attempts: 10
Mean Fetch Time (Sequential): 0.0025439024 seconds/item
Mean Fetch Time (Threaded): 0.0021895885 seconds/item
======= SUMMARY =======
Data Items: 2
Fetch Attempts: 10
Mean Fetch Time (Sequential): 0.0037424684 seconds/item
Mean Fetch Time (Threaded): 0.0042521000 seconds/item
======= SUMMARY =======
Data Items: 3
Fetch Attempts: 10
Mean Fetch Time (Sequential): 0.0027309577 seconds/item
Mean Fetch Time (Threaded): 0.0067453702 seconds/item
======= SUMMARY =======
Data Items: 4
Fetch Attempts: 10
Mean Fetch Time (Sequential): 0.0036935270 seconds/item
Mean Fetch Time (Threaded): 0.0087552428 seconds/item
======= SUMMARY =======
Data Items: 5
Fetch Attempts: 10
Mean Fetch Time (Sequential): 0.0041139285 seconds/item
Mean Fetch Time (Threaded): 0.0102155050 seconds/item
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.