ctengel / ctengel/simpler-objects
Revisit aiohttp vs pycurl client bake-off on target hardware/network
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
The `simpler_objects.client` library is currently built on `pycurl`, chosen over `aiohttp` via the throughput bake-off in #21 (see `upload-behavior-demo/bench_throughput.py`). That bake-off should be revisited under conditions closer to production before the choice is treated as final.
## Why revisit
The bake-off that picked `pycurl` (~1.45x faster upload: pycurl ~150 MiB/s vs aiohttp ~103 MiB/s) ran under conditions that do not match the deployment target:
- **x86 only** — the target hardware is ARM. The relative cost of `aiohttp`'s Python-level streaming vs. libcurl's C path may differ on ARM.
- **localhost only** — the target is real hosts over 1 Gbps Ethernet, which caps throughput at ~119 MiB/s. `pycurl`'s 150 MiB/s would be limited by the link while `aiohttp`'s 103 MiB/s would not — narrowing the gap considerably. If ARM results or aiohttp tuning lift aiohttp past the link cap, the two would effectively tie.
- `aiohttp` was measured with the straightforward `data=open_file` approach. Its upload bottleneck is reading the file in 64 KiB chunks via a thread-pool executor; a custom larger-chunk payload could close much of the gap.
## What to do
- [ ] Re-run `bench_throughput.py` on ARM target hardware.
- [ ] Re-run it between two hosts over a 1 Gbps link (not localhost).
- [ ] Try tuning `aiohttp` — a custom payload with ~1 MiB reads (must still report `Content-Length`; the locator requires it) — and re-measure.
## Possible outcomes
- If `aiohttp` ties or wins under realistic conditions: switch `simpler_objects/client.py` to `aiohttp`, which also restores the native asyncio support #21 asked for.
- If the result is hardware/workload dependent: ship two client libraries (a `pycurl` variant and an `aiohttp` variant) and let callers choose.
- If `pycurl` still clearly wins: keep the current implementation and close this issue.
Related: #21, #26.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.