ctengel / ctengel/simpler-objects

Revisit aiohttp vs pycurl client bake-off on target hardware/network

Open
#62 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

The `simpler_objects.client` library is currently built on `pycurl`, chosen over `aiohttp` via the throughput bake-off in #21 (see `upload-behavior-demo/bench_throughput.py`). That bake-off should be revisited under conditions closer to production before the choice is treated as final.

## Why revisit

The bake-off that picked `pycurl` (~1.45x faster upload: pycurl ~150 MiB/s vs aiohttp ~103 MiB/s) ran under conditions that do not match the deployment target:

- **x86 only** — the target hardware is ARM. The relative cost of `aiohttp`'s Python-level streaming vs. libcurl's C path may differ on ARM.
- **localhost only** — the target is real hosts over 1 Gbps Ethernet, which caps throughput at ~119 MiB/s. `pycurl`'s 150 MiB/s would be limited by the link while `aiohttp`'s 103 MiB/s would not — narrowing the gap considerably. If ARM results or aiohttp tuning lift aiohttp past the link cap, the two would effectively tie.
- `aiohttp` was measured with the straightforward `data=open_file` approach. Its upload bottleneck is reading the file in 64 KiB chunks via a thread-pool executor; a custom larger-chunk payload could close much of the gap.

## What to do

- [ ] Re-run `bench_throughput.py` on ARM target hardware.
- [ ] Re-run it between two hosts over a 1 Gbps link (not localhost).
- [ ] Try tuning `aiohttp` — a custom payload with ~1 MiB reads (must still report `Content-Length`; the locator requires it) — and re-measure.

## Possible outcomes

- If `aiohttp` ties or wins under realistic conditions: switch `simpler_objects/client.py` to `aiohttp`, which also restores the native asyncio support #21 asked for.
- If the result is hardware/workload dependent: ship two client libraries (a `pycurl` variant and an `aiohttp` variant) and let callers choose.
- If `pycurl` still clearly wins: keep the current implementation and close this issue.

Related: #21, #26.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.