Skip to content

Revisit aiohttp vs pycurl client bake-off on target hardware/network #62

Open

Description

@ctengel

The simpler_objects.client library is currently built on pycurl, chosen over aiohttp via the throughput bake-off in #21 (see upload-behavior-demo/bench_throughput.py). That bake-off should be revisited under conditions closer to production before the choice is treated as final.

Why revisit

The bake-off that picked pycurl (~1.45x faster upload: pycurl ~150 MiB/s vs aiohttp ~103 MiB/s) ran under conditions that do not match the deployment target:

  • x86 only — the target hardware is ARM. The relative cost of aiohttp's Python-level streaming vs. libcurl's C path may differ on ARM.
  • localhost only — the target is real hosts over 1 Gbps Ethernet, which caps throughput at ~119 MiB/s. pycurl's 150 MiB/s would be limited by the link while aiohttp's 103 MiB/s would not — narrowing the gap considerably. If ARM results or aiohttp tuning lift aiohttp past the link cap, the two would effectively tie.
  • aiohttp was measured with the straightforward data=open_file approach. Its upload bottleneck is reading the file in 64 KiB chunks via a thread-pool executor; a custom larger-chunk payload could close much of the gap.

What to do

  • Re-run bench_throughput.py on ARM target hardware.
  • Re-run it between two hosts over a 1 Gbps link (not localhost).
  • Try tuning aiohttp — a custom payload with ~1 MiB reads (must still report Content-Length; the locator requires it) — and re-measure.

Possible outcomes

  • If aiohttp ties or wins under realistic conditions: switch simpler_objects/client.py to aiohttp, which also restores the native asyncio support client library #21 asked for.
  • If the result is hardware/workload dependent: ship two client libraries (a pycurl variant and an aiohttp variant) and let callers choose.
  • If pycurl still clearly wins: keep the current implementation and close this issue.

Related: #21, #26.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions