source-cooperative / source-cooperative/data.source.coop
[Bug] Ranged GET truncates at ~2.3 MB across clients and buckets
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 24
- Forks
- 6
- Avg merge
- 1h 32m
- Merged PRs (30d)
- 1
Description
Clauded, obviously. But I've seen this a few times across different projects I've been working on using Source and am not sure if it's something on my end or Source itself. --Nissim
Description of Bug:
Ranged GET requests through the data proxy return about 2.3 MB and then the connection closes, whatever range was asked for. The cut lands at roughly 5.5 seconds every time, so the byte count drifts with throughput while the duration holds steady. Every client we tried hits it: the AWS CLI, DuckDB httpfs, and curl. It reproduces across buckets.
This breaks cloud-native Parquet reads. A reader fetches one column chunk per request, and any chunk over ~2 MB fails. Small attribute columns come back fine, geometry columns do not.
Steps to Reproduce:
- Ask for a 20 MB range with the AWS CLI, using the documented unauthenticated pattern:
aws s3api get-object \
--endpoint-url https://data.source.coop \
--no-sign-request \
--bucket tristangruppwri \
--key cadastral/Brazil_CAR_AREA_IMOVEL.parquet \
--range bytes=0-20971519 \
chunk.bin
- Repeat with curl to confirm the client is not the variable:
U=https://data.source.coop/tristangruppwri/cadastral/Brazil_CAR_AREA_IMOVEL.parquet
for mb in 1 4 8 32; do
curl -s --max-time 90 -r 0-$((mb*1024*1024-1)) "$U" -o /dev/null \
-w "asked ${mb}MB -> %{size_download} B in %{time_total}s\n"
done
- Repeat against a second bucket, such as
wri-data-lab/trazofields/trazo3-fields/trazo3_brazil_goias_2024.parquet.
Expected Behavior:
A ranged GET returns the bytes it was asked for, or an error naming a limit.
Actual Behavior:
The AWS CLI retried for two minutes, then reported:
2290289 read, but total bytes expected is 20971520
curl shows where the boundary sits:
asked 1MB -> 1048576 B in 3.20s
asked 4MB -> 2376597 B in 5.53s
asked 8MB -> 2870408 B in 5.50s
asked 32MB -> 2882244 B in 5.51s
One megabyte arrives whole. Everything larger stops near 2.3 MB at 5.5 seconds. The same pattern appears on wri-data-lab, so it is not specific to one bucket or one object.
Screenshots:
None. Terminal output is inline above.
Additional Context:
Throughput on the same machine, measured minutes apart:
| Host | Rate |
|---|---|
| speed.cloudflare.com | 63 MB/s |
| github.com | 45 MB/s |
| data.source.coop | 0.33 to 0.43 MB/s |
We have not identified a cause and cannot tell from outside whether this is a proxy timeout, a rate limit, or a routing problem. Testing from another network would settle whether the slow rate is general or particular to our path.
Client versions: aws-cli 2.28.11, DuckDB 1.5.5 with httpfs, curl 8.5.0, on
Ubuntu 24.04.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by running the AWS CLI and curl ranged-GET examples against both listed buckets, then trace the data proxy's range handling and connection lifetime. Done means requests return the complete requested range, or report an explicit documented limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- api, backend, cloud, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100