twitter / twitter/communitynotes
Wrong content-length header for datasets
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 350
- Avg merge
- 6m
- Merged PRs (30d)
- 2
Description
Describe the bug
When downloading the datasets from https://x.com/i/communitynotes/download-data using wget, it hangs, not receiving more data, because the content-length header is too big (566M) for the file being served (185M).
To Reproduce
wget https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
wget https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
--2024-09-07 13:43:04-- https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
Resolving ton.twimg.com (ton.twimg.com)... 2606:2800:233:7ee2:97c:ab4c:6c70:be36, 152.199.21.140
Connecting to ton.twimg.com (ton.twimg.com)|2606:2800:233:7ee2:97c:ab4c:6c70:be36|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 593120688 (566M) [text/tab-separated-values]
Saving to: ‘notes-00000.tsv’
notes-00000.tsv 32%[=======> ] 185.19M --.-KB/s eta 2m 58s
Expected behavior
The content-length header should be set to the file size.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the listed wget command against the dataset URL and compare the Content-Length header with the bytes actually served. The issue is done when the header matches the downloaded file size; the payload does not identify a repository file or entry point to change.
Written by the indexing model from the issue text.
Assessment
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100