Investigate missing jsonl files from s3 bucket
Open
@FedericoCeratto is already working on this.
Since Sep 13, 2021.
bug
pipeline
- Dominant language
- Python
- Stars
- 61
- Forks
- 33
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 3
Description
It was brought up by a community member that certain raw JSONL files are missing from 2020.
For example looking at the autoclavedlookup table I see we have this measurement entry:
8325347 36787 96223 2989 2020-08-01/20200801T003343Z-BY-AS6697-web_connectivity-20200801T003344Z_AS6697_JwVqBeEB4cqXDHwlgTdigXaFA1XiUkplDhRLucQ7YNkrdTWzZy-0.2.0-probe.json 2020-08-01/web_connectivity.02.tar.lz4 20200801T003344Z_AS6697_JwVqBeEB4cqXDHwlgTdigXaFA1XiUkplDhRLucQ7YNkrdTWzZy http://www.metacafe.com/
and it's present in
aws s3 --no-sign-request ls --recursive s3://ooni-data-eu-fra/autoclaved/jsonl.tar.lz4/2020-08-01/web_connectivity.02.tar.lz4
2020-08-02 03:25:27 13981487 autoclaved/jsonl.tar.lz4/2020-08-01/web_connectivity.02.tar.lz4
Yet when I try to look it up in the jsonl directory tree I get nothing:
% aws s3 --no-sign-request ls --recursive s3://ooni-data-eu-fra/jsonl/webconnectivity/BY/20200801/
~ %
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.