engine: submit compressed measurements
Open
Nobody has claimed this yet.
engine
feature
- Dominant language
- No language data
- Stars
- 919
- Forks
- 150
- PR merge metrics
- No merged PRs in 30d
Description
This issue is about submitting compressed measurements. We're going to start from compressing individual measurements using the stdlib's compress/gzip package. We'll close this issue when we've implemented compression.
This issue is a follow up to https://github.com/ooni/probe/issues/1802.
We discussed this matter in today's backend meeting with @FedericoCeratto and @hellais. Here are notes from the meeting regarding what we think we'll need to do to make this happen:
- the probe needs to set a header to indicate the measurement is compressed;
- it seems python's gzip implementation could use a context manager to limit the used memory;
- the ooni/api does not touch the incoming measurement in any way and just stores it to disk where the upload and the fastpath will pick it up and, respectively, save it, process it and insert it into the fastpath table;
- so the ooni/api can just receive the compressed measurement and put it into a can;
- we need to use the filename as metadata to tell the rest of the backend the postcan is compressed;
- the fastpath needs to know how to handle compressed measurements.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.