ooni / ooni/probe

engine: submit compressed measurements

Open
#2,014 0 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

engine feature
Dominant language
No language data
Stars
919
Forks
150
PR merge metrics
No merged PRs in 30d

Description

This issue is about submitting compressed measurements. We're going to start from compressing individual measurements using the stdlib's compress/gzip package. We'll close this issue when we've implemented compression.

This issue is a follow up to https://github.com/ooni/probe/issues/1802.

We discussed this matter in today's backend meeting with @FedericoCeratto and @hellais. Here are notes from the meeting regarding what we think we'll need to do to make this happen:

  1. the probe needs to set a header to indicate the measurement is compressed;
  2. it seems python's gzip implementation could use a context manager to limit the used memory;
  3. the ooni/api does not touch the incoming measurement in any way and just stores it to disk where the upload and the fastpath will pick it up and, respectively, save it, process it and insert it into the fastpath table;
  4. so the ooni/api can just receive the compressed measurement and put it into a can;
  5. we need to use the filename as metadata to tell the rest of the backend the postcan is compressed;
  6. the fastpath needs to know how to handle compressed measurements.

See also https://github.com/ooni/backend/issues/561.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.