ooni / ooni/data

Investigate corrupt file entries inside of buckets

Open
#93 0 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11
Forks
6
Avg merge
4d 11h
Merged PRs (30d)
4

Description

When running the reprocessing of data some corrupt files were encountered. Due to the lack of adequate logging in the temporal cloud interface it's unclear which file exact was affected, but it's for sure inside of the 2024-06-23 bucket and is caused by an empty JSON file, see following stack trace:

{"message":"Input is a zero-length, empty document: line 1 column 1 (char 0)","stackTrace":"  File \"/home/art/repos/ooni/data/oonipipeline/.venv/lib/python3.11/site-packages/temporalio/worker/_activity.py\", line 453, in _run_activity\n    result = await impl.execute_activity(input)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File \"/home/art/repos/ooni/data/oonipipeline/.venv/lib/python3.11/site-packages/temporalio/contrib/opentelemetry.py\", line 280, in execute_activity\n    return await super().execute_activity(input)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File \"/home/art/repos/ooni/data/oonipipeline/.venv/lib/python3.11/site-packages/temporalio/worker/_interceptor.py\", line 119, in execute_activity\n    return await self.next.execute_activity(input)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File \"/home/art/repos/ooni/data/oonipipeline/.venv/lib/python3.11/site-packages/temporalio/worker/_activity.py\", line 711, in execute_activity\n    return await input.fn(*input.args)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File \"/home/art/repos/ooni/data/oonipipeline/src/oonipipeline/temporal/activities/observations.py\", line 229, in make_observations\n    measurement_count = sum(await asyncio.gather(*awaitables))\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n","cause":{"message":"\n\"\"\"\nTraceback (most recent call last):\n  File \"/usr/lib/python3.11/concurrent/futures/process.py\", line 256, in _process_worker\n    r = call_item.fn(*call_item.args, **call_item.kwargs)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/art/repos/ooni/data/oonipipeline/src/oonipipeline/temporal/activities/observations.py\", line 130, in make_observations_for_file_entry_batch\n    measurement_count, failure_count = make_observations_for_file_entry(\n                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/art/repos/ooni/data/oonipipeline/src/oonipipeline/temporal/activities/observations.py\", line 74, in make_observations_for_file_entry\n    for msmt_dict in stream_measurements(\n  File \"/home/art/repos/ooni/data/oonipipeline/src/oonidata/dataclient.py\", line 249, in stream_measurements\n    yield from stream_postcan(body)\n  File \"/home/art/repos/ooni/data/oonipipeline/src/oonidata/dataclient.py\", line 162, in stream_postcan\n    post = orjson.loads(in_file.read())\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\norjson.JSONDecodeError: Input is a zero-length, empty document: line 1 column 1 (char 0)\n\"\"\"","applicationFailureInfo":{"type":"_RemoteTraceback"}},"applicationFailureInfo":{"type":"JSONDecodeError"}}

For the moment in order to avoid blocking the reprocessing we bypass it by checking for any exception in the stream_measurement call

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.