pypi / pypi/linehaul-cloud-function
linehaul parser for `download` should accept a `size` parameter
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 24
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 10
Description
In order to annotate how many bytes were transferred in a given download request, the upstream log needs to emit bytes that would be parsed here.
It appears that with parse_all=True, the parser format would need to be updated to allow for the existence of the new field. Does that mean duplicating MESSAGE_V3 to a MESSAGE_V4 with the added param at the end?
https://github.com/pypi/linehaul-cloud-function/blob/8b8ed9db4ed946722d011c7d5ffe08a03ab5f942/linehaul/events/parser.py#L121-L124
Will need to add resp.body_bytes_written to https://github.com/pypi/infra/blob/d6702898cbed406cc431a12b80270831ef292557/terraform/file-hosting/vcl/files.vcl#L589 to emit the value when this library can parse for it.
Will likely need coordination with BigQuery team to add column when the data is correctly parsed.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with linehaul/events/parser.py around lines 121-124 and inspect how parse_all handles MESSAGE_V3 and optional fields. Then review the referenced infra terraform/file-hosting/vcl/files.vcl location for the upstream byte emission. Done means download records parse the size value reliably, with the required upstream and BigQuery coordination completed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python, terraform
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100