influxdata / influxdata/datafusion-udf-wasm

Python guest Justfile: validate checksums for already-present case

Open
#84 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
20
Forks
3
Avg merge
6h 20m
Merged PRs (30d)
24

Description

# What
The Python guest Justfile has an optimization that avoids downloading the CPython WASM build and the WASI SDK again if they are already downloaded:

https://github.com/influxdata/datafusion-udf-wasm/blob/a5b9a1d2c2a887aae99394da36e0462f13f08081/guests/python/Justfile#L33-L37

https://github.com/influxdata/datafusion-udf-wasm/blob/a5b9a1d2c2a887aae99394da36e0462f13f08081/guests/python/Justfile#L90-L94

That should not only check presence but also the checksum of the already-downloaded files and should re-download if the checksums mismatch.

# Why
Whenever we upgrade the CPython version or the WASI SDK, developers will locally still have an outdated copy and without `just clean` they will not use the new versions. This isn't an issue for our CI though.

Note that this is NOT a security issue since the checksum was validated when the data was originally downloaded, it's just that it doesn't automatically upgrade the version of a local development setup.

# How
In the if-branch, do something similar to these checks

https://github.com/influxdata/datafusion-udf-wasm/blob/a5b9a1d2c2a887aae99394da36e0462f13f08081/guests/python/Justfile#L59-L60

https://github.com/influxdata/datafusion-udf-wasm/blob/a5b9a1d2c2a887aae99394da36e0462f13f08081/guests/python/Justfile#L106

and if they fail, wipe the downloaded data and re-download.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in guests/python/Justfile, especially the already-present branches around lines 33–37 and 90–94. Compare them with the checksum checks around lines 59–60 and 106, then verify that a checksum mismatch removes the downloaded data and triggers a fresh download; the existing-download case should also retain the current optimization.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, wasm
Domain
build-system
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.