influxdata / influxdata/datafusion-udf-wasm

Python guest: re-fetch Python deps if `requirements.txt` changes

Open
#132 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
20
Forks
3
Avg merge
6h 20m
Merged PRs (30d)
24

Description

# What
If [`requirements.txt`](https://github.com/influxdata/datafusion-udf-wasm/blob/9e3d58a95f8fad07d9476f34b9dca878dd11c117/guests/python/requirements.txt) changes, we should re-fetch the Python dependencies.

# Why
If one adds or updates dependencies, developers should get the newest version too. This is NOT an issue in CI since we don't cache that path. This is especially relevant once #104 is implemented.

# How
We deliberately don't re-call `pip` every time we build the guest, since `pip` needs network IO and is generally not the fastest:

https://github.com/influxdata/datafusion-udf-wasm/blob/9e3d58a95f8fad07d9476f34b9dca878dd11c117/guests/python/Justfile#L128-L134

However we should probably write a fingerprint file or something and check if that fingerprint is out-of-date.

Contributor guide

Open the contributing guide

Research direction

Start with guests/python/requirements.txt and the dependency-fetching section of guests/python/Justfile around lines 128-134. Trace the guest build flow and inspect how dependencies are currently cached; done means changes to requirements.txt trigger a new dependency fetch while unchanged requirements avoid unnecessary pip network access.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.