Optimize remote step detection for Run.code in the Client API
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10.3k
- Forks
- 1.4k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 9
Description
๐ Issue Description
๐ Issue Type
- Bug
- Feature Request
- Enhancement
- Documentation
- Refactor
- Other (please specify)
๐ Description
Optimize remote step detection for Run.code in the Client API
Currently, accessing the code property on a Run object (in metaflow/client/core.py) iterates through all steps in the run sequentially to check if any step has a remote code package available.
- What is happening? The current implementation uses a loop that runs through all steps in reverse order. In the worst-case scenarioโwhere only the first step executes remotely and the rest execute locallyโthis is highly inefficient as it makes multiple sequential checks before finally finding the code package.
- What should happen instead? There should be a more optimized way of figuring out if a run has remote steps (and thus a code package) available without sequential iteration.
- Why is this needed? As noted by a
TODOcomment in metaflow/client/core.py, this lookup can be quite slow and impacts performance when using the Client API for runs with many local steps.
๐ฏ Proposed Solution (Optional but Encouraged)
- High-level approach: Implement an optimized method to check for remote steps. This might involve updating the metadata service to return a flag or caching the remote availability at the run level.
- Relevant modules/files: metaflow/client/core.py (around line 2216 in the code property).
- Potential edge cases: Ensuring backward compatibility for older runs that might not have this optimized flag available in the metadata.
๐ Additional Context
- Reference to code: The
TODOcomment is located in metaflow/client/core.py inside the@property def code(self):method.
๐ Claiming This Issue
To avoid duplicated work:
- I'm willing to solve this issue by myself
๐ Important
If you check the box above:
- Comment below: "Iโd like to work on this."
- Wait for maintainer acknowledgment (if required).
- Then start working.
๐ Issue Description
๐ Issue Type
- Bug
- Feature Request
- Enhancement
- Documentation
- Refactor
- Other (please specify)
๐ Description
Sort input flows by foreach index in Inputs class for join steps
In metaflow/datastore/inputs.py, the Inputs class is used to provide access to input flows during a join step (e.g., after a foreach split).
- What is happening? The init method simply converts the provided flows to a list (
self.flows = list(flows)). This can lead to a non-deterministic or unordered list of inputs. - What should happen instead? The inputs should be explicitly sorted by their foreach index.
- Why is this needed? Without sorting, it becomes difficult to predictably access branches by index (e.g.,
inputs[0],inputs[1]), which is a common requirement when joining after a foreach split.
๐ฏ Proposed Solution
- High-level approach: Modify the initialization of the Inputs class to sort the incoming flows iterable based on the underlying foreach index attribute before storing it in
self.flows. - Relevant modules/files: metaflow/datastore/inputs.py (around line 9).
- Potential edge cases: Ensuring this sorting handles standard non-foreach joins gracefully (where the index might not exist or be relevant in the same way).
๐ Additional Context
- Reference to code: The
TODOcomment "TODO sort by foreach index" is located in metaflow/datastore/inputs.py.
๐ Claiming This Issue
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up โ it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the Run.code property in metaflow/client/core.py around line 2216 and the Inputs class in metaflow/datastore/inputs.py. The issue body contains two unrelated proposals, so confirm which behavior is in scope before investigating the TODOs and relevant tests. Done means the selected lookup or input ordering is optimized and covered without breaking older runs or non-foreach joins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100