Netflix / Netflix/metaflow

Optimize remote step detection for Run.code in the Client API

Open
#3,003 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
10.3k
Forks
1.4k
Avg merge
3d 1h
Merged PRs (30d)
9

Description

๐Ÿ” Issue Description

๐Ÿ“Œ Issue Type

  • Bug
  • Feature Request
  • Enhancement
  • Documentation
  • Refactor
  • Other (please specify)

๐Ÿ“ Description

Optimize remote step detection for Run.code in the Client API

Currently, accessing the code property on a Run object (in metaflow/client/core.py) iterates through all steps in the run sequentially to check if any step has a remote code package available.

  • What is happening? The current implementation uses a loop that runs through all steps in reverse order. In the worst-case scenarioโ€”where only the first step executes remotely and the rest execute locallyโ€”this is highly inefficient as it makes multiple sequential checks before finally finding the code package.
  • What should happen instead? There should be a more optimized way of figuring out if a run has remote steps (and thus a code package) available without sequential iteration.
  • Why is this needed? As noted by a TODO comment in metaflow/client/core.py, this lookup can be quite slow and impacts performance when using the Client API for runs with many local steps.

๐ŸŽฏ Proposed Solution (Optional but Encouraged)

  • High-level approach: Implement an optimized method to check for remote steps. This might involve updating the metadata service to return a flag or caching the remote availability at the run level.
  • Relevant modules/files: metaflow/client/core.py (around line 2216 in the code property).
  • Potential edge cases: Ensuring backward compatibility for older runs that might not have this optimized flag available in the metadata.

๐Ÿ“Ž Additional Context

  • Reference to code: The TODO comment is located in metaflow/client/core.py inside the @property def code(self): method.

๐Ÿ™‹ Claiming This Issue

To avoid duplicated work:

  • I'm willing to solve this issue by myself
๐Ÿ”” Important

If you check the box above:

  1. Comment below: "Iโ€™d like to work on this."
  2. Wait for maintainer acknowledgment (if required).
  3. Then start working.


๐Ÿ” Issue Description

๐Ÿ“Œ Issue Type

  • Bug
  • Feature Request
  • Enhancement
  • Documentation
  • Refactor
  • Other (please specify)

๐Ÿ“ Description

Sort input flows by foreach index in Inputs class for join steps

In metaflow/datastore/inputs.py, the Inputs class is used to provide access to input flows during a join step (e.g., after a foreach split).

  • What is happening? The init method simply converts the provided flows to a list (self.flows = list(flows)). This can lead to a non-deterministic or unordered list of inputs.
  • What should happen instead? The inputs should be explicitly sorted by their foreach index.
  • Why is this needed? Without sorting, it becomes difficult to predictably access branches by index (e.g., inputs[0], inputs[1]), which is a common requirement when joining after a foreach split.

๐ŸŽฏ Proposed Solution

  • High-level approach: Modify the initialization of the Inputs class to sort the incoming flows iterable based on the underlying foreach index attribute before storing it in self.flows.
  • Relevant modules/files: metaflow/datastore/inputs.py (around line 9).
  • Potential edge cases: Ensuring this sorting handles standard non-foreach joins gracefully (where the index might not exist or be relevant in the same way).

๐Ÿ“Ž Additional Context

  • Reference to code: The TODO comment "TODO sort by foreach index" is located in metaflow/datastore/inputs.py.

๐Ÿ™‹ Claiming This Issue

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up โ€” it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the Run.code property in metaflow/client/core.py around line 2216 and the Inputs class in metaflow/datastore/inputs.py. The issue body contains two unrelated proposals, so confirm which behavior is in scope before investigating the TODOs and relevant tests. Done means the selected lookup or input ordering is optimized and covered without breaking older runs or non-foreach joins.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.