Netflix / Netflix/metaflow

Local Storage Card with local K8s

Open
#2,139 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
10.3k
Forks
1.4k
Avg merge
3d 1h
Merged PRs (30d)
9

Description

Hi, I'm currently running a k3s cluster locally, and I've been wanting to test a workflow that has a @card decorator to see some html and see it from the UI that is being served from K8s. The worfllow is ran locally. I am testing this without an s3 instance (that was giving me some issues with locating the /cards content when the UI was asking for them to retrieve from the ui service)

here is my ~/.metaflowconfig/config.json
{
"METAFLOW_DEFAULT_METADATA": "service",
"METAFLOW_SERVICE_URL": "http://localhost:8080/"
}
I've created a /.metaflow directory with no contents and chowned to my user to work

  • the first run i make (Im testing this workflow ) has no .metaflow directory.
    This saves metadata about the run on /.metadata (a root folder owned by my user - not ideal but i want to mount this onto the ui service pod so that it can serve the contents later, i can do it from any folder but I haven't figured out how to have a consistent location for this yet
  • second run: there is a .metaflow/ directory all of the sudden, and my data is split. in /.metaflow directory where i expect things to be has only a NBFlow sub directory, and my local ./.metaflow directory has the mf.cards I'm missing on the other one.

Now every subsequent run after this, metaflow defaults everything to my local ./.metaflow directory, but what i need is
I want to figure out a way to have a consistent way of setting my local storage directory and make sure the cards content also land here.
is there some combination of env variables or config i can add to my config.json?
also, if I were to instead just go back to my s3 setup, what configurations should i look at? I haven't had luck finding sources on this.

I must be something pretty simple here but I'm very new to metaflow, any help would be appreciated!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the linked random_forest flow.py workflow with the shown ~/.metaflowconfig/config.json and inspect where /.metadata, ./.metaflow, and /.metaflow/NBFlow are created. Trace how mf.cards content is located by the UI service when running on local K8s. Done means the issue documents or implements a consistent local storage location for metadata and cards, with the relevant S3 configuration clarified if needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.