JuliaAI / JuliaAI/MLFlowClient.jl
Artificact upload for remote instances
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 59
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
As per https://github.com/JuliaAI/MLFlowClient.jl/blob/a31b41a91dc32107fa7dcd6ad5a581c116a4ff9c/src/loggers.jl#L69-72 we currently can't upload artifacts to remote servers
Assumes that artifact_uri is mapped to a local directory.
At the moment, this only works if both MLFlow and the client are running on the same host or they map a directory that leads to the same location over NFS, for example.
Some (untested) code along the lines of:
function logartifact(mlf::GitLabMLFlow, run_id::AbstractString, basefilename::AbstractString, data)
mlflowrun = getrun(mlf, run_id)
artifact_uri = mlflowrun.info.artifact_uri
filepath = joinpath(artifact_uri, basefilename)
try
HTTP.post(url, push!(copy(mlf.headers), "Content-Type" => "multipart/form-data" ),
HTTP.Form(Dict(:file => data)))
catch e
error("Unable to upload $(filepath): $e")
end
filepath
end
Would probably work for this purpose and not introduce new dependencies.
The question remains how to differentiate between local and remote file access, as they happen through different interfaces (the headers are irrelevant for file access for example).
I see three options to approach this and would be interested in making a pull request implementing one of these and testing them against a local mlflow instance and the gitlab end points to confirm they are working (for me).
- Use a different subtype of
AbstractMLFlowto denote remote instances. (Probably not preferred) - Keep using a String and just have an if checking for prefixes of "https://" and use HTTP then.
- Switch to using URIs and use file:// to denote file access and check the prefix in an if statement.
This is simply a question of taste i would leave to (some of) the maintainers.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/loggers.jl around lines 69-72 and inspect logartifact and getrun to understand how artifact_uri is currently handled. Decide with maintainers how local and remote artifact access should be distinguished, then verify the chosen implementation against a local MLflow instance and the GitLab endpoints; done means remote artifact uploads work without breaking local access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- api, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100