Azure / Azure/MachineLearningNotebooks

Using ParallelRunConfig and DockerConfig not working

未关闭
#1,979 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Jupyter Notebook
星标
4.4k
派生
2.6k
PR 合并指标
30 天内没有已合并 PR

描述

Hello, I am attempting to use the ParallelRunStep for a set of python scripts that I currently run inside a docker container. I am trying to deploy this using an AzureML pipeline that reads from the docker image saved in our ACR. However, the init and run scripts in my parallel script never seem to be able to access the files I have moved into the docker container. Particularly when I try and search for a hydra config file in my init and run scripts, I am unable to find the file, even when I look for it explicitly. I have included a snippet that is based on the example code from this repo: [example of parallel run config](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/parallel-run/tabular-dataset-partition-per-column.ipynb)

Any idea what I am doing wrong? I have included a snippet of what I am trying below

```import os

docker_config = DockerConfiguration(use_docker=True, arguments=docker_args)

environment_name = "my-environment"
environment = Environment(environment_name)
base_image_name = os.getenv("ACR_BASE_IMAGE_NAME")
base_image_tag = os.getenv("ACR_IMAGE_TAG")
environment.docker.base_image = f"{base_image_name}:{base_image_tag}"

environment.docker.base_image_registry.address = f"{acr_name}.azurecr.io"

environment.docker.base_image_registry.username = os.getenv("ACR_USER")
environment.docker.base_image_registry.password = os.getenv("ACR_PASSWORD")
environment.python.user_managed_dependencies = True
environment.docker.enabled=True

run_config = RunConfiguration()
run_config.environment = environment
run_config.docker = docker_config

parallel_run_config = ParallelRunConfig(
source_directory=".",
entry_script="path/to/docker/script_1.py",
compute_target=compute_target,
environment=environment,
node_count=2,
error_threshold=10,
output_action="append_row",
mini_batch_size=1,
logging_level='DEBUG'
)
step_parallel = ParallelRunStep(
name="parallel-step",
parallel_run_config=parallel_run_config,
inputs=my_inputs,
output=output_dir,
arguments=args,
allow_reuse=True,
)

step_parallel._runconfig.docker = docker_config #Tried with and without, does not seem to make a difference

pipeline_steps = StepSequence(steps=[step_parallel])
pipeline_run = Pipeline(workspace=ws, steps=pipeline_steps)

# Submit your pipeline run
submitted_pipeline_run = Experiment(ws, "Azure Pipeline").submit(pipeline_run, regenerate_outputs=True)```

贡献指南

这个仓库没有索引到贡献指南

调研方向

从链接的并行运行 notebook 开始,将其中的 ParallelRunConfig、DockerConfiguration 和 source_directory 设置与提供的代码片段进行比较。跟踪 path/to/docker/script_1.py 以及 init 和 run 脚本,以确定它们如何定位 image 中的文件。当 pipeline 运行时,脚本能够可靠地找到预期的 Hydra 配置,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
azure, docker, jupyter-notebook, python
领域
devops, machine-learning
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
30/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。