Azure / Azure/MachineLearningNotebooks

Using ParallelRunConfig and DockerConfig not working

Aberta
#1,979 2 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Jupyter Notebook
Estrelas
4.4k
Forks
2.6k
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

Hello, I am attempting to use the ParallelRunStep for a set of python scripts that I currently run inside a docker container. I am trying to deploy this using an AzureML pipeline that reads from the docker image saved in our ACR. However, the init and run scripts in my parallel script never seem to be able to access the files I have moved into the docker container. Particularly when I try and search for a hydra config file in my init and run scripts, I am unable to find the file, even when I look for it explicitly. I have included a snippet that is based on the example code from this repo: [example of parallel run config](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/parallel-run/tabular-dataset-partition-per-column.ipynb)

Any idea what I am doing wrong? I have included a snippet of what I am trying below

```import os

docker_config = DockerConfiguration(use_docker=True, arguments=docker_args)

environment_name = "my-environment"
environment = Environment(environment_name)
base_image_name = os.getenv("ACR_BASE_IMAGE_NAME")
base_image_tag = os.getenv("ACR_IMAGE_TAG")
environment.docker.base_image = f"{base_image_name}:{base_image_tag}"

environment.docker.base_image_registry.address = f"{acr_name}.azurecr.io"

environment.docker.base_image_registry.username = os.getenv("ACR_USER")
environment.docker.base_image_registry.password = os.getenv("ACR_PASSWORD")
environment.python.user_managed_dependencies = True
environment.docker.enabled=True

run_config = RunConfiguration()
run_config.environment = environment
run_config.docker = docker_config

parallel_run_config = ParallelRunConfig(
source_directory=".",
entry_script="path/to/docker/script_1.py",
compute_target=compute_target,
environment=environment,
node_count=2,
error_threshold=10,
output_action="append_row",
mini_batch_size=1,
logging_level='DEBUG'
)
step_parallel = ParallelRunStep(
name="parallel-step",
parallel_run_config=parallel_run_config,
inputs=my_inputs,
output=output_dir,
arguments=args,
allow_reuse=True,
)

step_parallel._runconfig.docker = docker_config #Tried with and without, does not seem to make a difference

pipeline_steps = StepSequence(steps=[step_parallel])
pipeline_run = Pipeline(workspace=ws, steps=pipeline_steps)

# Submit your pipeline run
submitted_pipeline_run = Experiment(ws, "Azure Pipeline").submit(pipeline_run, regenerate_outputs=True)```

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Start with the linked parallel-run notebook and compare its ParallelRunConfig, DockerConfiguration, and source_directory settings with the provided snippet. Trace path/to/docker/script_1.py and the init and run scripts to determine how they locate files in the image. Done means the scripts can reliably find the expected Hydra configuration when the pipeline runs.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
azure, docker, jupyter-notebook, python
Domínio
devops, machine-learning
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Precisa de esclarecimento
Facilidade para iniciantes
30/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.