Azure / Azure/MachineLearningNotebooks

AML Pipelines: add __str__() implementation or get_env_variable_name() to Dataset

未关闭
#1,531 0 条评论 2 个 reaction 已指派 0 人 在 GitHub 查看
ADO Data4ML Pipelines product-question
主要语言
Jupyter Notebook
星标
4.4k
派生
2.6k
PR 合并指标
30 天内没有已合并 PR

描述

In the `PipelineData` class, there is a `get_env_variable_name()` method which returns the name of the environment variable for this dataset (e.g. `"$AZUREML_DATAREFERENCE_my_pipelinedata"`). This is actually also the `__str__` implementation for `PipelineData`, so that it can easily be used in string formatting to pass it as an argument to a pipeline step, even if you use a custom format for arguments (such as the one of [hydra.cc](https://hydra.cc/), as also mentioned in https://github.com/MicrosoftDocs/azure-docs/issues/66599):

```python
my_pipelinedata = PipelineData("my_pipelinedata", datastore=datastore, is_directory=True)
train_step = PythonScriptStep(
script_name="train.py",
arguments=[
f"dataset.path={my_pipelinedata}"
]
# ...
)
```

Unfortunately, this is not the case if you want to consume a `Dataset`. The `DatasetConsumptionConfig` class does not provide a `get_env_variable_name()` method, and it doesn't have a custom `__str__()` implementation either. So, if you want to use it in string formatting for arguments, you have to manually construct the name of the environment variable, which is just a bit more code, but inconsistent with how it is done for `PipelineData`:

```python
def as_env_variable(dataset):
return f"${dataset.name}"

my_dataset = (
Dataset.get_by_name(workspace, name="my_dataset")
.as_named_input("my_dataset")
.as_mount()
)
train_step = PythonScriptStep(
script_name="train.py",
arguments=[
f"dataset1.path={as_env_variable(my_dataset)}",
f"dataset2.path={my_pipelinedata}"
]
# ...
)
```

So adding a `__str__()` implementation to the `DatasetConsumptionConfig` class, or at least a `get_env_variable_name()` function would make such code more consistent.

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先定位 DatasetConsumptionConfig 的 Python SDK 定义,并将其与 PipelineData.get_env_variable_name() 进行比较。确认 DatasetConsumptionConfig.name 如何用于环境变量引用,然后使字符串格式化或 accessor 的行为保持一致,并为请求的用法添加覆盖测试。

由索引模型根据 Issue 内容生成。

评估

技术栈
azure, python
领域
machine-learning
Issue 类型
功能
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。