aws / aws/sagemaker-python-sdk
Ignored IAM role passed to Pipeline Steps breaks Local Mode
- 主要言語
- Python
- スター
- 2.3k
- フォーク
- 1.3k
- 平均マージ
- 1日 22時間
- マージ済み PR(30日)
- 35
説明
**Describe the feature you'd like**
In non-local mode, I can pass a role to each step of my SageMaker Pipeline and this role will be assumed in that step. Hence, I can use different roles for different steps. This helps scoping permissions narrowly (for security) as well as working around the IAM limit of 10 managed policies per role and 10240 bytes character limit for inline policies, which are a problem once the role that is assumed in the pipeline needs too many permissions.
```python
# processor step
processor_select_delivery_periods = sagemaker.processing.Processor(
entrypoint=["bash", "processing.sh"],
role= role_arn_for_processing_job, # <- this is ignored in local mode
instance_type="ml.m5.large",
instance_count=1,
sagemaker_session=signals.sagemaker.pipeline_session.cache,
)
step_processing_args = processor_select_delivery_periods.run(
arguments=...,
)
step_processing = sagemaker.workflow.steps.ProcessingStep(step_args=step_args, ...)
# training step
training = sagemaker.estimator.Estimator(
image_uri=image_uri,
role=role_arn_for_training_job, # <- this is ignored in local mode
instance_type ='ml.5.large',
instance_count=1,
sagemaker_session=signals.sagemaker.pipeline_session.cache,
)
step_training_args = training.fit(
arguments=...,
)
step_train_model = sagemaker.workflow.steps.TrainingStep(step_args=train_args, ...)
# finally build the pipeline
pipeline = sagemaker.workflow.pipeline.Pipeline(
steps=[
step_processing,
step_train_model,
],
sagemaker_session=... # pass a local session if run locally, otherwise None
)
```
However, the argument `role` seems not respected in local mode. Even worse, not even the IAM role defined at pipeline level seems to be respected
```python
upsert_response = pipeline.upsert(
role_arn='arn:/xxx/yzzc',
description=description or "",
tags=tags,
)
```
The credentials seem to be purely generated from the `sagemaker_session` passed to `Pipeline`:
```python
sagemaker.workflow.pipeline.Pipeline(
name=str(name),
parameters=list(parameters),
steps=[step_a, step_b],
sagemaker_session=sagemaker_session
)
```
From looking at the source code of this SDK, in my understanding, the local mode's docker-compose file (one per step) contains credentials via environment variables (AWS_*) that are then used in the container. Before generating these credentials, one would need use `sagemaker_session` to assume the role passed to a step and then generate the credentials from these roles instead of directly from that session.
**How would this feature be used? Please describe.**
The inconsistency between local and remote execution breaks the highly efficient local developer workflow:
* Use per step roles makes sense from security and IAM limitations for large policies (both managed and unmanaged).
* However, it's practically unusable with local mode, a key ingredient of efficient pipeline development, as the roles are not respected in local execution and a local developer hence needs a role with the union of all permissions from all rules in remote execution, which defeats the purpose of per step roles.
**Describe alternatives you've considered**
* Refrain from using step-specific roles, i.e. one role for all steps, which means I'll face IAM restrictions.
* Splitting my pipeline into multiple pipelines since it contains many steps that are only loosely related. Drawback: Need another orchestration tool on top, seems an overkill.
* Not using local mode anymore: Not really an option, since without local mode, I get very slow in developing pipelines.
**Additional context**
I opened an AWS Support case: 174841854500356.
コントリビューションガイド
調査の方向性
まず、ステップごとの docker-compose セットアップにおけるローカルモードの認証情報生成を追跡し、次に Pipeline.upsert、Processor、Estimator のロールが sagemaker_session を介してどのように渡されるかを調べます。これをリモートパイプラインのパスと比較します。ローカル実行がステップ固有のロールとパイプラインレベルのロールを尊重し、リモート実行にリグレッションを生じさせなければ完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- aws, python
- 領域
- cloud, machine-learning
- issue の種類
- 機能追加
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100