aws / aws/sagemaker-python-sdk

Ignored IAM role passed to Pipeline Steps breaks Local Mode

Abierto
#5,031 1 comentario 0 reacciones 0 asignados Ver en GitHub
component: local mode component: pysdk-team type: feature request
Lenguaje dominante
Python
Estrellas
2.3k
Forks
1.3k
Merge medio
1 d 22 h
PR fusionados (30 d)
35

Descripción

**Describe the feature you'd like**

In non-local mode, I can pass a role to each step of my SageMaker Pipeline and this role will be assumed in that step. Hence, I can use different roles for different steps. This helps scoping permissions narrowly (for security) as well as working around the IAM limit of 10 managed policies per role and 10240 bytes character limit for inline policies, which are a problem once the role that is assumed in the pipeline needs too many permissions.

```python
# processor step
processor_select_delivery_periods = sagemaker.processing.Processor(
entrypoint=["bash", "processing.sh"],
role= role_arn_for_processing_job, # <- this is ignored in local mode
instance_type="ml.m5.large",
instance_count=1,
sagemaker_session=signals.sagemaker.pipeline_session.cache,
)

step_processing_args = processor_select_delivery_periods.run(
arguments=...,
)

step_processing = sagemaker.workflow.steps.ProcessingStep(step_args=step_args, ...)

# training step
training = sagemaker.estimator.Estimator(
image_uri=image_uri,
role=role_arn_for_training_job, # <- this is ignored in local mode
instance_type ='ml.5.large',
instance_count=1,
sagemaker_session=signals.sagemaker.pipeline_session.cache,
)

step_training_args = training.fit(
arguments=...,
)

step_train_model = sagemaker.workflow.steps.TrainingStep(step_args=train_args, ...)

# finally build the pipeline
pipeline = sagemaker.workflow.pipeline.Pipeline(
steps=[
step_processing,
step_train_model,
],
sagemaker_session=... # pass a local session if run locally, otherwise None

)
```

However, the argument `role` seems not respected in local mode. Even worse, not even the IAM role defined at pipeline level seems to be respected

```python
upsert_response = pipeline.upsert(
role_arn='arn:/xxx/yzzc',
description=description or "",
tags=tags,
)
```
The credentials seem to be purely generated from the `sagemaker_session` passed to `Pipeline`:

```python
sagemaker.workflow.pipeline.Pipeline(
name=str(name),
parameters=list(parameters),
steps=[step_a, step_b],
sagemaker_session=sagemaker_session
)
```

From looking at the source code of this SDK, in my understanding, the local mode's docker-compose file (one per step) contains credentials via environment variables (AWS_*) that are then used in the container. Before generating these credentials, one would need use `sagemaker_session` to assume the role passed to a step and then generate the credentials from these roles instead of directly from that session.

**How would this feature be used? Please describe.**

The inconsistency between local and remote execution breaks the highly efficient local developer workflow:
* Use per step roles makes sense from security and IAM limitations for large policies (both managed and unmanaged).
* However, it's practically unusable with local mode, a key ingredient of efficient pipeline development, as the roles are not respected in local execution and a local developer hence needs a role with the union of all permissions from all rules in remote execution, which defeats the purpose of per step roles.

**Describe alternatives you've considered**

* Refrain from using step-specific roles, i.e. one role for all steps, which means I'll face IAM restrictions.
* Splitting my pipeline into multiple pipelines since it contains many steps that are only loosely related. Drawback: Need another orchestration tool on top, seems an overkill.
* Not using local mode anymore: Not really an option, since without local mode, I get very slow in developing pipelines.

**Additional context**

I opened an AWS Support case: 174841854500356.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza rastreando la generación de credenciales en el modo local dentro de la configuración de docker-compose por paso y, después, inspecciona cómo se pasan los roles de Pipeline.upsert, Processor y Estimator a través de sagemaker_session. Compara esto con la ruta de pipeline remoto. Se considera completado cuando la ejecución local respeta los roles específicos de cada paso y el rol a nivel de pipeline sin introducir regresiones en la ejecución remota.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
aws, python
Área
cloud, machine-learning
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.