aws / aws/sagemaker-python-sdk
ParamValidationError thrown when setting parallelism_config in pipeline.upsert, pipeline.create etc.
- Lenguaje dominante
- Python
- Estrellas
- 2.3k
- Forks
- 1.3k
- Merge medio
- 1 d 22 h
- PR fusionados (30 d)
- 35
Descripción
# Quick version
## The Problem
- When calling `pipeline.upsert` ([code](https://github.com/qidewenwhen/sagemaker-python-sdk/blob/57c2b941d7bb712f2fe1870381e5ff2bc809ee66/src/sagemaker/workflow/pipeline.py)), if you pass in a`parallelism_config` value with the correct type of ParallelIismConfiguration like this:
```
pipeline.upsert(
config.sm_role,
parallelism_config=ParallelismConfiguration(max_parallel_execution_steps=5),
)
```
boto3 throws a `ParamValidationError`.
```
ParamValidationError: Parameter validation failed:
Invalid type for parameter ParallelismConfiguration, value: , type: , valid types:
```
- Boto3 is expecting a dict, and isn't able to handle the ParallelismConfiguration object, which is passed through to it.
- If you convert the ParallelismConfiguration to a dict before passing it in, using the `to_request()` method, the call succeeds.
- However, this does require passing in an object with an incorrect type, which angers the type checker. A recent commit has enabled type checking for this module, which caused this problem to show up.
- The tests in `test_workflow.py` [erroneously pass in a dict](https://github.com/qidewenwhen/sagemaker-python-sdk/blob/57c2b941d7bb712f2fe1870381e5ff2bc809ee66/tests/integ/sagemaker/workflow/test_workflow.py#L1006), rather than the correct ParallelismConfiguration object.
- Discovered in `v2.254.1`. It's been present for...quite a while I think. Issue became obvious in v2.245.0 when type validation was enabled.
- I quickly checked pipeline.create and pipeline.update too - the behaviour seems to be the same.
## The fix
Presumably, call `to_request()` somewhere in workflows/pipeline.py when handling the ParallelConfiguration parameters.
# Long Version
- We recently upgraded from v2.243.2 to v2.254.1
- This included the [fix that enabled type checking](https://github.com/aws/sagemaker-python-sdk/commit/5ce7249ee6c11033bbe4ab9e6f8879198245762e#diff-22d6ae36367d72d3712cabce03e290ce3ceaded610d5dd4eaffb59619836e773):
- Our build failed our mypy type checking, as we were calling pipeline.upsert like this:
```
pipeline.upsert(
config.sm_role,
parallelism_config=ParallelismConfiguration(max_parallel_execution_steps=5).to_request(),
)
```
`to_request()` was converting the ParallelismConfiguration into a RequestType, which under the hood was a dict.
- The definition for upsert has long been
```
def upsert(
self,
role_arn: str = None,
description: str = None,
tags: Optional[Tags] = None,
parallelism_config: ParallelismConfiguration = None,
) -> Dict[str, Any]:
```
- The fix of the types therefore caused our type validator mypy to flag the issue - we were passing in a dict, and the correct type was a ParallelismConfiguration.
- However, when we fixed it, removing the `to_request()` call, we get the following error when invoking the method:
```
ParamValidationError: Parameter validation failed:
Invalid type for parameter ParallelismConfiguration, value: , type: , valid types:
```
Guía de contribución
Línea de trabajo
Comienza en src/sagemaker/workflow/pipeline.py y sigue cómo pipeline.upsert, pipeline.create y pipeline.update pasan parallelism_config. Revisa tests/integ/sagemaker/workflow/test_workflow.py, especialmente el caso de configuración de paralelismo, y ejecuta las pruebas relevantes del workflow. Se considera terminado cuando estos métodos acepten objetos ParallelismConfiguration y boto3 ya no genere ParamValidationError.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- python
- Área
- api, machine-learning
- Tipo de issue
- Error
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Estado de actividad
- Estancado
- Claridad
- Bien especificado
- Aptitud para principiantes
- 35/100