aws / aws/sagemaker-python-sdk
ParamValidationError thrown when setting parallelism_config in pipeline.upsert, pipeline.create etc.
- Langage dominant
- Python
- Étoiles
- 2.3k
- Forks
- 1.3k
- Merge moyen
- 1 j 22 h
- PR mergées (30 j)
- 35
Description
# Quick version
## The Problem
- When calling `pipeline.upsert` ([code](https://github.com/qidewenwhen/sagemaker-python-sdk/blob/57c2b941d7bb712f2fe1870381e5ff2bc809ee66/src/sagemaker/workflow/pipeline.py)), if you pass in a`parallelism_config` value with the correct type of ParallelIismConfiguration like this:
```
pipeline.upsert(
config.sm_role,
parallelism_config=ParallelismConfiguration(max_parallel_execution_steps=5),
)
```
boto3 throws a `ParamValidationError`.
```
ParamValidationError: Parameter validation failed:
Invalid type for parameter ParallelismConfiguration, value: , type: , valid types:
```
- Boto3 is expecting a dict, and isn't able to handle the ParallelismConfiguration object, which is passed through to it.
- If you convert the ParallelismConfiguration to a dict before passing it in, using the `to_request()` method, the call succeeds.
- However, this does require passing in an object with an incorrect type, which angers the type checker. A recent commit has enabled type checking for this module, which caused this problem to show up.
- The tests in `test_workflow.py` [erroneously pass in a dict](https://github.com/qidewenwhen/sagemaker-python-sdk/blob/57c2b941d7bb712f2fe1870381e5ff2bc809ee66/tests/integ/sagemaker/workflow/test_workflow.py#L1006), rather than the correct ParallelismConfiguration object.
- Discovered in `v2.254.1`. It's been present for...quite a while I think. Issue became obvious in v2.245.0 when type validation was enabled.
- I quickly checked pipeline.create and pipeline.update too - the behaviour seems to be the same.
## The fix
Presumably, call `to_request()` somewhere in workflows/pipeline.py when handling the ParallelConfiguration parameters.
# Long Version
- We recently upgraded from v2.243.2 to v2.254.1
- This included the [fix that enabled type checking](https://github.com/aws/sagemaker-python-sdk/commit/5ce7249ee6c11033bbe4ab9e6f8879198245762e#diff-22d6ae36367d72d3712cabce03e290ce3ceaded610d5dd4eaffb59619836e773):
- Our build failed our mypy type checking, as we were calling pipeline.upsert like this:
```
pipeline.upsert(
config.sm_role,
parallelism_config=ParallelismConfiguration(max_parallel_execution_steps=5).to_request(),
)
```
`to_request()` was converting the ParallelismConfiguration into a RequestType, which under the hood was a dict.
- The definition for upsert has long been
```
def upsert(
self,
role_arn: str = None,
description: str = None,
tags: Optional[Tags] = None,
parallelism_config: ParallelismConfiguration = None,
) -> Dict[str, Any]:
```
- The fix of the types therefore caused our type validator mypy to flag the issue - we were passing in a dict, and the correct type was a ParallelismConfiguration.
- However, when we fixed it, removing the `to_request()` call, we get the following error when invoking the method:
```
ParamValidationError: Parameter validation failed:
Invalid type for parameter ParallelismConfiguration, value: , type: , valid types:
```
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez dans src/sagemaker/workflow/pipeline.py et suivez la manière dont parallelism_config est transmis par pipeline.upsert, pipeline.create et pipeline.update. Examinez tests/integ/sagemaker/workflow/test_workflow.py, en particulier le cas de configuration du parallélisme, et exécutez les tests de workflow pertinents. Le travail est terminé lorsque ces méthodes acceptent des objets ParallelismConfiguration et que boto3 ne lève plus ParamValidationError.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python
- Domaine
- api, machine-learning
- Type d'issue
- Bug
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Activité
- À l'abandon
- Clarté
- Clairement spécifiée
- Accessibilité débutants
- 35/100