aws / aws/sagemaker-training-toolkit
Failed to parse string hyperparameter
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
**Describe the bug**
I would like to pass hyperparameters to my sagemaker job that are of type string. However, when I do this I get an error saying that they failed to parse
**To reproduce**
```python
hyperparams = {'test': 10,
'a': 50,
'b': 'some text'}
estimator = Estimator(
image_name=image_uri,
role=iam_role,
output_path=f"s3://{aws_params['SCW_S3_BUCKET']}/sagemaker/output/",
train_instance_count=instance_count,
input_mode='File',
train_instance_type='local',
tags=TR_TAGS,
subnets=aws_params['VPC_SUBNETS'],
security_group_ids=aws_params['VPC_SGS'],
output_kms_key=aws_params['SCW_KMS_KEY'],
hyperparameters=hyperparams,
)
```
I get the following error message:
```bash
algo-1-tfbx9_1 | 2020-06-26 11:46:29,060 - sagemaker-training-toolkit - INFO - Failed to parse hyperparameter b value some text to Json.
```
I suspect this has something to do with the code in https://github.com/aws/sagemaker-containers/blob/8ba4085548d28c8651cbd28b3049af58b3057fbc/src/sagemaker_containers/_env.py#L214
It seems as those these string valued hyperparameters aren't being passed to the command in the sagemaker job
Contributor guide
Research direction
Start by reproducing the Estimator setup with numeric and string values, then inspect the referenced sagemaker-containers _env.py parsing path around line 214. Trace how hyperparameters reach the training command; done means string-valued parameters are accepted without the JSON parse error and are available to the job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100