aws / aws/sagemaker-training-toolkit

Failed to parse string hyperparameter

Open
#66 6 comments 0 reactions 0 assignees View on GitHub
type: question
Dominant language
Python
Stars
530
Forks
140
Avg merge
1h 12m
Merged PRs (30d)
2

Description

**Describe the bug**
I would like to pass hyperparameters to my sagemaker job that are of type string. However, when I do this I get an error saying that they failed to parse

**To reproduce**
```python
hyperparams = {'test': 10,
'a': 50,
'b': 'some text'}

estimator = Estimator(
image_name=image_uri,
role=iam_role,
output_path=f"s3://{aws_params['SCW_S3_BUCKET']}/sagemaker/output/",
train_instance_count=instance_count,
input_mode='File',
train_instance_type='local',
tags=TR_TAGS,
subnets=aws_params['VPC_SUBNETS'],
security_group_ids=aws_params['VPC_SGS'],
output_kms_key=aws_params['SCW_KMS_KEY'],
hyperparameters=hyperparams,
)
```

I get the following error message:
```bash
algo-1-tfbx9_1 | 2020-06-26 11:46:29,060 - sagemaker-training-toolkit - INFO - Failed to parse hyperparameter b value some text to Json.
```

I suspect this has something to do with the code in https://github.com/aws/sagemaker-containers/blob/8ba4085548d28c8651cbd28b3049af58b3057fbc/src/sagemaker_containers/_env.py#L214

It seems as those these string valued hyperparameters aren't being passed to the command in the sagemaker job

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Estimator setup with numeric and string values, then inspect the referenced sagemaker-containers _env.py parsing path around line 214. Trace how hyperparameters reach the training command; done means string-valued parameters are accepted without the JSON parse error and are available to the job.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.