aws / aws/sagemaker-python-sdk

`metric_definitions` documentation claims arbitrary regex but requires metrics to be integers.

Open
#3,208 0 comments 0 reactions 1 assignee Claimed by @nargokul View on GitHub
type: bug
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the bug**
The documentation for [SKLearn](https://sagemaker.readthedocs.io/en/stable/frameworks/sklearn/sagemaker.sklearn.html?highlight=SKLearn#sagemaker.sklearn.estimator.SKLearn) constructor mentions that we can use custom `metric_definitions` with a regex. I checked that this kwarg is passed to `Framework` which then passes it to [EstimatorBase](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html#sagemaker.estimator.EstimatorBase).

This regex only captures numbers and not strings. This should either be mentioned in the documentation or support fort arbitrary strings needs to be present.

**To reproduce**

`cli.py`
```python
if __name__ == "__main__":
print("canary=1.01;")
print(f"cat=dog;")
```

`sagemaker.ipynb`
```python
from sagemaker.sklearn.estimator import SKLearn
import sagemaker
from sagemaker import get_execution_role
from copy import deepcopy
import itertools

sagemaker_session = sagemaker.Session()
role = get_execution_role()

FRAMEWORK_VERSION = "1.0-1"
script_path = "cli.py"

LOCAL=False

kwargs=dict(
instance_type="ml.m5.large",
instance_count=1,
sagemaker_session=sagemaker_session,
base_job_name="test-job-",
)

base_model_gen = lambda: EstimatorBase(
entry_point=script_path,
framework_version=FRAMEWORK_VERSION,
role=role,
source_dir="./",
metric_definitions=[dict(Name=m, Regex=f"{m}=(.*?);") for m in ["canary",'cat']],
**kwargs
)
base_model_gen().fit()
```

**Expected behavior**
Algorithm metrics should've displayed both `canary` and `cat`
Only `canary` is displayed

**Screenshots or logs**
![image](https://user-images.githubusercontent.com/108535060/176913863-29953eb1-6d04-4ce5-a43c-b61d50367679.png)

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: '2.97.0'
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: SKLearn
- **Framework version**: 1.0-1
- **Python version**: 3.8.12
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.