aws / aws/sagemaker-python-sdk

tensorflow: sm_drivers directory not found

Open
#5,493 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

**Describe the bug**
When running on the latest Tensorflow CPU training image: `763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-training:2.19-cpu-py312`, the job cannot properly run, because the `sm_drivers` directory seems to not be present on the container image (see logs).

**To reproduce**

```python
from pathlib import Path

from sagemaker.train.configs import Compute, InputData, SourceCode
from sagemaker.train.model_trainer import ModelTrainer

EXPERIMENT_NAME = "my-experiment"
ROLE_ARN = "arn:aws:iam::111222333444:role/SageMakerS3AccessRole"
TRAINING_IMAGE = (
"763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-training:2.19-cpu-py312"
)
INSTANCE_TYPE = "ml.c5.4xlarge"
DATASET_PATH = "s3://my-bucket/data/"

if __name__ == "__main__":
trainer = ModelTrainer(
base_job_name=EXPERIMENT_NAME,
role=ROLE_ARN,
training_image=TRAINING_IMAGE,
source_code=SourceCode(
source_dir=str(Path(__file__).parent),
entry_script="train.py",
requirements="requirements.sagemaker.txt",
ignore_patterns=[
"data",
".venv",
"notebooks",
"environment",
"scripts",
"pipelines",
".ipynb_checkpoints",
".github",
"__pycache__",
"*.ipynb",
],
),
compute=Compute(
instance_type=INSTANCE_TYPE
),
)

input_data = InputData(
channel_name="training",
data_source=DATASET_PATH,
)

trainer.train(input_data_config=[input_data])
```

**Expected behavior**
The job runs.

**Screenshots or logs**

See logs:
```

{"code":{"TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"},"sm_drivers":{"TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"},"training":{"
TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"}}
++ /usr/local/bin/python3 /opt/ml/input/data/sm_drivers/scripts/environment.py
Setting up environment variables
/usr/local/bin/python3: can't open file '/opt/ml/input/data/sm_drivers/scripts/environment.py': [Errno 2] No such file or directory
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**:
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: tensorflow
- **Framework version**: 2.19
- **Python version**: 3.12
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied ModelTrainer example with the TensorFlow 2.19 CPU image and inspect the logged invocation of /opt/ml/input/data/sm_drivers/scripts/environment.py. Trace how the training input and sm_drivers directory are prepared, then verify that the job runs successfully with the affected image.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.