aws / aws/sagemaker-training-toolkit
Issue when training in local mode with huggingface training container
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
**Describe the bug**
When attempting to train locally with a simple script using a huggingface training container (i.e. from [here](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#huggingface-training-containers)) I get the following error:
```jq: error: Could not open file /opt/ml/input/config/resourceconfig.json: No such file or directory```
**To reproduce**
The local training script is as follows:
```
import os
from sagemaker.huggingface import HuggingFace
from sagemaker.local import LocalSession
sagemaker_session = LocalSession()
sagemaker_session.config = {'local': {'local_code': True}}
huggingface_estimator = HuggingFace(
py_version=None,
entry_point="train.py",
image_uri="763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.28.1-gpu-py310-cu118-ubuntu20.04",
role="sagemaker-studio-user-prod",
source_dir="scripts/pipeline_scripts",
instance_type='local',
instance_count=1,
input_mode='File',
output_path=f"file://{os.getcwd()}/tests/test_output_data/trained_model",
code_location="path_to_s3_dir",
)
huggingface_estimator.fit({
'train': f"file://{os.getcwd()}/tests/test_output_data/preprocessed/train",
'val': f"file://{os.getcwd()}/tests/test_output_data/preprocessed/val",
})
```
This is run locally using python, leading to the error. I don't think the contents of the `train.py` file are relevant as this happens during setup of the [training environment](https://github.com/aws/sagemaker-training-toolkit/blob/5c85f3f9b8399d3256d967674cbdc4b5c8f9cf40/src/sagemaker_training/environment.py#L70). The train and val files are tokenized tensors in arrow format output by calling `.save_to_disk` on a transformers Dataset object.
Note this same error occurred on previous versions of the training container also.
**Expected behavior**
For the training to complete as it does when running in SageMaker. The same configuration runs okay as part of a sagemaker pipeline running on sagemaker managed instances. This local training also worked previously, and we can't isolate what has changed to now lead to the above error.
**Screenshots or logs**
Full logs:
```
INFO:sagemaker:Creating training-job with name: training-2023-09-08-09-31-48-957
INFO:sagemaker.local.local_session:Starting training job
WARNING:sagemaker.local.image:Using the short-lived AWS credentials found in session. They might expire while running.
INFO:sagemaker.local.image:docker compose file:
networks:
sagemaker-local:
name: sagemaker-local
services:
algo-1-si1mk:
command: train
container_name: c1sy6fk9n1-algo-1-si1mk
environment:
- '[Masked]'
- '[Masked]'
- '[Masked]'
- '[Masked]'
- '[Masked]'
image: 763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.28.1-gpu-py310-cu118-ubuntu20.04
networks:
sagemaker-local:
aliases:
- algo-1-si1mk
stdin_open: true
tty: true
volumes:
- /private/var/folders/dr/n0xslz555m128480ykmyr0t40000gp/T/tmp5jg5mcvx/algo-1-si1mk/input:/opt/ml/input
- /private/var/folders/dr/n0xslz555m128480ykmyr0t40000gp/T/tmp5jg5mcvx/algo-1-si1mk/output:/opt/ml/output
- /private/var/folders/dr/n0xslz555m128480ykmyr0t40000gp/T/tmp5jg5mcvx/algo-1-si1mk/output/data:/opt/ml/output/data
- /private/var/folders/dr/n0xslz555m128480ykmyr0t40000gp/T/tmp5jg5mcvx/model:/opt/ml/model
- /Users/owenturner/dev/banquo-bert/tests/test_output_data/preprocessed/train:/opt/ml/input/data/train
- /Users/owenturner/dev/banquo-bert/tests/test_output_data/preprocessed/val:/opt/ml/input/data/val
version: '2.3'
INFO:sagemaker.local.image:docker command: docker-compose -f /private/var/folders/dr/n0xslz555m128480ykmyr0t40000gp/T/tmp5jg5mcvx/docker-compose.yaml up --build --abort-on-container-exit
Creating network "sagemaker-local" with the default driver
Creating c1sy6fk9n1-algo-1-si1mk ...
Creating c1sy6fk9n1-algo-1-si1mk ... done
Attaching to c1sy6fk9n1-algo-1-si1mk
c1sy6fk9n1-algo-1-si1mk | jq: error: Could not open file /opt/ml/input/config/resourceconfig.json: No such file or directory
c1sy6fk9n1-algo-1-si1mk | changehostname.c: In function ‘gethostname’:
c1sy6fk9n1-algo-1-si1mk | changehostname.c:15:21: error: expected expression before ‘;’ token
c1sy6fk9n1-algo-1-si1mk | 15 | const char *val = ;
c1sy6fk9n1-algo-1-si1mk | | ^
c1sy6fk9n1-algo-1-si1mk | gcc: error: changehostname.o: No such file or directory
c1sy6fk9n1-algo-1-si1mk | ERROR: ld.so: object '/libchangehostname.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
c1sy6fk9n1-algo-1-si1mk | Reporting training FAILURE
c1sy6fk9n1-algo-1-si1mk | Framework Error:
c1sy6fk9n1-algo-1-si1mk | Traceback (most recent call last):
c1sy6fk9n1-algo-1-si1mk | File "/opt/conda/lib/python3.10/site-packages/sagemaker_training/trainer.py", line 70, in train
c1sy6fk9n1-algo-1-si1mk | env = environment.Environment()
c1sy6fk9n1-algo-1-si1mk | File "/opt/conda/lib/python3.10/site-packages/sagemaker_training/environment.py", line 576, in __init__
c1sy6fk9n1-algo-1-si1mk | resource_config = resource_config or read_resource_config()
c1sy6fk9n1-algo-1-si1mk | File "/opt/conda/lib/python3.10/site-packages/sagemaker_training/environment.py", line 254, in read_resource_config
c1sy6fk9n1-algo-1-si1mk | return _read_json(resource_config_file_dir)
c1sy6fk9n1-algo-1-si1mk | File "/opt/conda/lib/python3.10/site-packages/sagemaker_training/environment.py", line 201, in _read_json
c1sy6fk9n1-algo-1-si1mk | with open(path, "r") as f:
c1sy6fk9n1-algo-1-si1mk | FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/config/resourceconfig.json'
c1sy6fk9n1-algo-1-si1mk |
c1sy6fk9n1-algo-1-si1mk | [Errno 2] No such file or directory: '/opt/ml/input/config/resourceconfig.json'
c1sy6fk9n1-algo-1-si1mk | Encountered exit_code 2
c1sy6fk9n1-algo-1-si1mk exited with code 2
```
**System information**
A description of your system.
- Sagemaker training version - 4.5.0
- prebuild docker [image url](763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.28.1-gpu-py310-cu118-ubuntu20.04)
**Additional context**
Add any other context about the problem here.
Contributor guide
Research direction
Start with sagemaker_training/environment.py, especially Environment and read_resource_config, and compare that expectation with the docker-compose setup emitted by sagemaker.local.local_session. Reproduce the command using the Hugging Face training image and inspect how /opt/ml/input/config is created; done means local training no longer fails during environment setup and completes as it does on managed SageMaker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, huggingface, python
- Domain
- devops, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100