aws / aws/sagemaker-python-sdk

FrameworkProcessor.run(requirements=...) ignores custom requirements file in PySDK V3

Open
#5,805 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
2d 2h
Merged PRs (30d)
32

Description

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [X] PySDK V3 (3.x)

**Describe the bug**

FrameworkProcessor.run(requirements=...) in the SageMaker v3 sagemaker-core Processing API accepts a custom requirements file name, but the generated runtime script does not use it.

For example, this API call accepts:

framework_processor.run(
code="run_entrypoint.py",
source_dir=".",
requirements="cpu-requirements.txt",
)

However, the generated runproc.sh only checks for a hardcoded file named requirements.txt:

```
if [[ -f 'requirements.txt' ]]; then
pip uninstall --yes typing
pip install -r requirements.txt
fi
```

So cpu-requirements.txt may be packaged into sourcedir.tar.gz, but it is not installed at runtime.

Relevant file:

sagemaker-core/src/sagemaker/core/processing.py

Relevant method:

FrameworkProcessor._generate_framework_script()

Reference:

https://github.com/aws/sagemaker-python-sdk/blob/master/sagemaker-core/src/sagemaker/core/processing.py#L1187

To reproduce

Create the following project structure:

.
├── run_entrypoint.py
├── cpu-requirements.txt
└── scripts/
└── entrypoints/
└── step_preprocess_jobs.py

Example cpu-requirements.txt:

requests==2.32.3

Example run_entrypoint.py:

```
import importlib
import sys
module_name = sys.argv[1]
module = importlib.import_module(module_name)
if hasattr(module, "main"):
module.main()
```

Example scripts/entrypoints/step_preprocess_jobs.py:

```
def main():
import requests
print("requests imported successfully")
```

Then create a SageMaker Processing step using FrameworkProcessor.run(...) with a custom requirements file name:

```
from sagemaker.core.processing import FrameworkProcessor
from sagemaker.core.workflow.pipeline_context import PipelineSession
from sagemaker.core.shapes import ProcessingInput, ProcessingOutput, ProcessingS3Input, ProcessingS3Output
role = ""
image_uri = ""
bucket = ""
pipeline_session = PipelineSession()
framework_processor = FrameworkProcessor(
image_uri=image_uri,
role=role,
instance_count=1,
instance_type="ml.m5.large",
sagemaker_session=pipeline_session,
)
step_args = framework_processor.run(
code="run_entrypoint.py",
source_dir=".",
requirements="cpu-requirements.txt",
arguments=["scripts.entrypoints.step_preprocess_jobs"],
inputs=[
ProcessingInput(
input_name="dummy_input",
s3_input=ProcessingS3Input(
s3_uri=f"s3://{bucket}/dummy-input/",
local_path="/opt/ml/processing/input/dummy",
s3_data_type="S3Prefix",
s3_input_mode="File",
),
)
],
outputs=[
ProcessingOutput(
output_name="dummy_output",
s3_output=ProcessingS3Output(
s3_uri=f"s3://{bucket}/dummy-output/",
local_path="/opt/ml/processing/output",
s3_upload_mode="EndOfJob",
),
)
],
)
```

The generated processing runtime script only installs requirements.txt, not the file passed through the requirements argument.

The relevant generated script logic is:

if [[ -f 'requirements.txt' ]]; then
pip uninstall --yes typing
pip install -r requirements.txt
fi

As a result, dependencies listed in cpu-requirements.txt are not installed.

Expected behavior

When a user passes:

requirements="cpu-requirements.txt"

to FrameworkProcessor.run(...), the generated runtime script should install dependencies from that file:

pip install -r cpu-requirements.txt

The requirements argument should be respected.

For backward compatibility, if requirements is not provided, the implementation may continue checking for and installing from requirements.txt.

Current generated script behavior:

```
cd /opt/ml/processing/input/code/
if [ -f sourcedir.tar.gz ]; then
tar -xzf sourcedir.tar.gz
else
echo "ERROR: sourcedir.tar.gz not found!"
exit 1
fi
if [[ -f 'requirements.txt' ]]; then
pip uninstall --yes typing
pip install -r requirements.txt
fi
```

Expected generated script behavior when requirements="cpu-requirements.txt" is provided:

```
if [[ -f 'cpu-requirements.txt' ]]; then
pip uninstall --yes typing
pip install -r cpu-requirements.txt
else
echo "WARNING: requirements file not found: cpu-requirements.txt"
fi
```

System information

* SageMaker Python SDK version: PySDK V3 / sagemaker-core
* Framework name (eg. PyTorch) or algorithm (eg. KMeans): FrameworkProcessor
* Framework version: N/A
* Python version: N/A
* CPU or GPU: CPU
* Custom Docker image (Y/N): Y

Additional context

The requirements parameter is accepted by FrameworkProcessor.run(...) and passed into internal methods such as _pack_and_upload_code(...) and _package_code(...), but it is not used when generating the runtime shell script.

The root cause appears to be that _generate_framework_script(...) hardcodes requirements.txt instead of using the requirements argument passed by the user.

I would like to work on this issue and submit a PR to fix the behavior.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.