aws / aws/sagemaker-python-sdk
ModelBuilder with source_code + DJL LMI: /opt/ml/model becomes read-only, breaking HF Hub model downloads
- Ngôn ngữ chính
- Python
- Star
- 2.3k
- Fork
- 1.3k
- Merge trung bình
- 1 ngày 22 giờ
- Pull request đã merge (30 ngày)
- 35
Mô tả
## Description
When using `ModelBuilder` (SDK v3) with a pre-built DJL LMI container image and `source_code` (via `SourceCode`) to provide custom `requirements.txt`, the model directory `/opt/ml/model/` becomes read-only at runtime. This prevents the DJL container from downloading models from HuggingFace Hub, which tries to write cache files to `/opt/ml/model/`.
Additionally, `ModelBuilder` overrides user-provided `HF_MODEL_ID` environment variable with the value from the `model=` parameter, making it impossible to point the container to the local model path (`/opt/ml/model`) when S3 model artifacts are also provided via `s3_model_data_url`.
## How to Reproduce
```python
from sagemaker.serve import ModelBuilder, ModelServer
from sagemaker.serve.builder.schema_builder import SchemaBuilder
from sagemaker.serve.mode.function_pointers import Mode
from sagemaker.serve.model_builder import SourceCode
source_code = SourceCode(
source_dir="./model_code",
requirements="requirements.txt", # e.g. transformers>=4.55.0
)
mb = ModelBuilder(
model="chromadb/context-1", # HF Hub model ID
role_arn=ROLE,
image_uri="763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.36.0-lmi22.0.0-cu129",
model_server=ModelServer.DJL_SERVING,
schema_builder=SchemaBuilder(
{"inputs": "Hello", "parameters": {"max_new_tokens": 64}},
[{"generated_text": "Hi"}],
),
source_code=source_code,
env_vars={"OPTION_TENSOR_PARALLEL_DEGREE": "4", ...},
instance_type="ml.g6e.12xlarge",
mode=Mode.SAGEMAKER_ENDPOINT,
)
model = mb.build()
endpoint = mb.deploy(endpoint_name="test", wait=True)
# FAILS: OSError: [Errno 30] Read-only file system: /opt/ml/model/models--chromadb--context-1
```
## Observed Behavior
1. `ModelBuilder.build()` packages the `source_code` directory into a `model.tar.gz` and uploads to S3
2. At deploy time, SageMaker mounts this tar.gz at `/opt/ml/model/` — which becomes **read-only**
3. `ModelBuilder` sets `HF_MODEL_ID=chromadb/context-1` (from `model=`), overriding any user-provided value
4. DJL LMI container sees `HF_MODEL_ID=chromadb/context-1` and tries to download from HF Hub
5. HF Hub download tries to write cache to `/opt/ml/model/models--chromadb--context-1/`
6. Fails with `OSError: [Errno 30] Read-only file system`
CloudWatch logs confirm:
```
OSError: [Errno 30] Read-only file system: /opt/ml/model/models--chromadb--context-1
```
## Expected Behavior
Users should be able to use `ModelBuilder` with:
- A pre-built container image (e.g. DJL LMI)
- `source_code` with a custom `requirements.txt` to install additional dependencies at container startup
- A HuggingFace Hub model ID that the container downloads at runtime
The `requirements.txt` installation should not make `/opt/ml/model/` read-only, or the HF Hub cache should be redirected to a writable location (e.g. `/tmp`).
## Workaround Attempted
Setting `HF_HOME=/tmp/hf_home` and `HUGGINGFACE_HUB_CACHE=/tmp/hf_home/hub` in `env_vars` — these appear in the container environment but the DJL container still writes to `/opt/ml/model/`.
## Use Case
This is a common pattern for deploying newer models (e.g. OpenAI GPT-OSS based models like `chromadb/context-1`) that require a newer `transformers` version than what is bundled in the DJL LMI container. The `source_code` with `requirements.txt` is the natural SDK v3 mechanism for this, but it is incompatible with HF Hub model downloads.
## Environment
- SageMaker Python SDK: 3.6.0
- Container: `djl-inference:0.36.0-lmi22.0.0-cu129`
- Instance: `ml.g6e.12xlarge`
- Region: `us-east-1`
Hướng dẫn đóng góp
Hướng nghiên cứu
Start at ModelBuilder.build() and deploy(), along with SourceCode packaging, and reproduce the issue with the DJL LMI image and the supplied model configuration. Trace how source_code is packaged and how HF_MODEL_ID and env_vars are applied. Done means runtime model downloads can write to a writable location and a user-provided HF_MODEL_ID is not unexpectedly replaced.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- aws, python
- Lĩnh vực
- cloud, machine-learning
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 48/100