aws / aws/sagemaker-python-sdk

FrameworkProcessor wait/describe doesn't respect region

未关闭
#5,796 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [X] PySDK V3 (3.x)

**Describe the bug**

`FrameworkProcessor` respects a region provided via `sagemaker_session.boto_session` when creating the job, but then ignores it when attempting to call `DescribeProcessingJob` for monitoring when `wait=True`.

**To reproduce**

Create any processing job in a **different region from your current/default**, as follows, and run it with `wait=True`:

```python
import boto3
from sagemaker.core.helper.session_helper import Session as SageMakerSession
from sagemaker.core.processing import FrameworkProcessor

sagemaker_sess = SageMakerSession(
# Anywhere that ISN'T your current default/active region:
boto_session=boto3.Session(region_name="ap-southeast-1"),
)
pytorch_processor = FrameworkProcessor(
sagemaker_session=sagemaker_sess,
...
)

pytorch_processor.run(
wait=True,
...
)
```

For example could consider adjusting [this walkthrough sample](https://sagemaker.readthedocs.io/en/stable/v3-examples/ml-ops-examples/v3-processing-job-pytorch/v3-pytorch-processing-example.html).

**Expected behavior**

The job runs in the explicitly specified region (`ap-southeast-1` above), and tails logs as usual for `wait=True`

**Actual behavior**

The job starts just fine, but the SDK immediately exits with an error because it tries to call `DescribeProcessingJob` in the current default region - not the one where the job was started.

**Screenshots or logs**

```
[04/24/26 23:37:57] INFO Creating processing-job with name glm-47--2026-0424-18-07-54 [processing.py](file:///.../.venv/lib/python3.12/site-packages/sagemaker/core/processing.py):[612](file:///.../.venv/lib/python3.12/site-packages/sagemaker/core/processing.py#612)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in :70 │
│ │
│ 67 time_suffix = strftime("%Y-%m%d-%H-%M-%S", gmtime()) │
│ 68 job_name = "--".join((model_id_namepart, time_suffix)) │
│ 69 │
│ ❱ 70 pytorch_processor.run( │
│ 71 │ code="main.py", │
│ 72 │ source_dir=os.path.abspath("code"), │
│ 73 │ job_name=job_name, │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/workflow/pipeline_c │
│ ontext.py:346 in wrapper │
│ │
│ 343 │ │ │ │
│ 344 │ │ │ return _StepArguments(retrieve_caller_name(self_instance), run_func, *args, │
│ 345 │ │ │
│ ❱ 346 │ │ return run_func(*args, **kwargs) │
│ 347 │ │
│ 348 │ return wrapper │
│ 349 │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/processing.py:1229 │
│ in run │
│ │
│ 1226 │ │ ) │
│ 1227 │ │ │
│ 1228 │ │ # Submit a processing job. │
│ ❱ 1229 │ │ return super().run( │
│ 1230 │ │ │ code=s3_runproc_sh, │
│ 1231 │ │ │ inputs=inputs, │
│ 1232 │ │ │ outputs=outputs, │
│ │
│ /....venv/lib/python3.12/site-packages/sagemaker/core/workflow/pipeline_c │
│ ontext.py:346 in wrapper │
│ │
│ 343 │ │ │ │
│ 344 │ │ │ return _StepArguments(retrieve_caller_name(self_instance), run_func, *args, │
│ 345 │ │ │
│ ❱ 346 │ │ return run_func(*args, **kwargs) │
│ 347 │ │
│ 348 │ return wrapper │
│ 349 │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/processing.py:846 │
│ in run │
│ │
│ 843 │ │ if not isinstance(self.sagemaker_session, PipelineSession): │
│ 844 │ │ │ self.jobs.append(self.latest_job) │
│ 845 │ │ │ if wait: │
│ ❱ 846 │ │ │ │ self.latest_job.wait(logs=logs) │
│ 847 │ │
│ 848 │ def _include_code_in_inputs(self, inputs, code, kms_key=None): │
│ 849 │ │ """Converts code to appropriate input and includes in input list. │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/resources.py:143 in │
│ wrapper │
│ │
│ 140 │ │ @functools.wraps(func) │
│ 141 │ │ def wrapper(*args, **kwargs): │
│ 142 │ │ │ config = dict(arbitrary_types_allowed=True) │
│ ❱ 143 │ │ │ return validate_call(config=config)(func)(*args, **kwargs) │
│ 144 │ │ │
│ 145 │ │ return wrapper │
│ 146 │
│ │
│ /.../.venv/lib/python3.12/site-packages/pydantic/_internal/_validate_call. │
│ py:39 in wrapper_function │
│ │
│ 36 │ │ │
│ 37 │ │ @functools.wraps(wrapped) │
│ 38 │ │ def wrapper_function(*args, **kwargs): │
│ ❱ 39 │ │ │ return wrapper(*args, **kwargs) │
│ 40 │ │
│ 41 │ # We need to manually update this because `partial` object has no `__name__` and `__ │
│ 42 │ wrapper_function.__name__ = extract_function_name(wrapped) │
│ │
│ /.../.venv/lib/python3.12/site-packages/pydantic/_internal/_validate_call. │
│ py:136 in __call__ │
│ │
│ 133 │ │ if not self.__pydantic_complete__: │
│ 134 │ │ │ self._create_validators() │
│ 135 │ │ │
│ ❱ 136 │ │ res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, │
│ 137 │ │ if self.__return_pydantic_validator__: │
│ 138 │ │ │ return self.__return_pydantic_validator__(res) │
│ 139 │ │ else: │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/resources.py:32145 │
│ in wait │
│ │
│ 32142 │ │ │ transient=True, │
│ 32143 │ │ ): │
│ 32144 │ │ │ while True: │
│ ❱ 32145 │ │ │ │ self.refresh() │
│ 32146 │ │ │ │ current_status = self.processing_job_status │
│ 32147 │ │ │ │ status.update(f"Current status: [bold]{current_status}") │
│ 32148 │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/resources.py:143 in │
│ wrapper │
│ │
│ 140 │ │ @functools.wraps(func) │
│ 141 │ │ def wrapper(*args, **kwargs): │
│ 142 │ │ │ config = dict(arbitrary_types_allowed=True) │
│ ❱ 143 │ │ │ return validate_call(config=config)(func)(*args, **kwargs) │
│ 144 │ │ │
│ 145 │ │ return wrapper │
│ 146 │
│ │
│ /.../.venv/lib/python3.12/site-packages/pydantic/_internal/_validate_call. │
│ py:39 in wrapper_function │
│ │
│ 36 │ │ │
│ 37 │ │ @functools.wraps(wrapped) │
│ 38 │ │ def wrapper_function(*args, **kwargs): │
│ ❱ 39 │ │ │ return wrapper(*args, **kwargs) │
│ 40 │ │
│ 41 │ # We need to manually update this because `partial` object has no `__name__` and `__ │
│ 42 │ wrapper_function.__name__ = extract_function_name(wrapped) │
│ │
│ /.../.venv/lib/python3.12/site-packages/pydantic/_internal/_validate_call. │
│ py:136 in __call__ │
│ │
│ 133 │ │ if not self.__pydantic_complete__: │
│ 134 │ │ │ self._create_validators() │
│ 135 │ │ │
│ ❱ 136 │ │ res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, │
│ 137 │ │ if self.__return_pydantic_validator__: │
│ 138 │ │ │ return self.__return_pydantic_validator__(res) │
│ 139 │ │ else: │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/resources.py:32025 │
│ in refresh │
│ │
│ 32022 │ │ logger.debug(f"Serialized input request: {operation_input_args}") │
│ 32023 │ │ │
│ 32024 │ │ client = Base.get_sagemaker_client() │
│ ❱ 32025 │ │ response = client.describe_processing_job(**operation_input_args) │
│ 32026 │ │ │
│ 32027 │ │ # deserialize response and update self │
│ 32028 │ │ transform(response, "DescribeProcessingJobResponse", self) │
│ │
│ /.../.venv/lib/python3.12/site-packages/botocore/client.py:602 in │
│ _api_call │
│ │
│ 599 │ │ │ │ │ f"{py_operation_name}() only accepts keyword arguments." │
│ 600 │ │ │ │ ) │
│ 601 │ │ │ # The "self" in this scope is referring to the BaseClient. │
│ ❱ 602 │ │ │ return self._make_api_call(operation_name, kwargs) │
│ 603 │ │ │
│ 604 │ │ _api_call.__name__ = str(py_operation_name) │
│ 605 │
│ │
│ /.../.venv/lib/python3.12/site-packages/botocore/context.py:123 in wrapper │
│ │
│ 120 │ │ │ with start_as_current_context(): │
│ 121 │ │ │ │ if hook: │
│ 122 │ │ │ │ │ hook() │
│ ❱ 123 │ │ │ │ return func(*args, **kwargs) │
│ 124 │ │ │
│ 125 │ │ return wrapper │
│ 126 │
│ │
│ /.../.venv/lib/python3.12/site-packages/botocore/client.py:1078 in │
│ _make_api_call │
│ │
│ 1075 │ │ │ │ 'error_code_override' │
│ 1076 │ │ │ ) or error_info.get("Code") │
│ 1077 │ │ │ error_class = self.exceptions.from_code(error_code) │
│ ❱ 1078 │ │ │ raise error_class(parsed_response, operation_name) │
│ 1079 │ │ else: │
│ 1080 │ │ │ return parsed_response │
│ 1081 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ClientError: An error occurred (ValidationException) when calling the DescribeProcessingJob operation: Could not
find requested job with name: glm-47--2026-0424-18-07-54
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: `sagemaker-core` v2.6.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: PyTorch
- **Framework version**: 2.8
- **Python version**: 3.12
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**

N/A

贡献指南

打开贡献指南

调研方向

首先跟踪 sagemaker/core/processing.py 中的 FrameworkProcessor.run,直到 latest_job.wait,然后检查 sagemaker/core/resources.py 中 processing-job 资源的刷新路径。在默认区域不同时,使用设置为 ap-southeast-1 的 boto3 session 重现该问题。完成的标准是,wait=True 在显式配置的区域中监控并跟踪该 job,且不会出现 DescribeProcessingJob 错误。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python, pytorch
领域
backend, machine-learning
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
68/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。