aws / aws/sagemaker-python-sdk

FrameworkProcessor wait/describe doesn't respect region

Open
#5,796 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [X] PySDK V3 (3.x)

**Describe the bug**

`FrameworkProcessor` respects a region provided via `sagemaker_session.boto_session` when creating the job, but then ignores it when attempting to call `DescribeProcessingJob` for monitoring when `wait=True`.

**To reproduce**

Create any processing job in a **different region from your current/default**, as follows, and run it with `wait=True`:

```python
import boto3
from sagemaker.core.helper.session_helper import Session as SageMakerSession
from sagemaker.core.processing import FrameworkProcessor

sagemaker_sess = SageMakerSession(
# Anywhere that ISN'T your current default/active region:
boto_session=boto3.Session(region_name="ap-southeast-1"),
)
pytorch_processor = FrameworkProcessor(
sagemaker_session=sagemaker_sess,
...
)

pytorch_processor.run(
wait=True,
...
)
```

For example could consider adjusting [this walkthrough sample](https://sagemaker.readthedocs.io/en/stable/v3-examples/ml-ops-examples/v3-processing-job-pytorch/v3-pytorch-processing-example.html).

**Expected behavior**

The job runs in the explicitly specified region (`ap-southeast-1` above), and tails logs as usual for `wait=True`

**Actual behavior**

The job starts just fine, but the SDK immediately exits with an error because it tries to call `DescribeProcessingJob` in the current default region - not the one where the job was started.

**Screenshots or logs**

```
[04/24/26 23:37:57] INFO Creating processing-job with name glm-47--2026-0424-18-07-54 [processing.py](file:///.../.venv/lib/python3.12/site-packages/sagemaker/core/processing.py):[612](file:///.../.venv/lib/python3.12/site-packages/sagemaker/core/processing.py#612)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in :70 │
│ │
│ 67 time_suffix = strftime("%Y-%m%d-%H-%M-%S", gmtime()) │
│ 68 job_name = "--".join((model_id_namepart, time_suffix)) │
│ 69 │
│ ❱ 70 pytorch_processor.run( │
│ 71 │ code="main.py", │
│ 72 │ source_dir=os.path.abspath("code"), │
│ 73 │ job_name=job_name, │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/workflow/pipeline_c │
│ ontext.py:346 in wrapper │
│ │
│ 343 │ │ │ │
│ 344 │ │ │ return _StepArguments(retrieve_caller_name(self_instance), run_func, *args, │
│ 345 │ │ │
│ ❱ 346 │ │ return run_func(*args, **kwargs) │
│ 347 │ │
│ 348 │ return wrapper │
│ 349 │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/processing.py:1229 │
│ in run │
│ │
│ 1226 │ │ ) │
│ 1227 │ │ │
│ 1228 │ │ # Submit a processing job. │
│ ❱ 1229 │ │ return super().run( │
│ 1230 │ │ │ code=s3_runproc_sh, │
│ 1231 │ │ │ inputs=inputs, │
│ 1232 │ │ │ outputs=outputs, │
│ │
│ /....venv/lib/python3.12/site-packages/sagemaker/core/workflow/pipeline_c │
│ ontext.py:346 in wrapper │
│ │
│ 343 │ │ │ │
│ 344 │ │ │ return _StepArguments(retrieve_caller_name(self_instance), run_func, *args, │
│ 345 │ │ │
│ ❱ 346 │ │ return run_func(*args, **kwargs) │
│ 347 │ │
│ 348 │ return wrapper │
│ 349 │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/processing.py:846 │
│ in run │
│ │
│ 843 │ │ if not isinstance(self.sagemaker_session, PipelineSession): │
│ 844 │ │ │ self.jobs.append(self.latest_job) │
│ 845 │ │ │ if wait: │
│ ❱ 846 │ │ │ │ self.latest_job.wait(logs=logs) │
│ 847 │ │
│ 848 │ def _include_code_in_inputs(self, inputs, code, kms_key=None): │
│ 849 │ │ """Converts code to appropriate input and includes in input list. │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/resources.py:143 in │
│ wrapper │
│ │
│ 140 │ │ @functools.wraps(func) │
│ 141 │ │ def wrapper(*args, **kwargs): │
│ 142 │ │ │ config = dict(arbitrary_types_allowed=True) │
│ ❱ 143 │ │ │ return validate_call(config=config)(func)(*args, **kwargs) │
│ 144 │ │ │
│ 145 │ │ return wrapper │
│ 146 │
│ │
│ /.../.venv/lib/python3.12/site-packages/pydantic/_internal/_validate_call. │
│ py:39 in wrapper_function │
│ │
│ 36 │ │ │
│ 37 │ │ @functools.wraps(wrapped) │
│ 38 │ │ def wrapper_function(*args, **kwargs): │
│ ❱ 39 │ │ │ return wrapper(*args, **kwargs) │
│ 40 │ │
│ 41 │ # We need to manually update this because `partial` object has no `__name__` and `__ │
│ 42 │ wrapper_function.__name__ = extract_function_name(wrapped) │
│ │
│ /.../.venv/lib/python3.12/site-packages/pydantic/_internal/_validate_call. │
│ py:136 in __call__ │
│ │
│ 133 │ │ if not self.__pydantic_complete__: │
│ 134 │ │ │ self._create_validators() │
│ 135 │ │ │
│ ❱ 136 │ │ res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, │
│ 137 │ │ if self.__return_pydantic_validator__: │
│ 138 │ │ │ return self.__return_pydantic_validator__(res) │
│ 139 │ │ else: │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/resources.py:32145 │
│ in wait │
│ │
│ 32142 │ │ │ transient=True, │
│ 32143 │ │ ): │
│ 32144 │ │ │ while True: │
│ ❱ 32145 │ │ │ │ self.refresh() │
│ 32146 │ │ │ │ current_status = self.processing_job_status │
│ 32147 │ │ │ │ status.update(f"Current status: [bold]{current_status}") │
│ 32148 │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/resources.py:143 in │
│ wrapper │
│ │
│ 140 │ │ @functools.wraps(func) │
│ 141 │ │ def wrapper(*args, **kwargs): │
│ 142 │ │ │ config = dict(arbitrary_types_allowed=True) │
│ ❱ 143 │ │ │ return validate_call(config=config)(func)(*args, **kwargs) │
│ 144 │ │ │
│ 145 │ │ return wrapper │
│ 146 │
│ │
│ /.../.venv/lib/python3.12/site-packages/pydantic/_internal/_validate_call. │
│ py:39 in wrapper_function │
│ │
│ 36 │ │ │
│ 37 │ │ @functools.wraps(wrapped) │
│ 38 │ │ def wrapper_function(*args, **kwargs): │
│ ❱ 39 │ │ │ return wrapper(*args, **kwargs) │
│ 40 │ │
│ 41 │ # We need to manually update this because `partial` object has no `__name__` and `__ │
│ 42 │ wrapper_function.__name__ = extract_function_name(wrapped) │
│ │
│ /.../.venv/lib/python3.12/site-packages/pydantic/_internal/_validate_call. │
│ py:136 in __call__ │
│ │
│ 133 │ │ if not self.__pydantic_complete__: │
│ 134 │ │ │ self._create_validators() │
│ 135 │ │ │
│ ❱ 136 │ │ res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, │
│ 137 │ │ if self.__return_pydantic_validator__: │
│ 138 │ │ │ return self.__return_pydantic_validator__(res) │
│ 139 │ │ else: │
│ │
│ /.../.venv/lib/python3.12/site-packages/sagemaker/core/resources.py:32025 │
│ in refresh │
│ │
│ 32022 │ │ logger.debug(f"Serialized input request: {operation_input_args}") │
│ 32023 │ │ │
│ 32024 │ │ client = Base.get_sagemaker_client() │
│ ❱ 32025 │ │ response = client.describe_processing_job(**operation_input_args) │
│ 32026 │ │ │
│ 32027 │ │ # deserialize response and update self │
│ 32028 │ │ transform(response, "DescribeProcessingJobResponse", self) │
│ │
│ /.../.venv/lib/python3.12/site-packages/botocore/client.py:602 in │
│ _api_call │
│ │
│ 599 │ │ │ │ │ f"{py_operation_name}() only accepts keyword arguments." │
│ 600 │ │ │ │ ) │
│ 601 │ │ │ # The "self" in this scope is referring to the BaseClient. │
│ ❱ 602 │ │ │ return self._make_api_call(operation_name, kwargs) │
│ 603 │ │ │
│ 604 │ │ _api_call.__name__ = str(py_operation_name) │
│ 605 │
│ │
│ /.../.venv/lib/python3.12/site-packages/botocore/context.py:123 in wrapper │
│ │
│ 120 │ │ │ with start_as_current_context(): │
│ 121 │ │ │ │ if hook: │
│ 122 │ │ │ │ │ hook() │
│ ❱ 123 │ │ │ │ return func(*args, **kwargs) │
│ 124 │ │ │
│ 125 │ │ return wrapper │
│ 126 │
│ │
│ /.../.venv/lib/python3.12/site-packages/botocore/client.py:1078 in │
│ _make_api_call │
│ │
│ 1075 │ │ │ │ 'error_code_override' │
│ 1076 │ │ │ ) or error_info.get("Code") │
│ 1077 │ │ │ error_class = self.exceptions.from_code(error_code) │
│ ❱ 1078 │ │ │ raise error_class(parsed_response, operation_name) │
│ 1079 │ │ else: │
│ 1080 │ │ │ return parsed_response │
│ 1081 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ClientError: An error occurred (ValidationException) when calling the DescribeProcessingJob operation: Could not
find requested job with name: glm-47--2026-0424-18-07-54
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: `sagemaker-core` v2.6.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: PyTorch
- **Framework version**: 2.8
- **Python version**: 3.12
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**

N/A

Contributor guide

Open the contributing guide

Research direction

Start by tracing FrameworkProcessor.run in sagemaker/core/processing.py through latest_job.wait, then inspect the processing-job resource's refresh path in sagemaker/core/resources.py. Reproduce the issue with a boto3 session set to ap-southeast-1 while the default region differs. Done means wait=True monitors and tails the job in the explicitly configured region without the DescribeProcessingJob error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.