googleapis / googleapis/python-aiplatform

Specifying artifact regististry images with tags fails

Đang mở
#2,181 7 bình luận 0 reaction 0 người được giao Xem trên GitHub
api: vertex-ai type: bug
Ngôn ngữ chính
Python
Star
905
Fork
465
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
44

Mô tả

Creating a PipelineJob based on a artifact registry URI using a **tag** instead of a **version** raises an internal server error.
This is pretty annoying.

### to be clear:

- Using an artifact with sha256 version number works:
https://europe-west1-kfp.pkg.dev/my-project/my-project-kfp-template-repository/my-pipeline/sha256:3456789874334a...

- Using a tag gives a 500 internal server error:
https://europe-west1-kfp.pkg.dev/my-project/my-project-kfp-template-repository/my-pipeline/my-tag

- Using a tag that doesn't exist gives a 404 error when creating the PipelineJob
https://europe-west1-kfp.pkg.dev/my-project/my-project-kfp-template-repository/my-pipeline/my-non-existent-tag

All of this combined makes it seem like there's a bug for executing tagged artifacts.

#### Environment details

- OS type and version: Ubuntu 22.04.2 LTS
- Python version: Python 3.9.16
- pip version: pip 22.0.4
- `google-cloud-aiplatform` version: 1.24.1

#### Steps to reproduce

1. Create a PipelineJob with a artifact repository URI using a tag instead of the hash
2. Run the pipeline job

#### Code example

This fails with a 500 internal server error on the "run" call.
```python
from google.cloud import aiplatform as aip

service_account = ...
pipeline_name = ...
job_id = f"{pipeline_name}-{datetime.datetime.now().strftime('%Y%m%d%H%M%S')}"

compiled_job = f"https://{region}-kfp.pkg.dev/{PROJECTID}/{REPOSITORY_NAME}/{PIPELINE_NAME}/{TAG}"
pipeline_job = aip.PipelineJob(
display_name="test-name",
job_id=job_id,
template_path=compiled_job,
)

pipeline_job.run(network=None,
service_account=service_account,
sync=True)
```

This runs without problem:

```python
from google.cloud import aiplatform as aip

service_account = ...
pipeline_name = ...
job_id = f"{pipeline_name}-{datetime.datetime.now().strftime('%Y%m%d%H%M%S')}"

compiled_job = f"https://{region}-kfp.pkg.dev/{PROJECTID}/{REPOSITORY_NAME}/{PIPELINE_NAME}/sha:...."
pipeline_job = aip.PipelineJob(
display_name="test-name",
job_id=job_id,
template_path=compiled_job,
)

pipeline_job.run(network=None,
service_account=service_account,
sync=True)
```

#### Stack trace
```
Traceback (most recent call last):
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/grpc/_channel.py", line 1030, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Internal error encountered."
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B2a00:1450:400e:800::200a%5D:443 {created_time:"2023-05-09T00:09:30.434205595+02:00", grpc_status:13, grpc_message:"Internal error encountered."}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/.../Desktop/test/mlpipelines/monthly_pipeline.py", line 98, in
pipeline_job.run(network=None,
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/google/cloud/aiplatform/pipeline_jobs.py", line 314, in run
self._run(
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 814, in wrapper
return method(*args, **kwargs)
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/google/cloud/aiplatform/pipeline_jobs.py", line 345, in _run
self.submit(
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/google/cloud/aiplatform/pipeline_jobs.py", line 419, in submit
self._gca_resource = self.api_client.create_pipeline_job(
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/pipeline_service/client.py", line 1347, in create_pipeline_job
response = rpc(
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
return wrapped_func(*args, **kwargs)
File "/home/.../Desktop/test/venv/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InternalServerError: 500 Internal error encountered.
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.