Vertex AI Pipelines using TPUs Internal Error encountered

#### Environment details

- Python version: 3.11
- pip version: 23.1
- `google-cloud-aiplatform` version: 1.25.0
- kfp=2.0.0rc1 (all versions tested up to 2.0.0b16)

#### Steps to reproduce

1. This pipeline:
```
from kfp import dsl
import google_cloud_pipeline_components.v1.custom_job as gcp_components_custom_job

@dsl.component
def print_test():
print("Hello World!")

# Using TPUs

printOp = gcp_components_custom_job.create_custom_training_job_from_component(
print_test,
display_name="Print Test",
machine_type="cloud-tpu",
accelerator_type="TPU_V3",
accelerator_count=8,
)

@dsl.pipeline
def pipeline():
print_task = printOp(
project="REDACTED",
location="us-central1",
)

if __name__ == "__main__":
import kfp.compiler as compiler

compiler.Compiler().compile(pipeline, "problem_pipeline.yaml")
```

2. send pipeline to run in vertex

I end up with 40 min laters with a lot of internal errors and contact Vertex:


Thanks!

I have the quota for this machines and I have no idea what more to provide.

Start with the provided Python pipeline and the call to compiler.Compiler().compile, then run the compiled problem_pipeline.yaml in Vertex AI using the stated TPU settings. Compare the run output and internal errors with the environment versions and reproduction steps; done means the TPU pipeline runs without the reported internal error or the failure is narrowed to a documented external service issue.

Vom Indexierungsmodell aus dem Issue-Text verfasst.