googleapis / googleapis/python-aiplatform

CreateReasoningEngine fails with code 13 after deleting previous resource (Dockerfile/image_spec path, us-central1)

Aperta
#6,754 0 commenti 3 reazioni 1 assegnatario Rivendicata da @maxgasztych Vedi su GitHub
api: vertex-ai
Lingua principale
Python
Stelle
905
Fork
465
Merge medio
1g 13h
PR unite (30g)
44

Descrizione

#### Environment details

- OS type and version: macOS 24.6.0 (darwin arm64)
- Python version: N/A (using REST API directly via curl, not the Python SDK)
- pip version: N/A
- `google-cloud-aiplatform` version: N/A — REST API v1beta1

#### Summary

After successfully deploying a Dockerfile-based reasoning engine via the REST API (`image_spec: {}` + `inline_source.source_archive`), deleting that resource with `force=true`, and then attempting to create a new one, **all subsequent `CreateReasoningEngine` operations fail with code 13** in `us-central1`. The same request succeeds in `us-east4` within the same project.

Cloud Logging confirms the build completes and the container starts healthy — the failure is in Agent Engine's internal post-deploy verification.

#### Steps to reproduce

1. Deploy a Dockerfile-based reasoning engine via REST API to `us-central1`:
```bash
curl -X POST \
"https://us-central1-aiplatform.googleapis.com/v1beta1/projects/{PROJECT}/locations/us-central1/reasoningEngines" \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{
"display_name": "my-dockerfile-agent",
"spec": {
"source_code_spec": {
"inline_source": { "source_archive": "" },
"image_spec": {}
},
"agent_framework": "custom",
"class_methods": [{"name": "query", "api_mode": ""}],
"deployment_spec": {
"env": [{"name": "SOME_VAR", "value": "some_value"}],
"min_instances": 1,
"max_instances": 1,
"resource_limits": {"cpu": "4", "memory": "8Gi"}
}
}
}'
```
Result: **Success** — operation completes, resource created, `:query` works.

2. Delete the reasoning engine:
```bash
curl -X DELETE \
"https://us-central1-aiplatform.googleapis.com/v1beta1/projects/{PROJECT}/locations/us-central1/reasoningEngines/{RESOURCE_ID}?force=true" \
-H "Authorization: Bearer $(gcloud auth print-access-token)"
```
Result: Delete succeeds (`done: true`).

3. Create a new reasoning engine with the same or different payload:
```bash
# Same curl as step 1, different display_name
```
Result: **Fails with code 13** every time.

4. Deploy the **exact same payload** to `us-east4` in the same project:
```bash
# Same curl but with us-east4 in the URL
```
Result: **Success** — deploys fine, container starts, `:query` works.

#### Observed behavior

- The operation is accepted and returns an operation name
- Cloud Logging (`reasoning_engine_build`) shows the Dockerfile build completing successfully ("DONE", image pushed with SHA digest)
- Cloud Logging (`reasoning_engine_stdout`) shows the container starting and logging that it's listening on port 8080
- Despite the container being healthy, the operation completes with code 13

#### Expected behavior

`CreateReasoningEngine` should succeed since the build completes and the container starts healthy. Deleting and recreating a reasoning engine should not permanently break the region for the project.

#### Minimal Dockerfile used for testing

```dockerfile
FROM node:22-slim
WORKDIR /app
COPY server.js ./
CMD ["node", "server.js"]
```

```javascript
// server.js
const http = require("http");
http.createServer((req, res) => {
if (req.url === "/ping") { res.end(JSON.stringify({status:"ok"})); return; }
let body = "";
req.on("data", c => body += c);
req.on("end", () => {
res.writeHead(200, {"content-type":"application/json"});
res.end(JSON.stringify({output: "echo: " + body}));
});
}).listen(8080, () => console.log("listening on 8080"));
```

Even this minimal 2-file container fails with code 13 in `us-central1` after the delete, but deploys fine in `us-east4`.

#### Error response

```json
{
"name": "projects/{NUMBER}/locations/us-central1/reasoningEngines/{ID}/operations/{OP_ID}",
"metadata": {
"@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.CreateReasoningEngineOperationMetadata",
"genericMetadata": {
"createTime": "2026-05-07T23:59:49.660727Z",
"updateTime": "2026-05-07T23:59:49.660727Z"
}
},
"done": true,
"error": {
"code": 13,
"message": "Please refer to our documentation (https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/troubleshooting/deploy) for checking logs and other troubleshooting tips."
}
}
```

#### Additional context

- Region: `us-central1` is broken, `us-east4` works — same project, same payload, same permissions
- The issue started immediately after deleting a previously deployed reasoning engine
- All IAM roles verified (reasoningEngineServiceAgent, artifactregistry.reader, storage.objectAdmin, logging.logWriter)
- Staging bucket exists and is accessible
- Cloud Resource Manager API is enabled
- No VPC-SC configured
- Multiple retries over 2+ hours — issue does not self-heal
- Operations cannot be cancelled via the API ("not cancellable")

#### Hypothesis

Deleting the reasoning engine left orphaned internal state (Cloud Run revision, internal AR image reference, or routing configuration) in `us-central1` that blocks new reasoning engine deployments from completing their post-deploy verification step. The build and container startup succeed, but the orchestration layer's readiness check fails against stale internal state.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.