googleapis / googleapis/nodejs-agentplatform

Using Provisioned Throughput for base model gemini-1.5-pro-002 through v1.9.2

Open
#483 1 comment 0 reactions 0 assignees View on GitHub
api: aiplatform api: billingbudgets api: cloudbilling priority: p3 type: question
Dominant language
TypeScript
Stars
182
Forks
69
Avg merge
1d 8h
Merged PRs (30d)
10

Description

I'm calling gemini-1.5-pro-002 base model through this library and after purchasing Provisioned Throughput, all my calls to Gemini are still using shared resources instead of a dedicated one.

The code is pretty simple and straightforward:

```
const vertex_ai = new VertexAI({
project: projectId.value(),
location: location.value(),
});

const req: GenerateContentRequest = {
contents: [{ role: 'user', parts: [fullPrompt] }],
};

const generativeModel = vertex_ai.preview.getGenerativeModel({
model: 'gemini-1.5-pro-002',
safetySettings: [ ... ],
});

const streamingResp = await generativeModel.generateContentStream(req);
const response = await streamingResp.response;
```

According to PT (Provisioned Throughput) documentation, I would not need to change my code to get the default behavior (PT and overages on pay-as-you-go basis).
PT doc: https://cloud.google.com/vertex-ai/generative-ai/docs/provisioned-throughput

However, by checking the metrics in Google Console, all my requests are still using shared resources.
Metric: aiplatform.googleapis.com/publisher/online_serving/consumed_throughput
request_type is always shared

I haven't created an endpoint on my project because I want to use the base model gemini-1.5-pro-002. I tried to create an endpoint in my project for the base model but it's not possible neither on the console nor using gcloud commands.

Is it something wrong with my server setup? The PT doc mention it can be used with Base models.
Do I need to force PT somehow on this library?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.