GoogleCloudPlatform / GoogleCloudPlatform/generative-ai

[Feat]: Batch Prediction and Cached Prompt should work together

Open
#2,122 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
17.7k
Forks
4.5k
Avg merge
12h 38m
Merged PRs (30d)
42

Description

### Is your feature request related to a problem? Please describe.

Batch predictions should be able to use a cached context.

### Describe the solution you'd like

We are currently working on a classifying prompt which requires an extensive system prompt. We have been experimenting using batch processing together with context caching (leveraging the 50% discount from batch processing, with the 75% discount from context caching).

We have tried several approaches now, but the batch job fails. Here's and entry from the responding `predictions.jsonl`:

```json
{"status":"Internal error occurred. Failed to get generateContentResponse: {\"error\": {\"code\": 404, \"message\": \"Not found: cached content metadata for 3814716010150232064.\", \"status\": \"NOT_FOUND\"}}"
```

* There is no official Google material (tutorials, documentation, model cards, etc) stating if batch predicitons and context caching is supported when used together.
* If it is supported, it would be lovely to see a tutorial.
* If it is NOT supported, please add a disclaimer to the batch / context cache docs.

### Describe alternatives you've considered

It is supported on OpenAI.

### Additional context

_No response_

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported batch prediction and cached context failure, then review the batch prediction and context caching documentation referenced by the issue. Done means establishing whether the combination is supported and documenting the result with either a tutorial or a clear disclaimer.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud
Domain
ai, cloud, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.