GoogleCloudPlatform / GoogleCloudPlatform/generative-ai
[Feat]: Batch Prediction and Cached Prompt should work together
- Dominant language
- Jupyter Notebook
- Stars
- 17.7k
- Forks
- 4.5k
- Avg merge
- 12h 38m
- Merged PRs (30d)
- 42
Description
### Is your feature request related to a problem? Please describe.
Batch predictions should be able to use a cached context.
### Describe the solution you'd like
We are currently working on a classifying prompt which requires an extensive system prompt. We have been experimenting using batch processing together with context caching (leveraging the 50% discount from batch processing, with the 75% discount from context caching).
We have tried several approaches now, but the batch job fails. Here's and entry from the responding `predictions.jsonl`:
```json
{"status":"Internal error occurred. Failed to get generateContentResponse: {\"error\": {\"code\": 404, \"message\": \"Not found: cached content metadata for 3814716010150232064.\", \"status\": \"NOT_FOUND\"}}"
```
* There is no official Google material (tutorials, documentation, model cards, etc) stating if batch predicitons and context caching is supported when used together.
* If it is supported, it would be lovely to see a tutorial.
* If it is NOT supported, please add a disclaimer to the batch / context cache docs.
### Describe alternatives you've considered
It is supported on OpenAI.
### Additional context
_No response_
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start by reproducing the reported batch prediction and cached context failure, then review the batch prediction and context caching documentation referenced by the issue. Done means establishing whether the combination is supported and documenting the result with either a tutorial or a clear disclaimer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud
- Domain
- ai, cloud, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100