googleapis / googleapis/python-aiplatform
Adding examples for correct prompt structure for custom endpoints on Vertex AI
- Dominant language
- Python
- Stars
- 905
- Forks
- 465
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 44
Description
Thanks for stopping by to let us know something could be better!
**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":
- Search the issues already opened: https://github.com/googleapis/python-aiplatform/issues
- Search StackOverflow: https://stackoverflow.com/questions/tagged/google-cloud-platform+python
If you are still having issues, please be sure to include as much information as possible:
#### Environment details
- OS type and version: Jupyter Notebook 6.5.5 on Colab enterprise
- Python version: `python --version` Python 3.10.12
- pip version: `pip --version`
- `google-cloud-aiplatform` version: `pip show google-cloud-aiplatform`
#### Steps to reproduce
1. Add Python def with the correct promot structure to pass on to open source models. I.e Llama 3.1
2. Add example REST API payload to prompt deployed endpoint on Vertex AI
#### Code example for Llama 3.1 managed infra endpoints
```python
prompt_payload = " What is the capital of Paris?" # @param {type:"string"}
prompt ="<|start_header_id|>user<|end_header_id|>"+prompt_payload+"<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
max_tokens = 8000 # @param {type:"integer"}
temperature = 1.0 # @param {type:"number"}
top_p = 1.0 # @param {type:"number"}
top_k = 1.0 # @param {type:"number"}
raw_response = True # @param {type:"boolean"}
# Overides parameters for inferences.
# If you encounter the issue like `ServiceUnavailable: 503 Took too long to respond when processing`,
# you can reduce the max length, such as set max_tokens as 20.
instances = [
{
"prompt": prompt,
"max_tokens": max_tokens,
"temperature": temperature,
"top_p": top_p,
"top_k": top_k,
"raw_response": raw_response
}
]
predict_custom_trained_model_sample(
project="yourprojectid",
endpoint_id="yourprojectendpoint",
instances=instances
)
```
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!
Contributor guide
Research direction
Start by locating the predict_custom_trained_model_sample entry point and the existing custom endpoint examples. Review the requested Llama 3.1 prompt structure and REST API payload, then add examples showing both forms for Vertex AI managed infrastructure endpoints. Done means users can follow the examples to construct the prompt and payload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100