googleapis / googleapis/python-aiplatform
Adding examples for correct prompt structure for custom endpoints on Vertex AI
- Lingua principale
- Python
- Stelle
- 905
- Fork
- 465
- Merge medio
- 1g 13h
- PR unite (30g)
- 44
Descrizione
Thanks for stopping by to let us know something could be better!
**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":
- Search the issues already opened: https://github.com/googleapis/python-aiplatform/issues
- Search StackOverflow: https://stackoverflow.com/questions/tagged/google-cloud-platform+python
If you are still having issues, please be sure to include as much information as possible:
#### Environment details
- OS type and version: Jupyter Notebook 6.5.5 on Colab enterprise
- Python version: `python --version` Python 3.10.12
- pip version: `pip --version`
- `google-cloud-aiplatform` version: `pip show google-cloud-aiplatform`
#### Steps to reproduce
1. Add Python def with the correct promot structure to pass on to open source models. I.e Llama 3.1
2. Add example REST API payload to prompt deployed endpoint on Vertex AI
#### Code example for Llama 3.1 managed infra endpoints
```python
prompt_payload = " What is the capital of Paris?" # @param {type:"string"}
prompt ="<|start_header_id|>user<|end_header_id|>"+prompt_payload+"<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
max_tokens = 8000 # @param {type:"integer"}
temperature = 1.0 # @param {type:"number"}
top_p = 1.0 # @param {type:"number"}
top_k = 1.0 # @param {type:"number"}
raw_response = True # @param {type:"boolean"}
# Overides parameters for inferences.
# If you encounter the issue like `ServiceUnavailable: 503 Took too long to respond when processing`,
# you can reduce the max length, such as set max_tokens as 20.
instances = [
{
"prompt": prompt,
"max_tokens": max_tokens,
"temperature": temperature,
"top_p": top_p,
"top_k": top_k,
"raw_response": raw_response
}
]
predict_custom_trained_model_sample(
project="yourprojectid",
endpoint_id="yourprojectendpoint",
instances=instances
)
```
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.