NVIDIA-NeMo / NVIDIA-NeMo/Guardrails
not able to connect llamaguard with nemoguardrail
Open
@prasoonvarshney is already working on this.
Since May 24, 2024.
bug
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 842
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 25
Description
hi i am not able to connect my llamaguard api with nemoguardrail . below are the configs i have used . please help !!!
config :
models:
- type: main
engine: openai
model: gpt-3.5-turbo-instruct
- type: llama_guard
engine: vllm_openai
parameters:
openai_api_base: "https://localhost:8001/v1"
model_name: "meta-llama/LlamaGuard-7b"
rails:
input:
flows:
- llama guard check input
output:
flows:
- llama guard check output
chat.py file :
import os
from langchain_community.chat_models import AzureChatOpenAI
from nemoguardrails import LLMRails, RailsConfig
import pandas as pd
# Define LLM and parameters to pass to the guardrails configuration
chat_model = AzureChatOpenAI(****** )
# Load configuration
config = RailsConfig.from_path("./config")
# Configuration of LLMs is passed
app = LLMRails(config=config, llm=chat_model)
# sample user input
new_message = app.generate(messages=[{
"role": "user",
"content": "what is the length of the atm pin?"
}])
prompt file same as shown in the example.
command used to deploy model : python -m vllm.entrypoints.openai.api_server --model meta-llama/LlamaGuard-7b --port 8005 --dtype float16 --max-model-len 1500
logs show this :
/home/rndadmin/nemoguardrail/guard/lib/python3.10/site-packages/langchain_core/_api/deprecation.py:119: Lang ChainDeprecationWarning: The class `AzureChatOpenAI` was deprecated in LangChain 0.0.10 and will be removed in 0.3.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import AzureChatOpenAI `.
warn_deprecated(
/home/rndadmin/nemoguardrail/guard/lib/python3.10/site-packages/langchain_community/chat_models/azure_openai .py:167: UserWarning: As of openai>=1.0.0, Azure endpoints should be specified via the `azure_endpoint` para m not `openai_api_base` (or alias `base_url`). Updating `openai_api_base` from https://autogenaillmapi.opena i.azure.com/ to https://autogenaillmapi.openai.azure.com/openai.
warnings.warn(
/home/rndadmin/nemoguardrail/guard/lib/python3.10/site-packages/langchain_community/chat_models/azure_openai .py:174: UserWarning: As of openai>=1.0.0, if `deployment_name` (or alias `azure_deployment`) is specified t hen `openai_api_base` (or alias `base_url`) should not be. Instead use `deployment_name` (or alias `azure_de ployment`) and `azure_endpoint`.
warnings.warn(
Entered verbose mode.
Fetching 5 files: 100%|████████████████████████████████████████████████████| 5/5 [00:00<00:00, 69905.07it/s]
Event UtteranceUserActionFinished {'final_transcript': 'what is the length of the atm pin?'}
Event StartInternalSystemAction {'uid': '057f...', 'action_name': 'create_event', 'action_params': {'event':
{'_type': 'StartInputRails'}}, 'action_result_key': None, 'action_uid': 'f514...', 'is_system_action': True}
Executing action create_event
Event StartInputRails {'uid': '2728...'}
Event StartInternalSystemAction {'uid': 'ca3c...', 'action_name': 'create_event', 'action_params': {'event':
{'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid':
'9536...', 'is_system_action': True}
Executing action create_event
Event StartInputRail {'uid': '995b...', 'flow_id': 'llama guard check input'}
Event StartInternalSystemAction {'uid': '23a0...', 'action_name': 'llama_guard_check_input',
'action_params': {}, 'action_result_key': 'llama_guard_response', 'action_uid': '907d...',
'is_system_action': False}
Executing action llama_guard_check_input
Event hide_prev_turn {}
Total processing took 0.01 seconds. LLM Stats: 0 total calls, 0 total time, 0 total tokens, 0 total prompt
tokens, 0 total completion tokens, [] as latencies
{'role': 'assistant', 'content': "I'm sorry, an internal error has occurred."}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.