NVIDIA-NeMo / NVIDIA-NeMo/Guardrails

bug: Differences between OpenAi and Gemini

Open
#900 1 comment 0 reactions 1 assignee View on GitHub

@schuellc-nvidia is already working on this.

Since Jan 7, 2025.

bug enhancement
Dominant language
Python
Stars
7.2k
Forks
843
Avg merge
3d 1h
Merged PRs (30d)
25

Description

Did you check docs and existing issues?
  • I have read all the NeMo-Guardrails docs
  • I have updated the package to the latest version before submitting this issue
  • (optional) I have used the develop branch
  • I have searched the existing issues of NeMo-Guardrails
Python version (python --version)

3.10.15

Operating system/version

MacOs 14.6.1

NeMo-Guardrails version (if you must use a specific version and not the latest

No response

Describe the bug

Hi, thank you for you work.
I noticed that i obtain different results with the response generator ("..." in Colang v2) when i use different providers.
If I use OpenAi, I obtain the desired response, while if i use Gemini, I receive a syntax error. Simply by changing ONLY the config file.
I am testing this using two different nemo servers: action and non action one. But they work fine and locally I have the same problem.
For the config files, i used the following ones:

colang_version: 2.x
models:
  - type: main
    engine: vertexai
    model: gemini-1.5-pro

colang_version: 2.x
models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo-instruct

Using prints and logs as debug, I noticed that the error is in the line $output = ..."'{$ref_use}'"
I have tried also variation of way to send the input (for instance: $ref_use.transcript), but they did not work.
This is the error with gemini:

# This is the current conversation between the user and the bot:                                                                                                                                           
user action: user said "Hi! Can you do the spelling of the following name <PERSON>? Thanks"                                                                                                                
                                                                                                                                                                                                           
# 'Hi! Can you do the spelling of the following name <PERSON>? Thanks'                                                                                                                                     
$output =                                                                                                                                                                                                  

/Users/vrige/Library/Caches/pypoetry/virtualenvs/llm-gateway-C7tEgGBA-py3.10/lib/python3.10/site-packages/proto/message.py:389: DeprecationWarning: The argument `including_default_value_fields` has been removed from
                Protobuf 5.x. Please use `always_print_fields_with_no_presence` instead.
                
  warnings.warn(

LLM Completion (d9093..)
user intent: user asked for spelling of a name                                                                                                                                                             

17:47:27.685 | Output Stats None
17:47:27.685 | LLM call took 2.36 seconds
WARNING:nemoguardrails.actions.action_dispatcher:Error while execution 'GenerateValueAction' with parameters '{'var_name': 'output', 'instructions': "'Hi! Can you do the spelling of the following name <PERSON>? Thanks'"}': Invalid LLM response: `user intent: user asked for spelling of a name`
ERROR:nemoguardrails.actions.action_dispatcher:Invalid LLM response: `user intent: user asked for spelling of a name`
Traceback (most recent call last):
  File "/Users/vrige/Library/Caches/pypoetry/virtualenvs/llm-gateway-C7tEgGBA-py3.10/lib/python3.10/site-packages/nemoguardrails/actions/v2_x/generation.py", line 813, in generate_value
    return literal_eval(value)
  File "/Users/vrige/.pyenv/versions/3.10.15/lib/python3.10/ast.py", line 64, in literal_eval
    node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
  File "/Users/vrige/.pyenv/versions/3.10.15/lib/python3.10/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    user intent: user asked for spelling of a name
         ^^^^^^
SyntaxError: invalid syntax

The input message is the following one:
'Hi! Can you do the spelling of the following name <PERSON>? Thanks'

Finally, the simpleAction makes a custom validation of the output, but it works fine.

I need to work with Gemini, is there a way to overcome this problem?

Steps To Reproduce
import core
import llm

flow main
    user said something as $ref_use
    $output = ..."'{$ref_use}'"
    await analyse_output(ref_use=$output) as $ref_act_out

flow analyse_output $ref_use
    $result = await simpleAction(inputs=$ref_use)
    if "Error: " in $result
        bot say "I do not like it. Change your input"
    else
        bot say $result

UPDATE: I tried also the following simplified flow, but I got the same error.

import core
import llm

flow main
    user said something as $ref_use
    $output = ..."'{$ref_use}'"
Expected Behavior

The OpenAi response is the following one:
Sure, the spelling of the name << PERSON >> is <US_ITIN>. Is there anything else I can assist you with?

Actual Behavior

This is the error with gemini:

# This is the current conversation between the user and the bot:                                                                                                                                           
user action: user said "Hi! Can you do the spelling of the following name <PERSON>? Thanks"                                                                                                                
                                                                                                                                                                                                           
# 'Hi! Can you do the spelling of the following name <PERSON>? Thanks'                                                                                                                                     
$output =                                                                                                                                                                                                  

/Users/vrige/Library/Caches/pypoetry/virtualenvs/llm-gateway-C7tEgGBA-py3.10/lib/python3.10/site-packages/proto/message.py:389: DeprecationWarning: The argument `including_default_value_fields` has been removed from
                Protobuf 5.x. Please use `always_print_fields_with_no_presence` instead.
                
  warnings.warn(

LLM Completion (d9093..)
user intent: user asked for spelling of a name                                                                                                                                                             

17:47:27.685 | Output Stats None
17:47:27.685 | LLM call took 2.36 seconds
WARNING:nemoguardrails.actions.action_dispatcher:Error while execution 'GenerateValueAction' with parameters '{'var_name': 'output', 'instructions': "'Hi! Can you do the spelling of the following name <PERSON>? Thanks'"}': Invalid LLM response: `user intent: user asked for spelling of a name`
ERROR:nemoguardrails.actions.action_dispatcher:Invalid LLM response: `user intent: user asked for spelling of a name`
Traceback (most recent call last):
  File "/Users/vrige/Library/Caches/pypoetry/virtualenvs/llm-gateway-C7tEgGBA-py3.10/lib/python3.10/site-packages/nemoguardrails/actions/v2_x/generation.py", line 813, in generate_value
    return literal_eval(value)
  File "/Users/vrige/.pyenv/versions/3.10.15/lib/python3.10/ast.py", line 64, in literal_eval
    node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
  File "/Users/vrige/.pyenv/versions/3.10.15/lib/python3.10/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    user intent: user asked for spelling of a name
         ^^^^^^
SyntaxError: invalid syntax

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.