traceloop / traceloop/openllmetry

๐Ÿ› Bug Report: Whole message history is stringified into a single 'user prompt'

Open
#2,513 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
7.4k
Forks
1.1k
Avg merge
8d 14h
Merged PRs (30d)
2

Description

Which component is this bug for?

VertexAI Instrumentation

๐Ÿ“œ Description

Passing a list of messages to generate_content combines them all into a single string and stores/presents them as a single user prompt instead of keeping them separate.

๐Ÿ‘Ÿ Reproduction steps
import vertexai
from traceloop.sdk import Traceloop
from vertexai.generative_models import Content, GenerativeModel, Part

Traceloop.init()
vertexai.init()
model = GenerativeModel('gemini-1.5-flash-002')
response = model.generate_content(
    [
        Content(
            role='user',
            parts=[
                Part.from_text("What's 2+2?"),
            ],
        ),
        Content(
            role='assistant',
            parts=[
                Part.from_text('5'),
            ],
        ),
        Content(
            role='user',
            parts=[
                Part.from_text('really?'),
            ],
        ),
    ]
)
๐Ÿ‘ Expected behavior

Attributes look something like:

{
  "gen_ai.prompt.0.role": "user",
  "gen_ai.prompt.0.content": "What's 2+2?\n",
  "gen_ai.prompt.1.role": "assistant",
  "gen_ai.prompt.1.content": "5\n",
  "gen_ai.prompt.2.role": "user",
  "gen_ai.prompt.2.content": "really?\n",
  "gen_ai.completion.3.role": "assistant",
  "gen_ai.completion.3.content": "Oops!  My apologies. 2 + 2 = 4.  I'm still under development and learning to perform these calculations correctly.\n"
}
๐Ÿ‘Ž Actual Behavior with Screenshots

Attributes look like:

{
  "gen_ai.prompt.0.user": "role: \"user\"\nparts {\n  text: \"What\\'s 2+2?\"\n}\n\nrole: \"assistant\"\nparts {\n  text: \"5\"\n}\n\nrole: \"user\"\nparts {\n  text: \"really?\"\n}\n\n",
  "gen_ai.completion.0.role": "assistant",
  "gen_ai.completion.0.content": "Oops!  My apologies. 2 + 2 = 4.  I'm still under development and learning to perform these calculations correctly.\n"
}
Screenshot 2025-01-14 at 16 50 21
๐Ÿค– Python Version

No response

๐Ÿ“ƒ Provide any additional context for the Bug.

https://github.com/traceloop/openllmetry/blob/d92a50e9f5662d215e5808c8059ca9732480343e/packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/__init__.py#L124-L138

๐Ÿ‘€ Have you spent some time to check if this bug has been raised before?
  • I checked and didn't find similar issue
Are you willing to submit PR?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up โ€” it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in opentelemetry/instrumentation/vertexai/init.py at lines 124-138, then run the provided VertexAI generate_content reproduction. Trace how the list of Content messages becomes prompt attributes and preserve each message's role and content separately. Done means the attributes match the expected prompt.0, prompt.1, and prompt.2 structure rather than one stringified user prompt.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.