traceloop / traceloop/openllmetry
๐ Bug Report: incompatibilities with LLM semantics
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.4k
- Forks
- 1.1k
- Avg merge
- 8d 14h
- Merged PRs (30d)
- 2
Description
Which component is this bug for?
LLM Semantic Conventions
๐ Description
As a first timer, I tried the ollama instrumentation, and sent a trace to a local collector. Then I compared the output with llm semantics defined by otel. I noticed as many compatibilities as incompatibilities, and it made me concerned that other instrumentation may have other large glitches.
๐ Reproduction steps
use olllama-python with the instrumentation here. It doesn't matter if you use the traceloop-sdk or normal otel to initialize the instrumentation ( I checked both just in case)
๐ Expected behavior
otel specs should be a subset of openllmetry semantics, so no incompatible attributes.
๐ Actual Behavior with Screenshots
compatible:
- kind=client
- name=ollama.chat
- attributes['gen_ai.system']='Ollama'
- attributes['gen_ai.response.model']='codegemma:2b-code'
- attributes['gen_ai.usage.completion_tokens']=11
Incompatible:
- attributes['gen_ai.prompt.0.content']='prompt_text' otel semantics declare this as a non-indexed attribute 'gen_ai.prompt'
- attributes['gen_ai.completion.0.role']='assistant' otel semantics declare this as a non-indexed attribute 'gen_ai.request.model.role'
not yet defined in the standard:
- attributes['llm.request.type']='chat'
- attributes['llm.is_streaming']=false
- attributes['llm.usage.total_tokens']=11
๐ค Python Version
3.12
๐ Provide any additional context for the Bug.
partially addressed by @gyliu513 in https://github.com/traceloop/openllmetry/pull/884
๐ Have you spent some time to check if this bug has been raised before?
- I checked and didn't find similar issue
Are you willing to submit PR?
None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up โ it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ollama-python instrumentation and compare its emitted attributes with the OpenTelemetry LLM span semantic conventions linked in the issue. Review pull request #884 for the partial fix, then bring the reported incompatible attributes into alignment and verify that no incompatible attributes remain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ollama, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100