traceloop / traceloop/openllmetry

๐Ÿ› Bug Report: incompatibilities with LLM semantics

Open
#1,455 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.4k
Forks
1.1k
Avg merge
8d 14h
Merged PRs (30d)
2

Description

Which component is this bug for?

LLM Semantic Conventions

๐Ÿ“œ Description

As a first timer, I tried the ollama instrumentation, and sent a trace to a local collector. Then I compared the output with llm semantics defined by otel. I noticed as many compatibilities as incompatibilities, and it made me concerned that other instrumentation may have other large glitches.

๐Ÿ‘Ÿ Reproduction steps

use olllama-python with the instrumentation here. It doesn't matter if you use the traceloop-sdk or normal otel to initialize the instrumentation ( I checked both just in case)

๐Ÿ‘ Expected behavior

otel specs should be a subset of openllmetry semantics, so no incompatible attributes.

๐Ÿ‘Ž Actual Behavior with Screenshots

compatible:

  • kind=client
  • name=ollama.chat
  • attributes['gen_ai.system']='Ollama'
  • attributes['gen_ai.response.model']='codegemma:2b-code'
  • attributes['gen_ai.usage.completion_tokens']=11

Incompatible:

  • attributes['gen_ai.prompt.0.content']='prompt_text' otel semantics declare this as a non-indexed attribute 'gen_ai.prompt'
  • attributes['gen_ai.completion.0.role']='assistant' otel semantics declare this as a non-indexed attribute 'gen_ai.request.model.role'

not yet defined in the standard:

  • attributes['llm.request.type']='chat'
  • attributes['llm.is_streaming']=false
  • attributes['llm.usage.total_tokens']=11
๐Ÿค– Python Version

3.12

๐Ÿ“ƒ Provide any additional context for the Bug.

partially addressed by @gyliu513 in https://github.com/traceloop/openllmetry/pull/884

๐Ÿ‘€ Have you spent some time to check if this bug has been raised before?
  • I checked and didn't find similar issue
Are you willing to submit PR?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up โ€” it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ollama-python instrumentation and compare its emitted attributes with the OpenTelemetry LLM span semantic conventions linked in the issue. Review pull request #884 for the partial fix, then bring the reported incompatible attributes into alignment and verify that no incompatible attributes remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.