Add prompt and completion information to spans

Nobody has claimed this yet.

Prompt and completion data being added to spans (to Observations, as high cardinality data) was previously implemented but we removed it in 1.0.0-RC1 and moved the data to logs due to several issues:

The implementation directly used the OTel SDK so it did not work with other tracers
The implementation used deprecated APIs (of the OTel SDK)
The OTel Semantic conventions were also deprecated and instead of adding this data to spans, it moved them to logs
The data was not limited in size which could have been problematic for backends that receive that data
This did not work for ChatClient in reactive/streaming scenarios

See connected issues/changes/PRs:

I believe we should be able to implement this in a way that eliminates most of the issues above:

We can use the Observation API instead of the OTel SDK so this can work with different backends and no deprecated APIs will be used
The Langfuse documentation being stable or not and OTel SemConv not being stable makes this a moving target, not sure there is a good solution here, making the keys configurable is probably good enough?
We can limit (configurable) the size of the prompt/completion that is attached to the Observation
Getting prompt/completion in the reactive/streaming use-case was fixed by #4417 in 1.1.0

Older proof of concept: promt-on-span (also contains changes that were already merged in #4417).

Start by reviewing the Observation API approach described in the issue, the changes in PR #4417, and the older promt-on-span proof of concept. Define how configurable attribute keys and prompt/completion size limits should work, then verify support for reactive and streaming use cases without relying directly on the OTel SDK.

Written by the indexing model from the issue text.

Description

Contributor guide

First steps

Research direction

Assessment