Issues with OTLP exporter with None values

Describe your environment

OS: Ubuntu
Python version: 3.12
SDK version: opentelemetry-sdk==1.36.0
API version: opentelemetry-api==1.36.0
Celery: opentelemetry-instrumentation-celery==0.57b0

What happened?

I am using CeleryInstrumentor() to get logs of celery into Grafana/Loki

First of the issues I get is that I see an incompatibility with the latest opentelemetry-sdk (1.41) version with opentelemetry-api (it does not allow me to install).

Second issue is with None values in the celery results exported to Grafana. I am seeing a warning like:
Invalid type <class 'dict'> of value {'id': 'b2f12a15-b57d-4e9a-a439-345a323e7c1b', 'name': 'app.tasks.tasks.say_hello', 'args': '()', 'kwargs': '{}', 'eta': None}

I noticed that in opentelemetry 1.41 you already updated the encoder: https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-common/src/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py

But I am uncapable of setting allow_null to True for CeleryInstrumentor().

Steps to Reproduce

Setup CeleryInstrumentor() and have a task that retuns None values somewhere.
In my case:

# API route to trigger the task
@bp.route('/long_task', methods=['GET'])
def trigger_task():
    task = say_hello.delay()
    return jsonify({"task_id": task.id, "status": "Task has been submitted"}), 202

@celery.task
def say_hello():
    sleep(10)
    logger.info(f"[HEALTHCHECK] say_hello config_class={type(Config).__name__} service_name={Config.SERVICE_NAME}")
    return "Hello, World!"

I also have a custom logger setup, which might be the one causing None to get inserted over the values.

Expected Result

No errors or warning escalated to Loki/Grafana

Actual Result

I monkey patched _encode_key_value:

def _encode_key_value(key: str, value: _Any, allow_null: bool = False) -> _PB2KeyValue:
if value is None:
return _PB2KeyValue(key=key, value=_encode_value(value, allow_null=True))
return _PB2KeyValue(key=key, value=_encode_value(value, allow_null=allow_null))

Additional context

I am unsure if this is a proper solution, I guess not and that it should be configured and CeleryInstrumentor some setting that allows to pass the allow_none=True, but I do not know how to do it.

Would you like to implement a fix?

None

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

Start with the OTLP encoder at exporter/opentelemetry-exporter-otlp-proto-common/src/opentelemetry/exporter/otlp/proto/common/_internal/init.py and trace how CeleryInstrumentor() exports task results and log values. Reproduce the None-valued result with the provided Celery task, then verify that the exporter handles it without warnings or errors and that the SDK/API version incompatibility is addressed or clearly separated.