Current handling of OTEL_EXPORTER_OTLP_TIMEOUT is not in line with spec

Describe your environment

No response

What happened?

In OTLPExporterMixin.__init__ the environment variable OTEL_EXPORTER_OTLP_TIMEOUT looks like is handled as seconds while the SDK says it has to be in milliseconds (https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_timeout) and the compliance matrix says the Python implementation is compliant.

The description in opentelemetry-sdk/src/opentelemetry/sdk/environment_variables.py does not mention the exact unit.

Since the timeout is currently handed over as timeout to grpc.experimental.unary_unary, which is documented like this:

      timeout: An optional duration of time in seconds to allow for the RPC,
        after which an exception will be raised. If timeout is unspecified,
        defaults to a timeout controlled by the
        GRPC_PYTHON_DEFAULT_TIMEOUT_SECONDS environment variable. If that is
        unset, defaults to 60 seconds. Supply a value of None to indicate that
        no timeout should be enforced.

To my understanding this only an effect in case there is a connection made, but no answer on the RPC sent back, which might not happen very often.

Steps to Reproduce

Start a server.py which accepts the client but does not respond:

#!/usr/bin/env python3

import time
from concurrent import futures

import grpc
from opentelemetry.proto.collector.trace.v1.trace_service_pb2_grpc import add_TraceServiceServicer_to_server, TraceServiceServicer

class NoResponseTraceService(TraceServiceServicer):
    def Export(self, request, context):
        while True:
            time.sleep(1)  # Sleep forever

server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
add_TraceServiceServicer_to_server(NoResponseTraceService(), server)

server.add_insecure_port('[::]:55680')
server.start()

try:
    while True:
        time.sleep(86400)
except KeyboardInterrupt:
    server.stop(0)

Use this test client.py:

#!/usr/bin/env python3

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())

trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:55680", insecure=True))
)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("Hello") as span:
    span.set_attribute("language", "Python")

Preparation: Start the server

Test 1: Start the client without environment variable. It uses the default timeout of 10 seconds per try.

> python3 client.py 2>&1 | ts
Jul 11 08:55:59 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 1s.
Jul 11 08:56:10 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 2s.
Jul 11 08:56:22 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 4s.
Jul 11 08:56:36 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 8s.
Jul 11 08:56:54 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 16s.
Jul 11 08:57:20 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 32s.

Test 2: Start the client with the environment variable. The time waiting for the export call is now 5 seconds instead of 10 seconds. But according to the spec it should be 5ms with this value.

> OTEL_EXPORTER_OTLP_TIMEOUT=5 python3 client.py 2>&1 | ts
Jul 11 08:58:56 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 1s.
Jul 11 08:59:02 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 2s.
Jul 11 08:59:09 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 4s.
Jul 11 08:59:18 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 8s.
Jul 11 08:59:31 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 16s.
Jul 11 08:59:52 Transient error StatusCode.DEADLINE_EXCEEDED encountered while exporting traces to localhost:55680, retrying in 32s.

Expected Result

The Python implementation is in line with the spec and the documentation is clear about the unit.

Actual Result

Setting OTEL_EXPORTER_OTLP_TIMEOUT=1000 sets a timeout of 1000 seconds instead of 1 second.

Additional context

No response

Would you like to implement a fix?

None

Start at OTLPExporterMixin.init to trace how OTEL_EXPORTER_OTLP_TIMEOUT reaches the gRPC export call, then read its description in opentelemetry-sdk/src/opentelemetry/sdk/environment_variables.py. Compare both paths with the linked specification; done means the Python implementation uses the specified unit and the environment-variable documentation states that unit clearly.