Support OpenTelemetry Auto-Instrumentation, Distros & Configurators

### Description

## Background

In attempting to leverage the auto-instrumentation mechanism with a vendored distribution , I have run into a few issues. I believe with a few small tweaks this could be much polished and remove the need for workarounds.

## Setup

Auto instrumentation has two ways of being invoked, one using the command line wrapper `opentelemetry-instrument` and the other explicitly invoking it in code. Some of the issues identified later in this document force the latter approach.

Placing the following at the top of `airflow_local_settings.py` achieves autoinstrumentation:

```python
from opentelemetry.instrumentation import auto_instrumentation

auto_instrumentation.initialize()

from airflow._shared.observability.traces import OverrideableRandomIdGenerator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import get_tracer_provider

if isinstance((tracer_provider := get_tracer_provider()), TracerProvider):
tracer_provider.id_generator = OverrideableRandomIdGenerator()
```

## Issue 1: OverrideableRandomIdGenerator

*Change Difficulty: Easy*

This class was introduced in #63839. My understanding is this a key piece in the task span propagation strategy, but also looks alot like an anti-pattern: creating a span, just to throw it away, and re-use it's IDs later. This is not a blocker. See my comment requesting a more detailed explanation https://github.com/apache/airflow/pull/63839#discussion_r3544019341.

The workaround is to explicitly override any Distro's ID Generator, e.g.

```python
tracer_provider.id_generator = OverrideableRandomIdGenerator()
```

This class is marked private.

OpenTelemetry offers an environment variable to choose your ID generator, so long as the class is a defined entry point.

Could it be exposed using a Python Entry Point so that others could enforce it via ENV VAR such that I wouldn't have to reference it directly?

* [Upstream ID Generator Creation](https://github.com/open-telemetry/opentelemetry-python/blob/634cec5f2a2fecb40cb9d8216888c7b8865b845a/opentelemetry-sdk/src/opentelemetry/sdk/_configuration/__init__.py#L627)
* [Upstream ID Generator Environment Variable](https://github.com/open-telemetry/opentelemetry-python/blob/634cec5f2a2fecb40cb9d8216888c7b8865b845a/opentelemetry-api/src/opentelemetry/environment_variables/__init__.py#L49)
* [Upstream ID Generator Entry Point](https://github.com/open-telemetry/opentelemetry-python/blob/634cec5f2a2fecb40cb9d8216888c7b8865b845a/opentelemetry-sdk/pyproject.toml#L43-L44)
* [Lack of Entry Point in Airflow](https://github.com/apache/airflow/blob/a050861fc59193347a60d6ab306a5e0a03c719a7/shared/observability/pyproject.toml)

## Issue 2: Hardcoded Propagation logic

*Change Difficulty: Easy*

OpenTelemetry vendors may choose to leverage TraceState to propagate implementation-specific details, see definition in [SpanContext](https://opentelemetry.io/docs/concepts/signals/traces/#span-context).

Coupled with the fact that [TraceState is immutable](https://opentelemetry.io/docs/specs/otel/trace/api/#tracestate), vendors may also supply custom propagators (not just because of a different carrier format):

> Please note, since SpanContext is immutable, it is not possible to update SpanContext with a new TraceState. Such changes then make sense only right before SpanContext propagation or telemetry data exporting. In both cases, Propagators and SpanExporters may create a modified TraceState copy before serializing it to the wire.

The current strategy at the Airflow boundary was introduced in #69633. See callout to [hardcoding the propagator](https://github.com/apache/airflow/pull/69633/changes#r3640039700).

Suggest using the global [extract](https://opentelemetry-python.readthedocs.io/en/latest/api/propagate.html#opentelemetry.propagate.extract)/[inject](https://opentelemetry-python.readthedocs.io/en/latest/api/propagate.html#opentelemetry.propagate.inject) functions instead of instantiating a specific propagator, e.g. `TraceContextTextMapPropagator().extract(carrier)` becomes just `extract(carrier)`.

This issue also holds true for Baggage.

Workarounds may exist for this depending on the situation.

## Issue 3: DAG spans and cross-task propagation

*Change Difficulty: Moderate*

A incongruency exists between the DAG / trace waterfall and task dependency.

How do we intend to offer OpenTelemetry vendors a way to propagate information across tasks via context_carriers?

Distros may leverage TraceState or Baggage.

The task span is created from the dag run context and not from the previous task context https://github.com/apache/airflow/blob/83edc67794b0e33233c6d4cdca8300e66b345939/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py#L568

Because the dag_run.context_carrier is not exposed at the task level, there is no way OpenTelemetry native way to inject or propagate info to downstream tasks.

Ideally a task should be able to inject information into a context_carrier and have it be extracted for dependent tasks.

Workarounds using XCom exist, and maybe this is the correct place for it seeing as it is the intended cross-task communication medium.

See my comment regarding the [resolution of context_carrier vs parent_context_carrier](https://github.com/apache/airflow/issues/63281#issuecomment-4916089591).

### Use case/motivation

There's been significant progress in integration OpenTelemetry to Airflow. As it stands right now, OpenTelemetry configuration is baked into the `airflow.cfg` configuration flow. However, OpenTelemetry also provides a way to auto-instrument Python applications as well as for vendors to ship their own bundled set of configurations and key SDK components, see: [OpenTelemetry distribution](https://opentelemetry.io/docs/languages/python/distro/), [OpenTelemetry autoinstrumentation](https://opentelemetry.io/docs/zero-code/python/).

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Start with shared/observability/pyproject.toml and the task-instance route at airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py near the cited task-span creation. Review the existing OverrideableRandomIdGenerator and the hardcoded propagation logic, then compare them with OpenTelemetry's global extract/inject and entry-point behavior. Done requires an agreed implementation for distro configuration, propagation, and cross-task context, with the related work validated against the stated use cases.

Written by the indexing model from the issue text.