apache / apache/airflow

Remove pre-release OpenTelemetry pins from generated constraints

Open
#71,176 0 comments 0 reactions 0 assignees View on GitHub
area:dependencies area:dev-env upstream-dependency
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

Our generated constraints currently pin two packages to pre-release (beta) versions:

```
opentelemetry-exporter-prometheus==0.65b0
opentelemetry-semantic-conventions==0.65b0
```

These are the only pre-releases in the constraints files we tag and ship, and both require a hardcoded exception in [`scripts/in_container/run_generate_constraints.py`](https://github.com/apache/airflow/blob/main/scripts/in_container/run_generate_constraints.py) (`additional_constraints_for_highest_resolution`).

### Why the exception exists

Constraints cut for a release candidate resolve with `--prerelease explicit`, which permits a pre-release only for packages some requirement explicitly marks as such, and drops the `if-necessary` fallback. Neither OpenTelemetry package has *ever* published a final release - every version on PyPI is `0.NNbM` (61 releases for `opentelemetry-semantic-conventions`, 55 for `opentelemetry-exporter-prometheus`, zero non-pre-release among them). Without an explicit pre-release lower bound in the resolution, they cannot resolve at all and constraints generation fails with `No solution found`.

### Where they come from

- `opentelemetry-exporter-prometheus` - direct dependency of the `otel` extra of `apache-airflow-core` (which is part of the `all` extra), `>=0.47b0`.
- `opentelemetry-semantic-conventions` - hard transitive dependency of `opentelemetry-sdk`, which comes in via `opentelemetry-exporter-otlp` (a core `apache-airflow-core` dependency) and directly via `shared/observability` and `shared/listeners`.

So neither pin can simply be dropped today: `semantic-conventions` is unavoidable as long as we depend on the OpenTelemetry SDK at all.

### What needs to happen

Any of the following would let us delete the exception:

1. **Upstream stabilisation** - OpenTelemetry Python graduates these components to `1.x` finals. This is the real fix and is outside our control; it needs tracking against [`open-telemetry/opentelemetry-python`](https://github.com/open-telemetry/opentelemetry-python).
2. **Drop `opentelemetry-exporter-prometheus`** from the `otel` extra and document it as a user-installed optional dependency - removes one of the two pins, but not `semantic-conventions`.
3. **Reconsider whether shipping the OTel SDK in core dependencies is required**, since it is what drags `semantic-conventions` in.

Option 1 is the only one that removes both pins without a functional trade-off, so this issue mainly serves as the tracking anchor referenced from the code, to be revisited whenever the OTel Python packages change their versioning scheme.

### Acceptance criteria

- No pre-release version appears in tagged constraints files.
- `additional_constraints_for_highest_resolution` in `scripts/in_container/run_generate_constraints.py` no longer needs entries for either OpenTelemetry package, and constraints generation still succeeds with `--allow-pre-releases` (the RC path).

Contributor guide

Open the contributing guide

Research direction

Start with scripts/in_container/run_generate_constraints.py, especially additional_constraints_for_highest_resolution, and inspect how tagged constraints are generated with --allow-pre-releases. Track the OpenTelemetry Python versioning change or evaluate the dependency options described in the issue. Done means tagged constraints contain no pre-releases, the two exception entries are removed, and RC constraint generation succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.