OTEL_SERVICE_NAME set in telemetry setup is not respected.

OTEL_SERVICE_NAME is not included in the recorded base environment variables (https://github.com/rapidsai/shared-actions/blob/c20a4ad8af627e8c75dc2693d75596323838550a/telemetry-impls/stash-base-env-vars/action.yml)

I went back and forth on this, and I guess I left a mess. Some repos have this, like cudf, but it isn't doing anything. It only happens to be set to the same value by the summarize action: https://github.com/rapidsai/shared-actions/blob/c20a4ad8af627e8c75dc2693d75596323838550a/telemetry-impls/summarize/action.yml#L38

We should either:

• fix it so that OTEL_SERVICE_NAME in telemetry setup applies
• change docs to specify that OTEL_SERVICE_NAME can be set in the summarize step instead
• not support custom names
• Some other way to support custom naming of top-level jobs

Chat with Jason Pascucci, who raised this issue (Jason in quote blocks, me not):

Although I can't tell what effect it has due to the grafana being offline, I notice another thing I wasn't expecting: telemetry-dispatch-setup sets the OTEL_SERVICE_NAME to the name of the current job. I expected that it would keep the OTEL_SERVICE_NAME that was configured in the telemetry-dispatch-stash-base-env-vars call.

OTEL_SERVICE_NAME is for the outer-most job. We use it to distinguish projects from each other. It really only matters in the Grafana/Tempo views. The same info is available in the attributes. which job in particular are you wanting to change?

I set OTEL_SERViCE_NAME to ci-cupynumeric in the outtermost setup telemetry call, but telemetry-dispatch-setup appears to overwrite it here, which I think might be wrong.

are you setting it like this? https://github.com/rapidsai/cudf/blob/main/.github/workflows/pr.yaml#L50-L58

pr.yaml

  telemetry-setup:
    continue-on-error: true
    runs-on: ubuntu-latest
    env:
      OTEL_SERVICE_NAME: 'pr-cudf'

the important difference is this different action, which happens only for the top-level job: https://github.com/rapidsai/cudf/blob/main/.github/workflows/pr.yaml#L58

pr.yaml

        uses: rapidsai/shared-actions/telemetry-dispatch-stash-base-env-vars@main

though I'm not seeing OTEL_SERVICE_NAME in those files. Let me see if I can track down where this comes in.

this is where it ultimately has to be set: https://github.com/rapidsai/shared-actions/blob/main/telemetry-impls/summarize/action.yml#L35-L46
OTEL_SERVICE_NAME is not explicitly set when calling that action: https://github.com/rapidsai/cudf/blob/main/.github/workflows/pr.yaml#L430-L440
ah, found it. It's not respecting the value in the setup env var. It is only coincidentally coming out to be the same value:
https://github.com/rapidsai/shared-actions/blob/c20a4ad8af627e8c75dc2693d75596323838550a/telemetry-impls/summarize/action.yml#L38-L46
You can probably set the env: entry on the summarize job, like here: https://github.com/rapidsai/cudf/blob/main/.github/workflows/pr.yaml#L430-L440

Review telemetry-impls/stash-base-env-vars/action.yml, telemetry-dispatch-setup/action.yml, and telemetry-impls/summarize/action.yml, starting with the linked environment-variable handling. Trace how OTEL_SERVICE_NAME moves from setup to summarize and compare the cudf workflow examples. Done should establish and document one supported behavior for custom top-level job names.