Adding custom attributes to traces at step runtime

**Is your feature request related to a problem? Please describe.**
Currently the attributes added to a trace (in open telemetry at least) by buildkite are static. For example a trace stored in tempo had the following attribute keys on the root span:
```
buildkite.agent
buildkite.branch
buildkite.build_id
buildkite.build_number
buildkite.build_url
buildkite.job_id
buildkite.job_key
buildkite.job_label
buildkite.job_url
buildkite.org
buildkite.parallel
buildkite.pipeline
buildkite.queue
buildkite.rebuilt_from_id
buildkite.retry
buildkite.source
buildkite.triggered_from_id
buildkite.version
cluster
deployment.environment
service.name
service.version
service_component
```
There's no way to add another attribute dynamically at run time (or statically either currently).

Yet this could be useful to allow filtering by different dimensions:
- Commit hash
- Team of author (which might or might not be related to the buildkite team but come from another source)
- ...

This would give flexibility to extend traces as needed without having to cover too many cases statically in the agent and makes them very powerful.

**Describe the solution you'd like**
The CLI could allow setting an attribute (ideally at the root span level) dynamically at run time:
```sh
# Example cli interface:
buildkite-agent tracing set-attribute "attr-name" "json-encoded-attr-value"
```
This would then be propagated to the spans (except already closed ones, i.e. all the pre-run ones).

According to the [open telemetry attribute docs](https://opentelemetry.io/docs/specs/otel/common/#attribute) the value can be primitives, or an array of primitives. Using json encoded would make it easy to tell if for example 1 is a string or a number and to make arrays as opposed to a plain string value. Or it could be another encoding.

**Describe alternatives you've considered**
There's lots of ways attributes could be added/requested and this could be done at different times of the lifecycle.

There would be other ways to explicitly or implicitly expose attributes:
* Static attributes could be added via ENV/Config/Startup flags like other settings
* Expose buildkite meta data variables as attributes
* only if they match a default or customizable prefix (e.g. `trace_*`)
* list keys that need to be exposed/forwarded as a global settings
* Expose env variables as attributes if they match a certain patter (e.g. `TRACE_*` or more backend specific `OTEL_ATTR_*`)
* Expose env variables from a list of exposed ones (e.g. `TRACE_FORWARD_ENV_VARS="custom1 custom2 custom.experiment*"

**Additional context**
I'm not sure how close or different datadog is from open telemetry, ideally this feature could be done in a way that supports both backends to avoid as much backend specific code as possible.

Would buildkite accept a PR allowing addition of trace attributes? If so what kind of solution/approach would be accepted?

No files, tests, or entry points are named. Start by locating the tracing CLI implementation and the OpenTelemetry and Datadog integrations, then determine how runtime state reaches open spans. Done requires an agreed interface, supported attribute-value validation, propagation behavior, and coverage for the selected backends.

Written by the indexing model from the issue text.