Provide hooks to modify span content before conversion into ReadableSpan

Is your feature request related to a problem?

I want to control the redaction and modification of sensitive information in spans including attributes, and exceptions from a central location.

In our specific case and as described in this pull request, we are running into problems where an exception that is raised as a result of hitting a sensitive URL is logged in the trace. In an effort to reduce potential vectors of attack, I do not want to persist this information at all to any telemetry backend. Instead, I want to make sure this data is dropped out of any log/trace before it leaves the machine that produced the message (In this case it's AWS ECS, or a developers local machine).

Describe the solution you'd like

I would like to execute one or more hooks before a ReadableSpan is created in this code. The purpose of these hooks would allow user defined code to modify the Span in any way before it's converted into a read-only span. This is the hack I suggested to my developer in this comment.

https://github.com/Sage-Bionetworks/schematic/pull/1574 Is an example to monkey patch the code that allows us to perform modifications on the Span before it gets exported to the telemetry backend.

Alternatively, I question what the benefit of a read-only span gives in the python SDK. I can see there being some concerns around race conditions when working with the ConcurrentMultiSpanProcessor, but SynchronousMultiSpanProcessor don't have that same problem (What I am using). If the span could be modified within the SpanProcessor's that we implement, then there is no need for this hook mechanism, we would be able to do this data modification in the on_end function call.

Describe alternatives you've considered

I am aware that the OTEL collector may be used handle the transformation of data in the content flowing through it as described in this document. However, this is not a solution that I wanted to start with because we may have developers running the application code locally and exporting telemetry data directly to our telemetry backend. In this scenario there is no OTEL collector in the path of that data that we may use to transform the data and redact any sensitive information.

Additional Context

No response

Would you like to implement a fix?

None

Start with opentelemetry-sdk/src/opentelemetry/sdk/trace/init.py at the ReadableSpan creation around lines 906-921, then read the ConcurrentMultiSpanProcessor and SynchronousMultiSpanProcessor references. Determine how a hook or processor could modify span attributes and exceptions before export without retaining sensitive data. Done means the chosen design supports central redaction before telemetry leaves the machine.