Filtering span attributes

Background
We have a team that needs to scrub personally identifiable information(PII) from their spans, which will ultimately be exported to Datadog. The applications use Fastapi and most of them are relying on botocore instrumentation. The spans for calls to DynamoDB generated by the botocore instrumentor need to have PII scrubbed/filtered from the span attributes.

Would greatly appreciate any advice/suggestions on achieving this.

Is your feature request related to a problem?
Our problem is that the SpanProcessor is not enough to achieve the behavior that we need. Our first pass at filtering the span attributes was to add a new SpanProcessor that overwrites PII in the span attributes. However, the approach failed because on_end and on_start` do not have to right context.

Using on_start is too early to filter attributes because some instrumentors(like botocore) call set_attributes after the span has been created.

Using on_end is too late to filter attributes as per the specification.

Describe the solution you'd like
Solution 1:
Ideally, we would have a hook to filter attributes sometime between on_start and on_end when the instrumentor has finished setting attributes for the span. Perhaps a filtering class could be called before self._end_time is set on the span.

Solution 2:
Add the filter to the exporter. Ex: The DatadogExporter could scrub attributes during the translation from opentelemetry Span to ddtrace Span.

Solution 3:
Add the filter to the instrumentation that is creating the targeted spans. Ex: The botocore instrumentor has filter when adding attributes to the span. Would be awesome if this could be added to the BaseInstrumentor.

Personally, I think it makes most sense to add the filter to the instrumentor since each instrumentation governs the names of the attributes that are added to the span. Having targeted filters for each instrumentation seems cleaner than a generic filter on the SpanProcessor that scrubs span attributes from multiple instrumentation. Downside would be that the targeted instrumentation can no longer use auto instrumentation.

Start by reading the SpanProcessor on_start/on_end lifecycle and the botocore instrumentor entry point where span attributes are added. Compare that with the DatadogExporter translation path and BaseInstrumentor mentioned in the proposed solutions. Done requires a decided filtering location and a specified way to scrub targeted span attributes without losing automatic instrumentation.