Deadlock when using on_finishing to modify span object

Description of the bug

Calling on_finishing from processor will cause deadlock if trying to modify any object inside span.

ERROR -- : OpenTelemetry error: deadlock; recursive locking - /usr/local/bundle/gems/opentelemetry-sdk-1.8.0/lib/opentelemetry/sdk/trace/span.rb:80:in `synchronize'

This is caused by double @mutex.synchronize on finish(end_timestamp: nil) and other function such as add_attributes, etc. from processor.on_finishing(self).

Possible solutions

1. Use Monitor: since Monitor is reentrant.

2. Put the on_finishing outside the @mutex.synchronize. Since most operation on span object are within mutex.synchronize, there is no need to double synchronize.

def finish(end_timestamp: nil)
  @mutex.synchronize do
    if @ended
      OpenTelemetry.logger.warn('Calling finish on an ended Span.')
      return self
    end
  end
  @span_processors.each do |processor|
    processor.on_finishing(self) if processor.respond_to?(:on_finishing)
  end
  @mutex.synchronize do
    @end_timestamp = relative_timestamp(end_timestamp)
    @attributes = validated_attributes(@attributes).freeze
    @events.freeze
    @links.freeze
    @ended = true
  end
  @span_processors.each { |processor| processor.on_finish(self) }
  self
end

Share details about your runtime

Operating system details: Linux, Ubuntu 20.04 LTS
RUBY_ENGINE: "ruby"
RUBY_VERSION: "3.1.0"
RUBY_DESCRIPTION: "ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [aarch64-linux]"

Share a simplified reproduction if possible

class TestProcessor
  def on_start(span, parent_context); end
  def on_finishing(span)
    span.set_attribute('hello', 'world')
  end
  def on_finish(span); end
end

Start in lib/opentelemetry/sdk/trace/span.rb at finish and reproduce the deadlock with the TestProcessor example from the issue. Add a regression test showing that on_finishing can call set_attribute without recursive locking, then verify the span finishes successfully and the attribute is retained.