fastapi_app becomes non-serializable after passing through `FastAPIInstrumentor.instrument_app`

Describe your environment

OS: linux
Python version: (e.g., Python 3.9)
Package version:

opentelemetry-api==1.32.1
opentelemetry-exporter-opencensus==0.20b0
opentelemetry-exporter-otlp==1.32.1
opentelemetry-exporter-otlp-proto-common==1.32.1
opentelemetry-exporter-otlp-proto-grpc==1.32.1
opentelemetry-exporter-otlp-proto-http==1.32.1
opentelemetry-instrumentation==0.53b1
opentelemetry-instrumentation-asgi==0.53b1
opentelemetry-instrumentation-fastapi==0.53b1
opentelemetry-proto==1.32.1
opentelemetry-sdk==1.32.1
opentelemetry-semantic-conventions==0.53b1
opentelemetry-util-http==0.53b1

What happened?

get_meter has a threading lock in its function path, but unfortunately threading lock is not compatible with cloudpickle.

https://github.com/open-telemetry/opentelemetry-python/blob/f55bceca6f13265204c0e1b1813d059df9fcb518/opentelemetry-api/src/opentelemetry/metrics/_internal/__init__.py#L167-L175

Steps to Reproduce

Not a clean repro script because it requires setting up and building ray. Ray is needed to make fastapi serializable

import cloudpickle
from ray import cloudpickle
from ray._private.serialization import pickle_dumps
from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from ray.serve._private.utils import ensure_serialization_context

fastapi_app = FastAPI()
# add instrumentation
FastAPIInstrumentor.instrument_app(fastapi_app)

ensure_serialization_context()
cloudpickle.loads(pickle_dumps(fastapi_app, ""))

Expected Result

fastapi_app serializes and deserializes cleanly

Actual Result

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File ~/ray/python/ray/_private/serialization.py:72, in pickle_dumps(obj, error_msg)
     [71](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/_private/serialization.py:71) try:
---> [72](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/_private/serialization.py:72)     return pickle.dumps(obj)
     [73](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/_private/serialization.py:73) except (TypeError, ray.exceptions.OufOfBandObjectRefSerializationException) as e:

File ~/ray/python/ray/cloudpickle/cloudpickle.py:1479, in dumps(obj, protocol, buffer_callback)
   [1478](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/cloudpickle/cloudpickle.py:1478) cp = Pickler(file, protocol=protocol, buffer_callback=buffer_callback)
-> [1479](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/cloudpickle/cloudpickle.py:1479) cp.dump(obj)
   [1480](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/cloudpickle/cloudpickle.py:1480) return file.getvalue()

File ~/ray/python/ray/cloudpickle/cloudpickle.py:1245, in Pickler.dump(self, obj)
   [1244](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/cloudpickle/cloudpickle.py:1244) try:
-> [1245](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/cloudpickle/cloudpickle.py:1245)     return super().dump(obj)
   [1246](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/cloudpickle/cloudpickle.py:1246) except RuntimeError as e:

TypeError: cannot pickle '_thread.lock' object

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[1], [line 13](vscode-notebook-cell:?execution_count=1&line=13)
     [10](vscode-notebook-cell:?execution_count=1&line=10) FastAPIInstrumentor.instrument_app(fastapi_app)
     [12](vscode-notebook-cell:?execution_count=1&line=12) ensure_serialization_context()
---> [13](vscode-notebook-cell:?execution_count=1&line=13) cloudpickle.loads(pickle_dumps(fastapi_app, ""))

File ~/ray/python/ray/_private/serialization.py:78, in pickle_dumps(obj, error_msg)
     [76](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/_private/serialization.py:76) msg = f"{error_msg}:\n{sio.getvalue()}"
     [77](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/_private/serialization.py:77) if isinstance(e, TypeError):
---> [78](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/_private/serialization.py:78)     raise TypeError(msg) from e
     [79](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/_private/serialization.py:79) else:
     [80](https://vscode-remote+ssh-002dremote-002bdevbox.vscode-resource.vscode-cdn.net/home/ubuntu/apps/~/ray/python/ray/_private/serialization.py:80)     raise ray.exceptions.OufOfBandObjectRefSerializationException(msg)

TypeError: :
================================================================================
Checking Serializability of <fastapi.applications.FastAPI object at 0x7f62ad490880>
================================================================================
!!! FAIL serialization: cannot pickle '_thread.lock' object
    Serializing '_is_instrumented_by_opentelemetry' True...
    Serializing 'add_api_route' <bound method FastAPI.add_api_route of <fastapi.applications.FastAPI object at 0x7f62ad490880>>...
    !!! FAIL serialization: cannot pickle '_thread.lock' object
        Serializing '__func__' <function FastAPI.add_api_route at 0x7f628cd9e0d0>...
    WARNING: Did not find non-serializable object in <bound method FastAPI.add_api_route of <fastapi.applications.FastAPI object at 0x7f62ad490880>>. This may be an oversight.
================================================================================
Variable: 

	FailTuple(add_api_route [obj=<bound method FastAPI.add_api_route of <fastapi.applications.FastAPI object at 0x7f62ad490880>>, parent=<fastapi.applications.FastAPI object at 0x7f62ad490880>])

was found to be non-serializable. There may be multiple other undetected variables that were non-serializable. 
Consider either removing the instantiation/imports of these variables or moving the instantiation into the scope of the function/class. 
================================================================================
Check https://docs.ray.io/en/master/ray-core/objects/serialization.html#troubleshooting for more information.
If you have any suggestions on how to improve this error message, please reach out to the Ray developers on github.com/ray-project/ray/issues/
================================================================================

Additional context

This problem is particularly interesting to Ray Serve is because it take user code, ships it over the wire and runs it on the remote replica. This requires ser/deser user code that include fastapi_app definition.

Any recommendations on how to circumvent this problem? What is a downside of not setting up a meter inside instrument_app?

Would you like to implement a fix?

Yes

Start with the get_meter path in instrumentation/opentelemetry-instrumentation-fastapi/src/opentelemetry/instrumentation/fastapi/init.py and the referenced metrics implementation in opentelemetry-api/src/opentelemetry/metrics/_internal/init.py. Reproduce the failure with the FastAPIInstrumentor, Ray serialization context, and cloudpickle example; done means the instrumented fastapi_app serializes and deserializes cleanly without the _thread.lock error.