Bug in FlaxDiffusionPipeline's from_pt option in Google Collab's TPUv2

### Describe the bug

I get an error when I try to load a model with the from_pt option of FlaxDiffusionPipeline in GoogleCollab TPU v2.

Incidentally, it sometimes succeeds in loading.

### Reproduction

pip install -q diffusers
from diffusers import FlaxDiffusionPipeline
import jax
pipe = FlaxDiffusionPipeline.from_pretrained("admruul/anything-v3.0",
dtype=jax.numpy.bfloat16,
from_pt = True)

### Logs

```shell
---------------------------------------------------------------------------
XlaRuntimeError Traceback (most recent call last)
in ()
2 from diffusers import FlaxDiffusionPipeline
3 import jax
----> 4 pipe = FlaxDiffusionPipeline.from_pretrained("admruul/anything-v3.0",
5 dtype=jax.numpy.bfloat16,
6 from_pt = True)

18 frames
/usr/local/lib/python3.10/dist-packages/jax/_src/interpreters/pxla.py in batched_device_put(aval, sharding, xs, devices, committed)
179 return array.ArrayImpl(
180 aval, sharding, bufs, committed=committed, _skip_checks=True)
--> 181 return xc.batched_device_put(aval, sharding, xs, list(devices), committed) # type: ignore
182
183 def _shard_aval(size, axis: int, aval):

XlaRuntimeError: RESOURCE_EXHAUSTED: Error allocating device buffer: Attempting to allocate 14.06M. That was not possible. There are 12.00M free.; (0x0x0_HBM0)

When I tried to run it again, it returned this error.

---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in ()
2 from diffusers import FlaxDiffusionPipeline
3 import jax
----> 4 pipe = FlaxDiffusionPipeline.from_pretrained("admruul/anything-v3.0",
5 dtype=jax.numpy.bfloat16,
6 from_pt = True)

11 frames
[... skipping hidden 9 frame]

[... skipping hidden 2 frame]

[... skipping hidden 2 frame]

[... skipping hidden 5 frame]

[... skipping hidden 2 frame]

/usr/local/lib/python3.10/dist-packages/jax/_src/random.py in normal(key, shape, dtype)
709 dtype = dtypes.canonicalize_dtype(dtype)
710 shape = core.as_named_shape(shape)
--> 711 return _normal(key, shape, dtype) # type: ignore
712
713 @partial(jit, static_argnums=(1, 2))

ValueError: RESOURCE_EXHAUSTED: Error allocating device buffer: Attempting to allocate 144.75M. That was not possible. There are 11.88M free.; (0x0x0_HBM0)
```

### System Info

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/jax/_src/xla_bridge.py", line 853, in backends
backend = _init_backend(platform)
File "/usr/local/lib/python3.10/dist-packages/jax/_src/xla_bridge.py", line 935, in _init_backend
backend = registration.factory()
File "/usr/local/lib/python3.10/dist-packages/jax/_src/xla_bridge.py", line 154, in tpu_client_timer_callback
client = xla_client.make_tpu_client(_get_tpu_library_path())
File "/usr/local/lib/python3.10/dist-packages/jaxlib/xla_client.py", line 206, in make_tpu_client
return make_tfrt_tpu_c_api_client()
File "/usr/local/lib/python3.10/dist-packages/jaxlib/xla_client.py", line 127, in make_tfrt_tpu_c_api_client
initialize_pjrt_plugin('tpu')
File "/usr/local/lib/python3.10/dist-packages/jaxlib/xla_client.py", line 175, in initialize_pjrt_plugin
_xla.initialize_pjrt_plugin(plugin_name)
jaxlib.xla_extension.XlaRuntimeError: ABORTED: The TPU is already in use by process with pid 10084. Not attempting to load libtpu.so in this process.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/diffusers-cli", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/diffusers/commands/diffusers_cli.py", line 39, in main
service.run()
File "/usr/local/lib/python3.10/dist-packages/diffusers/commands/env.py", line 77, in run
jax_backend = jax.lib.xla_bridge.get_backend().platform
File "/usr/local/lib/python3.10/dist-packages/jax/_src/xla_bridge.py", line 981, in get_backend
return _get_backend_uncached(platform)
File "/usr/local/lib/python3.10/dist-packages/jax/_src/xla_bridge.py", line 960, in _get_backend_uncached
bs = backends()
File "/usr/local/lib/python3.10/dist-packages/jax/_src/xla_bridge.py", line 869, in backends
raise RuntimeError(err_msg)
RuntimeError: Unable to initialize backend 'tpu': ABORTED: The TPU is already in use by process with pid 10084. Not attempting to load libtpu.so in this process. (set JAX_PLATFORMS='' to automatically choose an available backend)

### Who can help?

_No response_