429 Client Error: Too Many Requests for url azuredatabricks.net/api/2.0/dbfs/mkdirs

I'm using the "fsspec.implementations.dbfs import DatabricksFileSystem" with pyarrow to write a parquet dataset on the DatabricksFilesystem DBFS, but when using the DatabricksFilesystem implementation with the write_to_dataset method from pyarrow I'm getting the following error:

429 Client Error: Too Many Requests for url: https://adb-<databricks_instance>.azuredatabricks.net/api/2.0/dbfs/mkdirs

The code used to write to DBFS is the following:

"""
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq
from fsspec.implementations.dbfs import DatabricksFileSystem

base_path = "/FileStore/write"
test_df = pd.read_csv("../data/diabetes/csv/nopart/diabetes.csv")

filesystem = DatabricksFileSystem(
instance=<databricks_instance>,
token=<databricks_token>
)

pq.write_to_dataset(arr_table, filesystem=filesystem, compression='none',
existing_data_behavior='error', partition_cols=["Pregnancies"],
root_path=f"{base_path}/parquet/part",
use_threads= False)
"""

And the full stack trace is:

"""
Traceback (most recent call last):
File "/venv/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib/python3.9/site-packages/fsspec/implementations/dbfs.py", line 255, in _send_to_api
exception_json = e.response.json()
File "/venv/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tests/dbfs/test_dbfs_dataset.py", line 35, in setUpClass
pq.write_to_dataset(arr_table, filesystem=cls._filesystem, compression='none',
File "/venv/lib/python3.9/site-packages/pyarrow/parquet/core.py", line 3422, in write_to_dataset
ds.write_dataset(
File "/venv/lib/python3.9/site-packages/pyarrow/dataset.py", line 1018, in write_dataset
_filesystemdataset_write(
File "pyarrow/_dataset.pyx", line 3919, in pyarrow._dataset._filesystemdataset_write
File "pyarrow/types.pxi", line 88, in pyarrow.lib._datatype_to_pep3118
File "pyarrow/_fs.pyx", line 1529, in pyarrow._fs._cb_create_dir
File "/venv/lib/python3.9/site-packages/pyarrow/fs.py", line 374, in create_dir
self.fs.mkdir(path, create_parents=recursive)
File "/venv/lib/python3.9/site-packages/fsspec/implementations/dbfs.py", line 138, in mkdir
self.mkdirs(path, **kwargs)
File "/venv/lib/python3.9/site-packages/fsspec/spec.py", line 1498, in mkdirs
return self.makedirs(path, exist_ok=exist_ok)
File "/venv/lib/python3.9/site-packages/fsspec/implementations/dbfs.py", line 115, in makedirs
self._send_to_api(method="post", endpoint="mkdirs", json={"path": path})
File "/venv/lib/python3.9/site-packages/fsspec/implementations/dbfs.py", line 257, in _send_to_api
raise e
File "/venv/lib/python3.9/site-packages/fsspec/implementations/dbfs.py", line 250, in _send_to_api
r.raise_for_status()
File "/venv/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://adb-<databricks_instance>.azuredatabricks.net/api/2.0/dbfs/mkdirs
"""

Start with fsspec/implementations/dbfs.py, especially _send_to_api and makedirs, then inspect tests/dbfs/test_dbfs_dataset.py and the shown PyArrow call path. Reproduce the 429 response if possible and determine the expected handling for a non-JSON rate-limit response; done should include a verified behavior and corresponding test coverage.