[Python] Deadlock in the interaction of pyarrow FileSystem and ParquetDataset

@martindurant good news (for you): I have a repro test case that is 100% `pyarrow`, so it looks like `s3fs` is not involved.

@jorisvandenbossche how should I follow up with this, based on `pyarrow.filesystem.LocalFileSystem`?

Viewing the File System **directories** as a tree, one thread is required for every non-leaf node, in order to avoid deadlock.

1) dataset
2) dataset/foo=1
3) dataset/foo=1/bar=2
4) dataset/foo=1/bar=2/baz=0
5) dataset/foo=1/bar=2/baz=1
6) dataset/foo=1/bar=2/baz=2
\*) dataset/foo=1/bar=2/baz=0/qux=false
\*) dataset/foo=1/bar=2/baz=1/qux=false
\*) dataset/foo=1/bar=2/baz=1/qux=true
\*) dataset/foo=1/bar=2/baz=0/qux=true
\*) dataset/foo=1/bar=2/baz=2/qux=false
\*) dataset/foo=1/bar=2/baz=2/qux=true

```Java

import pyarrow.parquet as pq
import pyarrow.filesystem as fs

class LoggingLocalFileSystem(fs.LocalFileSystem):
def walk(self, path):
print(path)
return super().walk(path)

fs = LoggingLocalFileSystem()
dataset_url = "dataset"

threads = 6
dataset = pq.ParquetDataset(dataset_url, filesystem=fs, validate_schema=False, metadata_nthreads=threads)
print(len(dataset.pieces))

threads = 5
dataset = pq.ParquetDataset(dataset_url, filesystem=fs, validate_schema=False, metadata_nthreads=threads)
print(len(dataset.pieces))
```

**_Call with 6 threads completes._**

**_Call with 5 threads hangs indefinitely._**

```Java

$ python repro.py
dataset
dataset/foo=1
dataset/foo=1/bar=2
dataset/foo=1/bar=2/baz=0
dataset/foo=1/bar=2/baz=1
dataset/foo=1/bar=2/baz=2
dataset/foo=1/bar=2/baz=0/qux=false
dataset/foo=1/bar=2/baz=0/qux=true
dataset/foo=1/bar=2/baz=1/qux=false
dataset/foo=1/bar=2/baz=1/qux=true
dataset/foo=1/bar=2/baz=2/qux=false
dataset/foo=1/bar=2/baz=2/qux=true
6
dataset
dataset/foo=1
dataset/foo=1/bar=2
dataset/foo=1/bar=2/baz=0
dataset/foo=1/bar=2/baz=1
dataset/foo=1/bar=2/baz=2
^C
...
KeyboardInterrupt
^C
...
KeyboardInterrupt
```

\***NOTE:**\* this **also** happens with the un-decorated `LocalFileSystem`, and when omitting the `filesystem` argument.

**Reporter**: [David McGuire](https://issues.apache.org/jira/browse/ARROW-10029)
#### Original Issue Attachments:
- [repro.py](https://issues.apache.org/jira/secure/attachment/13011667/repro.py)
#### Externally tracked issue: [https://github.com/dask/s3fs/issues/365](https://github.com/dask/s3fs/issues/365)

_{**Note**: *This issue was originally created as [ARROW-10029](https://issues.apache.org/jira/browse/ARROW-10029). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*}

Start by running the attached repro.py and inspecting the interaction between pyarrow.parquet.ParquetDataset, pyarrow.filesystem.LocalFileSystem, and metadata_nthreads. Trace the directory walk and thread behavior for five workers, then add a regression test demonstrating that the reproduced dataset discovery completes without hanging.

Written by the indexing model from the issue text.