[Python] pyarrow dataset API fails to read s3 directory

When an s3 file system as `file_system` is passed to [pyarrow.dataset.dataset](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.dataset.html#pyarrow.dataset.dataset) API and the `source` is a directory name with bucket, there is an error:

```python

In [5]: from fsspec.core import get_fs_token_paths

In [6]: fs, _, path = get_fs_token_paths("s3://prem-rapids-test/folder/", mode="rb")

In [7]: fs
Out[7]:

In [8]: path
Out[8]: ['prem-rapids-test/folder']

In [10]: pa.dataset.dataset(path, filesystem=fs, format="parquet")
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [10], in ()
----> 1 pa.dataset.dataset(path, filesystem=fs, format="parquet")

File /nvme/0/pgali/envs/cudfdev/lib/python3.8/site-packages/pyarrow/dataset.py:670, in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
    668 elif isinstance(source, (tuple, list)):
    669     if all(_is_path_like(elem) for elem in source):
--> 670         return _filesystem_dataset(source, **kwargs)
    671     elif all(isinstance(elem, Dataset) for elem in source):
    672         return _union_dataset(source, **kwargs)

File /nvme/0/pgali/envs/cudfdev/lib/python3.8/site-packages/pyarrow/dataset.py:422, in _filesystem_dataset(source, schema, filesystem, partitioning, format, partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
    414 options = FileSystemFactoryOptions(
    415     partitioning=partitioning,
    416     partition_base_dir=partition_base_dir,
    417     exclude_invalid_files=exclude_invalid_files,
    418     selector_ignore_prefixes=selector_ignore_prefixes
    419 )
    420 factory = FileSystemDatasetFactory(fs, paths_or_selector, format, options)
--> 422 return factory.finish(schema)

File /nvme/0/pgali/envs/cudfdev/lib/python3.8/site-packages/pyarrow/_dataset.pyx:1680, in pyarrow._dataset.DatasetFactory.finish()

File /nvme/0/pgali/envs/cudfdev/lib/python3.8/site-packages/pyarrow/error.pxi:143, in pyarrow.lib.pyarrow_internal_check_status()

File /nvme/0/pgali/envs/cudfdev/lib/python3.8/site-packages/pyarrow/_fs.pyx:1179, in pyarrow._fs._cb_open_input_file()

File /nvme/0/pgali/envs/cudfdev/lib/python3.8/site-packages/pyarrow/fs.py:394, in FSSpecHandler.open_input_file(self, path)
    391 from pyarrow import PythonFile
    393 if not self.fs.isfile(path):
--> 394     raise FileNotFoundError(path)
    396 return PythonFile(self.fs.open(path, mode="rb"), mode="r")

FileNotFoundError: prem-rapids-test/folder
``` 

But it works only if the folder is passed as a full string:
```python

In [3]: import pyarrow.dataset

In [4]: pa.dataset.dataset("s3://prem-rapids-test/folder/", format="parquet")
Out[4]:

```

 

**Reporter**: [Prem Sagar Gali](https://issues.apache.org/jira/browse/ARROW-16438) / @galipremsagar

_{**Note**: *This issue was originally created as [ARROW-16438](https://issues.apache.org/jira/browse/ARROW-16438). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*}

Reproduce the failure with pyarrow.dataset.dataset using an fsspec S3 filesystem and a directory path, then compare it with the full s3:// string case. Start by tracing the Python dataset filesystem-path handling shown in the traceback; done means directory paths supplied with an S3 filesystem are discovered like equivalent full strings, with a regression test for both forms.

Written by the indexing model from the issue text.