Ambiguous S3FileSystem.isfile behavior when an object exists with a "/" suffix

I was debugging a problem with my pipeline, and I've reduced to the following code snippet:

import boto3
import pandas as pd

df = pd.DataFrame({"foo": ["a", "b"], "bar": [1,2]})
df.to_parquet("s3://my-bucket/dataset/", partition_cols=["foo"])
pd.read_parquet("s3://my-bucket/dataset/") # Works

# Now create an empty object with the dataset name:
s3 = boto3.client("s3")
s3.put_object(Bucket="my-bucket", Key="dataset/", Body=b"")
pd.read_parquet("s3://my-bucket/dataset/") # Throws FileNotFoundError
"""
File /nix/store/avq9131sdfjzn14fsilqgb0x3b76s6k7-python3-3.11.4-env/lib/python3.11/site-packages/pyarrow/fs.py:424, in FSSpecHandler.open_input_file(self, path)
    421 from pyarrow import PythonFile
    423 if not self.fs.isfile(path):
--> 424     raise FileNotFoundError(path)
    426 return PythonFile(self.fs.open(path, mode="rb"), mode="r")

FileNotFoundError: my-bucket/dataset/
"""

This is failing because pyarrow uses S3FileSystem.find("my-bucket/dataset", withdirs=True, detail=True) method to list all files in a partitioned parquet dataset, and s3fs is listing s3://my-bucket/dataset/ as a file and a directory, while S3FileSystem.isfile("s3://my-bucket/dataset/") is returning false. So I think there is an ambiguity happening here, as S3 doesn't have the concept of a directory.

Another example of the problem:

import boto3
import s3fs

boto3.put_object(Bucket="my-bucket", Key="fake_dir/", Body=b"")
boto3.put_object(Bucket="my-bucket", Key="fake_dir/file", Body=b"foo")

fs = s3fs.S3FileSystem()
fs.find("my-bucket/fake_dir", withdirs=True, detail=True)
result = {'my-bucket/fake_dir': {'Key': 'tmp/deleteme5',
  'Size': 0,
  'name': 'my-bucket/fake_dir',
  'StorageClass': 'DIRECTORY',
  'type': 'directory',
  'size': 0},
 'my-bucket/fake_dir/': {'Key': 'my-bucket/fake_dir/',
  'LastModified': datetime.datetime(2023, 9, 25, 13, 54, 48, tzinfo=tzutc()),
  'ETag': '"d41d8cd98f00b204e9800998ecf8427e"',
  'Size': 0,
  'StorageClass': 'STANDARD',
  'type': 'file',
  'size': 0,
  'name': 'my-bucket/fake_dir/'},
 'my-bucket/fake_dir/file': {'Key': 'my-bucket/fake_dir/file',
  'LastModified': datetime.datetime(2023, 9, 25, 13, 54, 54, tzinfo=tzutc()),
  'ETag': '"acbd18db4cc2f85cedef654fccc4a4d8"',
  'Size': 3,
  'StorageClass': 'STANDARD',
  'type': 'file',
  'size': 3,
  'name': 'my-bucket/fake_dir/file'}}

fs.isfile("my-bucket/fake_dir")  # False
fs.isfile("my-bucket/fake_dir/") # False

Start with S3FileSystem.isfile and find(..., withdirs=True, detail=True), using the two object layouts in the report as reproductions. Determine the intended result when a key ending in '/' exists alongside children, then add regression coverage for both cases and verify parquet access no longer raises FileNotFoundError.