Pagination problem when listing directories (1000 file limit)

I am prototyping an s3-compatible storage service called open storage network.

I have encountered a problem with how s3fs is listing directories which appears to be related to pagination. Basically, s3fs thinks there are only 1000 objects in the directory and refuses to even try to read objects that don't show up in this initial list.

import boto3
import s3fs
assert s3fs.__version__ == '0.4.0'

# read-only credentials to bucket, okay to share publicly 
access_key = "EL456I5ZRYB44RB6J7Q4"
secret_key = "QydNAjMWBTOLRjHiA36uMvhBvI4WeTxWYNJ5oaiP"
endpoint_url = "https://ncsa.osn.xsede.org"

# create boto client
s3 = boto3.client('s3',
                  aws_access_key_id=access_key,
                  aws_secret_access_key=secret_key,
                  endpoint_url=endpoint_url)
# verify credentials
assert s3.list_buckets()['Buckets'][0]['Name'] == 'Pangeo'

# list the bucket using recommend boto pagination technique
# https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html#filtering-results
paginator = s3.get_paginator('list_objects')
operation_parameters = {'Bucket': 'Pangeo',
                        'Prefix': 'cm26_control_temp.zarray'}
page_iterator = paginator.paginate(**operation_parameters)
# the directory should have 2402 objects in it
for page in page_iterator:
    print(len(page['Contents']))
# > 1000
# > 1000
# > 402 
# Correctly finds all 2402 objects

print(page['Contents'][-1]['Key'])
# > 'cm26_control_temp.zarray/99.9.0.0'

# now try with s3fs
fs = s3fs.S3FileSystem(key=access_key, secret=secret_key,
                       client_kwargs={'endpoint_url': endpoint_url})

listing = fs.listdir('Pangeo/cm26_control_temp.zarray')
print(len(listing))
# > 1000

# try to read a file that did not make it into the list
with fs.open('Pangeo/cm26_control_temp.zarray/99.9.0.0') as f:
    pass
# > FileNotFoundError: Pangeo/cm26_control_temp.zarray/99.9.0.0

This feels very much like a bug in s3fs. (A somewhat similar issue was noted in https://github.com/dask/s3fs/issues/253#issuecomment-557516952, including the 1000 file limit.) In fact, I would identify two distinct bugs:

• The directly listing is wrong
• s3fs is incorrectly raising a FileNotFoundError when I try to open an existing object (likely related to caching)

For the first issue, one possible hint could be that the aws cli also makes the same mistake:

aws s3 --profile osn-rw ls --recursive s3://Pangeo/cm26_control_temp.zarray/ | wc -l
# > 1000

So perhaps there is something in the metadata of the OSN service that is tricking the paginators in some circumstances.

This issue is rather important to Pangeo, as we are keen to get some accurate benchmarks on this new storage service. Help would be sincerely appreciated.

Start by reproducing the issue with the boto3 paginator, then compare its results with s3fs.S3FileSystem.listdir and fs.open using the credentials and endpoint in the report. Investigate why listing stops at 1000 objects and why the existing final object raises FileNotFoundError; done means all 2402 objects are listed and that object opens successfully.