[Bug]: KeyError: 'Key' in s3fs.__get_s3_meta with Ceph RADOS Gateway when bucket has 1000+ objects

What happened?

Description

When using s3fs backend with Ceph RADOS Gateway and more than 1000 objects in a bucket (S3 pagination threshold), fileserver.update fails with KeyError: 'Key'.

Environment

• Salt version: 3006.17.0 (official)
• S3 backend: Ceph RADOS Gateway (v16, also reproduced on Ceph demo container ceph/demo:latest)
• Mode: multiple environments per bucket (s3.buckets as list)
• Python: 3.10

Steps to reproduce

1. Configure s3fs backend with Ceph RADOS Gateway
2. Upload 1100+ objects to the bucket
3. Run salt-run fileserver.update
4. Observe error

Actual behaviour

Exception occurred in runner fileserver.update: Traceback (most recent call last):
  ...
  File ".../salt/fileserver/s3fs.py", line 369, in _init
    metadata = _refresh_buckets_cache_file(cache_file)
  File ".../salt/fileserver/s3fs.py", line 502, in _refresh_buckets_cache_file
    s3_meta = __get_s3_meta(bucket_name)
  File ".../salt/fileserver/s3fs.py", line 448, in __get_s3_meta
    marker = tmp[-1]["Key"]
KeyError: 'Key'

Expected behavior

fileserver.update should complete successfully regardless of object count.

Root cause analysis

In _refresh_buckets_cache_file, the nested function __get_s3_meta handles S3 pagination:

def __get_s3_meta(bucket, key=key, keyid=keyid):
    ret, marker = [], ""
    while True:
        tmp = __utils__["s3.query"](...)
        headers = []
        for header in tmp:
            if "Key" in header:
                break
            headers.append(header)
        ret.extend(tmp)
        if all(
            [header.get("IsTruncated", "false") == "false" for header in headers]
        ):
            break
        marker = tmp[-1]["Key"]  # <-- BUG: assumes last element always has 'Key'
    return ret

The bug: __tmp[-1]["Key"]__ assumes the last element of the parsed S3 response is always an object with a 'Key' field. However, Ceph RADOS Gateway returns service fields (Marker, NextMarker) after the object list in the XML response. When parsed into a Python list, these service fields become the last elements, which do not contain 'Key', causing KeyError.

Why it only happens with 1000+ objects: With fewer than 1000 objects, S3 returns IsTruncated == false, the loop exits via break before reaching the faulty line. With 1000+ objects, pagination is triggered, the loop enters a second iteration, and the bug manifests.

Why it doesn't happen on MinIO/AWS: MinIO and AWS S3 place service fields (IsTruncated, MaxKeys) before the object list in the XML. After parsing, tmp[-1] is always an object with 'Key'.

Confirmed Ceph response structure

From debug logs of actual Ceph responses:

Page with IsTruncated=true (1000+ objects):

[
    {"IsTruncated": "true", ...},    # service field
    {"Key": "file_0001.sls", ...},   # objects
    ...
    {"Key": "file_1000.sls", ...},   # last object
    {"NextMarker": "file_1000.sls"}  # <-- tmp[-1], NO 'Key' → KeyError
]

Page with IsTruncated=false (<1000 objects):

[
    {"IsTruncated": "false", ...},   # service field
    {"Key": "file_0001.sls", ...},   # objects
    ...
    {"Key": "file_0500.sls", ...},   # last object
    {"Marker": None}                 # <-- tmp[-1], NO 'Key' (but loop exits before)
]

Testing summary

Backend	Objects	Pagination triggered	KeyError
Ceph RADOS Gateway (v16)	1100	Yes	Yes
Ceph RADOS Gateway (v16)	<1000	No	No
MinIO	1100	Yes	No
MinIO	10000+	Yes	No

All tests were performed with verified pagination:

• Ceph: default MaxKeys=1000, confirmed via list_objects_v2
• MinIO: default MaxKeys=1000, confirmed via list_objects_v2

Proposed fix

Replace direct access to tmp[-1]["Key"] with a search for the last element that actually contains 'Key':

# Ceph RGW places service fields (Marker/NextMarker) AFTER
# Contents objects, unlike MinIO/AWS where they appear before.
# Therefore tmp[-1] does not always contain 'Key' —
# search for the last real object from the end of the list.
next_marker = next(
    (item["Key"] for item in reversed(tmp) if "Key" in item), None
)

if next_marker is None:
    log.error(
        "s3fs: unable to determine pagination marker for bucket "
        "'%s' — response page contained no object with 'Key'. "
        "Stopping pagination early; listing may be incomplete.",
        bucket,
    )
    break

if next_marker == marker:
    log.error(
        "s3fs: pagination marker did not advance for bucket '%s', "
        "aborting to avoid infinite loop.", bucket,
    )
    break

marker = next_marker

This fix:

• Iterates tmp in reverse to find the last element with 'Key' (compatible with all backends)
• Handles edge case where no 'Key' is present at all (logs error, breaks gracefully)
• Prevents infinite loop if the marker does not advance (logs error, breaks)

Backward compatibility

On MinIO/AWS, reversed(tmp) finds 'Key' immediately in the last element — behavior is identical to the original code. No regression.

Unit tests

Unit tests are ready and cover all scenarios:

Test	Scenario	Status without fix	Status with fix
test_ceph_two_pages	Ceph, 1100 objects, 2 pages	❌ KeyError	✅ PASS
test_ceph_single_page	Ceph, 500 objects, no pagination	✅ PASS	✅ PASS
test_minio_two_pages	MinIO, 1100 objects, 2 pages	✅ PASS	✅ PASS
test_no_key_in_response	Response without 'Key'	❌ KeyError	✅ PASS
test_ceph_ten_pages	Ceph, 10000 objects, 10 pages	❌ KeyError	✅ PASS
test_stuck_marker_does_not_hang	Stuck marker — guard against infinite loop	❌ Hang	✅ PASS
test_empty_bucket	Empty bucket, service fields only	✅ PASS	✅ PASS
test_exactly_1000_boundary	Exactly 1000 objects, boundary case	✅ PASS	✅ PASS

Tests require pytest-timeout for the stuck marker test:

pip install pytest-timeout
pytest tests/unit/fileserver/test_s3fs.py::TestS3FSPagination -v --timeout=10

Workaround

Keep bucket object count below 1000.

Related issues

• #66473 — similar KeyError in _prune_deleted_files, different root cause
• boto3 #470 — boto3 officially recommends using the last object key from Contents rather than relying on NextMarker, which is not guaranteed to be returned by all S3-compatible providers.

s3fs_ceph_pagination.patch
test_s3fs.py

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-24.04

salt --versions-report output

Salt Version:
          Salt: 3006.17
 
Python Version:
        Python: 3.10.19 (main, Oct 30 2025, 04:53:28) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: 1.9.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.17.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.21.2
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 4.5.3
           ZMQ: 4.3.4

Start in salt/fileserver/s3fs.py at __get_s3_meta and review the pagination handling in _refresh_buckets_cache_file. Run tests/unit/fileserver/test_s3fs.py::TestS3FSPagination with pytest-timeout, including the Ceph, empty-response, boundary, and stuck-marker cases; done means fileserver.update handles Ceph pages without KeyError or hanging.