Consider allowing provider to specifiy the level of database index they would like.

As of current i am post-init hacking the mongoDB to have the following indices on our database:

INDEX_CONFIG = [
    # Single-field unique
    {"fields": ["id"], "name": "idx_id", "unique": True},

    # Single-field regular
    {"fields": ["elements"], "name": "idx_elements", "unique": False},
    {"fields": ["nelements"], "name": "idx_nelements", "unique": False},
    {"fields": ["nsites"], "name": "idx_nsites_top", "unique": False},

    # Compound indexes
    {"fields": ["nelements", "nsites"], "name": "idx_nelements_nsites", "unique": False},
    {"fields": ["elements", "nelements"], "name": "idx_elements_nelements", "unique": False},
    {"fields": ["elements", "nsites"], "name": "idx_elements_nsites", "unique": False},

    # Descending / mixed order indexes
    {"fields": [("elements", -1)], "name": "idx_elements_desc", "unique": False},
    {"fields": [("elements", 1), ("nsites", -1)], "name": "idx_elements_nsites_desc", "unique": False},
]

and them am putting them inside the data via:

    # Create indexes based on INDEX_CONFIG
    for idx in INDEX_CONFIG:
        fields = idx["fields"]
        # Handle descending keys if tuple provided
        if all(isinstance(f, tuple) for f in fields):
            index_spec = {f[0]: f[1] for f in fields}
        else:
            index_spec = {f: 1 for f in fields}

        client[parent_doi_id]['structures'].create_index(index_spec, name=idx["name"], unique=idx["unique"])
        print(f"Created index {idx['name']} on fields {fields} (unique={idx['unique']})")

    with open(path / config_file, "r") as f:
        config = json.load(f)

    return config

These indices (some of which are probably effective duplicates) significantly improved the performance of our larger datasets, allowing filtering and sorting across 5-ish million entries.

I suggest putting an INDEX_CONFIG = "extended" | "standard" | "minimal".\

It might be nice to automatically add the single field indices as default and potentially the compound indices as opt-ins.
Will have to investigate whether this can be cross compatible with the alternative backend (elasticsearch) or if this should be isolated to mongo only.

Start by locating the MongoDB provider's index setup and the INDEX_CONFIG/create_index logic described in the issue, then compare how the Elasticsearch backend handles indexing. Done means providers can select an index level, with documented defaults and compatibility verified for both backends.