googleapis / googleapis/google-cloud-node
[FR] Enable fast, parallel listing of all document names, including missing/phantom documents
- Dominant language
- TypeScript
- Stars
- 3.2k
- Forks
- 712
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 99
Description
## Problem Description
There is currently no efficient method to list all document names in a very large collection when it's required to also include "phantom" documents (i.e., documents that do not actually exist but do have subcollections).
The goal is to perform a full scan of a collection's document IDs, similar to a keys-only query, but with the ability to find phantom documents, and to do so in parallel to make it practical for large collections (e.g., 500k to 1M+ documents).
Currently, there are two primary APIs for this, but they are mutually exclusive in their capabilities:
1. `v1.listDocuments` with `showMissing: true`:
- *What it does well*: It correctly returns all document names, including phantoms.
- *The limitation*: It is an inherently sequential, paginated API. It does not support range queries (e.g., `__name__ > 'a' AND __name__ < 'b'`) or partition cursors. This forces a single, long-running sequential scan, which is unacceptably slow for large collections. A scan of ~420,000 documents can take over two hours.
2. `v1.partitionQuery + v1.runQuery`:
- *What it does well*: This is the recommended and highly effective method for parallelizing reads across a large collection. It is very fast.
- *The limitation*: The `runQuery` RPC does not support a `showMissing` flag. As a result, it only returns existing documents and silently skips over phantom documents, making it unsuitable for use cases that require a complete list of all document paths.
This leaves users with a difficult choice: either a correct but impractically slow scan, or a fast but incomplete one.
## Use Case & Business Impact
This limitation is a significant bottleneck for critical back-office operations, data migrations, and large-scale data integrity checks. For example, when migrating data, it's essential to have a complete map of the existing structure, including documents that serve as parents for subcollections.
The inability to parallelize this task means that scripts which should take minutes can take many hours, making them difficult to run, monitor, and recover from in a production environment. This increases operational risk and engineering cost.
## Proposed Solutions
To address this, we propose adding capabilities to the Firestore API that would bridge this gap. Any of the following solutions would be a massive improvement:
1. Add `showMissing: true` support to the `runQuery` RPC.
- This is perhaps the most direct solution. If `runQuery` supported `showMissing`, we could continue using the existing `partitionQuery` API to generate cursors and then execute parallel, keys-only queries that also find phantom documents. This seems like a natural extension of the existing parallel-scan pattern.
2. Enhance listDocuments to support partitioning.
- Add support for `startAt` and `endAt` cursors (like those from `partitionQuery`) to the `IListDocumentsRequest`. This would allow us to partition the keyspace and then make parallel `listDocuments` calls, each with `showMissing: true`.
- *Alternative*: Allow `where` filters on the `__name__` property within a `listDocuments` request. This would enable manual partitioning of the keyspace (e.g., by character ranges: `a*`, `b*`, etc.) and allow for parallel execution. The client library documentation for `listDocuments` states that `showMissing` may not be used with `orderBy`, so this restriction would need to be lifted to allow for manual range scans. (From docs: "Requests with `show_missing` may not specify `where` or `order_by`".)
3. Introduce a new, dedicated API for parallel keyspace enumeration.
- Create a new RPC specifically designed for this task. It could be a "partitionable list" or a "keyspace scan" API that takes a parent path and returns a stream of all document names (existing and missing) within that path, with built-in support for parallel execution. This would provide a purpose-built tool for a common and important administrative task.
4. Enhance `IListenRequest` to notify on phantom document changes.
- Extend the `listen` RPC to send notifications when phantom documents are
implicitly created (because a subcollection document is added) or removed
(because their last subcollection document is removed). This would allow
for real-time tracking of the complete keyspace, which is crucial for
maintaining live caches or indexes of a collection's structure.
5. Introduce a new API for traversing and watching parameterized paths.
- Provide a new API that accepts a fully parameterized path pattern (e.g.,
`users/{userId}/posts/{postId}/comments/{commentId}`). This API would
stream all matching documents, including phantom documents along the path,
and return the extracted, typed parameters for each document (e.g.,
`{ userId: 'user-123', postId: 'post-abc', ... }`). It should also
support watching for new documents that match the pattern. This is
conceptually similar to how Firebase Functions v2 triggers can be defined
with wildcards (e.g., `onDocumentWritten("users/{userId}/posts/{postId}")`),
which has proven to be a very powerful and intuitive pattern. This would
massively simplify complex data traversal and synchronization logic.
For us, enabling an efficient, parallel, and complete scan of a collection's keyspace is a very useful tool to have for managing Firestore data at scale. We understand this requires changes to the backend and would be grateful if you could consider this proposal and forward it to the appropriate team.
Contributor guide
Assessment
This issue has not been assessed yet.