[Bug]: The default config for FirestoreV1.write().batchWrite() on smaller java jobs has an abysmal throughput
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
Firestore has an 500 write/sec limitation, the java implementation splits up that quota among workers. Although it has a write rampup, the throughput is being influenced by the intial value a lot. The problem is that the default split is 1/500. So smaller jobs with just a few workers, start with 1/s and ramp up very slowly. If the job isn't sufficiently big enough, the throughtput stays low through the whole execution.
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java#L119-L120
The note is there that it should be set, but you can easily miss it:
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosOptions.java#L258-L275
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java#L594-L680
IMO the 500 as the default value should be decreased to a sensible much smaller one.
Also due to how the check is implemented at the write DoFn even that 1 batch write / second doesn't have the maximum size it could have as it doesn't collect the new entries while "throttling" it's just sitting doing nothing. So instead of collecting messages until limit + blocking till throttling, it just collects some (many cases 1) message and blocks for the throttling.
I would suggest that the blocking should only happen when the write is ready to be executed.
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreV1WriteFn.java#L293-L325
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start with RpcQosImpl.java and RpcQosOptions.java, then trace the throttling path in FirestoreV1WriteFn.java. Compare the default quota split with the batching behavior described in the issue. Done means small jobs receive a sensible default throughput and batching waits at execution rather than idling before collecting entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100