apache / apache/beam

[Bug]: The default config for FirestoreV1.write().batchWrite() on smaller java jobs has an abysmal throughput

Open
#27,170 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage bug io java P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Firestore has an 500 write/sec limitation, the java implementation splits up that quota among workers. Although it has a write rampup, the throughput is being influenced by the intial value a lot. The problem is that the default split is 1/500. So smaller jobs with just a few workers, start with 1/s and ramp up very slowly. If the job isn't sufficiently big enough, the throughtput stays low through the whole execution.

https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java#L119-L120

The note is there that it should be set, but you can easily miss it:
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosOptions.java#L258-L275

https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java#L594-L680

IMO the 500 as the default value should be decreased to a sensible much smaller one.

Also due to how the check is implemented at the write DoFn even that 1 batch write / second doesn't have the maximum size it could have as it doesn't collect the new entries while "throttling" it's just sitting doing nothing. So instead of collecting messages until limit + blocking till throttling, it just collects some (many cases 1) message and blocks for the throttling.

I would suggest that the blocking should only happen when the write is ready to be executed.
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreV1WriteFn.java#L293-L325

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start with RpcQosImpl.java and RpcQosOptions.java, then trace the throttling path in FirestoreV1WriteFn.java. Compare the default quota split with the batching behavior described in the issue. Done means small jobs receive a sensible default throughput and batching waits at execution rather than idling before collecting entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.