SpannerIO.read() parallelism limited by partition count
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Setting the partition size / count is not possible:
```
Note: This hint is currently ignored by sessions.partitionQuery and sessions.partitionRead requests.
```
https://cloud.google.com/spanner/docs/reference/rest/v1/PartitionOptions
Adding a Reshuffle might be the only choice, but it adds time and resource usage.
cc: [~chamikara][~nielm]
Imported from Jira [BEAM-11722](https://issues.apache.org/jira/browse/BEAM-11722). Original Jira may contain additional context.
Reported by: udim.
Contributor guide
Research direction
Start with SpannerIO.read() and the Google Cloud PartitionOptions documentation linked in the issue. Trace how partitionQuery and partitionRead requests are built, then clarify the desired partition size/count behavior and define tests showing those options are honored without requiring a Reshuffle.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- cloud, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100