SolrIO: Expose commitWithin to the Solr write
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
A good improvement for to SolrIO would be to allow the caller to provide a `commitWithin` parameter. Currently the batch is passed to the underlying `solrClient` which results in defaulting to the configured server behavior.
The justification for exposing this is that the collection in the target SOLR server might be configured in a way that is not suitable for this beam job. E.g. a server tuned to accept real time updates with fast flush times from streaming Beam job 1, while Beam job 2 is doing an nightly bulk load.
This is related to (BEAM-3849, BEAM-3848, BEAM-3820) and should be considered together. I understand that the policy of Beam is not to expose parameters for tuning. When it comes to the IOs which are for interfacing with external systems I recommend this policy be reconsidered. The IO modules typically wrap clients to target systems (`CloudSolrClient` in this case) which all have tunable parameters for good reason. My recommendation would be to keep `SolrIO.write()` providing sensible defaults but expose an additional builder e.g. `SolrIO.writeBuilder().withCommitWithinMs(300000).withBatchSize(9000).build()` .
Please feel free to assign to me if of interest and I'll provide a PR.
Imported from Jira [BEAM-3862](https://issues.apache.org/jira/browse/BEAM-3862). Original Jira may contain additional context.
Reported by: timrobertson100.
Contributor guide
Assessment
This issue has not been assessed yet.