apache / apache/beam

SolrIO: Expose commitWithin to the Solr write

Open
#18,737 0 comments 0 reactions 0 assignees View on GitHub
improvement io java P4 solr
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

A good improvement for to SolrIO would be to allow the caller to provide a `commitWithin` parameter.  Currently the batch is passed to the underlying `solrClient` which results in defaulting to the configured server behavior.

The justification for exposing this is that the collection in the target SOLR server might be configured in a way that is not suitable for this beam job.  E.g. a server tuned to accept real time updates with fast flush times from streaming Beam job 1, while Beam job 2 is doing an nightly bulk load.

This is related to (BEAM-3849, BEAM-3848, BEAM-3820) and should be considered together. I understand that the policy of Beam is not to expose parameters for tuning.  When it comes to the IOs which are for interfacing with external systems I recommend this policy be reconsidered.  The IO modules typically wrap clients to target systems (`CloudSolrClient` in this case) which all have tunable parameters for good reason. My recommendation would be to keep `SolrIO.write()` providing sensible defaults but expose an additional builder e.g. `SolrIO.writeBuilder().withCommitWithinMs(300000).withBatchSize(9000).build()` .

Please feel free to assign to me if of interest and I'll provide a PR.

Imported from Jira [BEAM-3862](https://issues.apache.org/jira/browse/BEAM-3862). Original Jira may contain additional context.
Reported by: timrobertson100.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.