OpenRefine / OpenRefine/OpenRefine
Configurable batch size and sleep time for the reconciliation and data extension operations
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 12k
- Forks
- 2.2k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 27
Description
The reconciliation operation currently sends requests to services by batches of 10 rows.
This should be configurable by the user. Similarly, the sleep time to sleep between batches should also be configurable.
The same applies to the data extension operation, which also groups requests by batches.
Reconciliation services could potentially suggest sensible default values for these in their manifest.
Proposed solution
Reconciliation batch size and sleep time could be configured in the main reconciliation screen (just like the sleep time is configured for fetch URLs operation). Similarly for the data extension.
These settings could be stored in the preferences, perhaps on a per-service basis?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how reconciliation and data extension currently group requests into batches, then compare their configuration with the existing sleep-time setting for fetch URLs. The proposed work covers screen settings, preference persistence, and possible service-manifest defaults; it is done when users can configure batch size and sleep time for both operations and those values are applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100