ckan / ckan/ckanapi

SOLR error: exceeded limit of maxWarmingSearchers=2

Open
#118 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
212
Forks
81
PR merge metrics
No merged PRs in 30d

Description

When I load a large number of datasets with multiple processes using ckanapi like this:
``` ckanapi load datasets -I /data/ckan.jsonl.gz -z -p 4 -c /etc/ckan/testing/development.ini ```

Eventually, a Solr error is thrown:
``` ERROR [ckan.model.modification] Solr returned an error: (u'Solr responded with an error (HTTP 503): [Reason: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later.]',) ```

I could increase the number of WarmingSearchers in Solr or decrease the number of workers for the load, but IMHO the bigger issue here is whether it could be possible for bulk loading to have an option to commit to Solr only after loading all datasets, not one by one, as it still takes a long time this way (9 hours for 130k datasets to load using single worker)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.