ckan-pycsw load fails with 3.5M datasets

The ckan-pycsw load job isn't built to handle a large number of datasets. It pulls all the datasets in memory, then all the existing pycsw records, then does set operations in order to figure out new, changed, and deleted datasets. We started seeing the job run out of memory on the machine when working with 3.5 million datasets in CKAN. Additionally, as the datasets grow, the job expects to be the sole worker, running as a cron job once per day. It would be nice if this work could be split up over time and machines.

As a hack, I did some work to fetch datasets in batches of 1000 and process them. But ultimately, I think you would want the pycsw update to happen in "real time" as part of harvesting. If the dataset is updated, it should be updated in pycsw. If the package is deleted, it should be removed. If the dataset doesn't exist, add it.

Start by locating the ckan-pycsw load job and tracing how it fetches CKAN datasets and existing pycsw records. Compare the batch workaround described in the issue with a real-time harvesting path; done means updates, deletions, and new datasets stay synchronized without loading the full collections or requiring a sole daily worker.

Written by the indexing model from the issue text.