Dataset Loses Harvest Object on WAF file Timestamp Change
- Dominant language
- XSLT
- Stars
- 136
- Forks
- 207
- PR merge metrics
- No merged PRs in 30d
Description
related issue: https://github.com/GSA/data.gov/issues/4505
## Summary:
When the timestamp of a WAF source file changes without any actual content modification, the metadata information disappears from the UI.
The root cause is the the harvest_object_id does not change with the new harvest_object_id.
This was confirmed through the following API calls:
/api/action/package_show?id=
/api/action/package_search?q=id:
Additionally, testing on the most recent version of CKAN with only the ckanext-harvest and ckanext-spatial extensions replicated the problem.
### Observations from Testing:
1. Manually run `ckan search-index rebuild ` resolved the issue, as the above API calls return correct value of harvest_object_id.
2. Found the code block which should refresh the solr index:
https://github.com/ckan/ckanext-spatial/blob/master/ckanext/spatial/harvesters/base.py#L709C1-L710C70
Testing with the following code changes yielded positive results:
Invoking `package_update ` instead of `package_index.index_package` resolved the issue.
OR
Addition of `model.Session.commit()` before invoking package_index.index_package also resolved the issue.
OR
calling `rebuild` index instead of package_index.index_package does not solve the issue unless `model.Session.commit()` was called before invoking the `rebuild`.
It seems that the assumption that `package_index.index_package` doesn't need a database commit to refresh Solr isn't valid based on the tests conducted above.
Any alternative solutions to address this issue?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ckanext/spatial/harvesters/base.py around lines 709-710 and reproduce the issue using the listed package_show and package_search API calls. Compare the observed effects of package_update, package_index.index_package, rebuild, and committing the session first. Done means the harvest_object_id remains correct in the API responses and the metadata stays visible after a WAF timestamp-only change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100