Indexing: unnecessary permission documents are created
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
From slack earlier:
[The] discrepancy between the number of the permission documents and the indexed dvobjects is still there, even on an index created from scratch. So, it cannot be explained by permission documents being left behind when objects are deleted.
curl "http://172.31.88.120:8983/solr/collection1/query?debug=query&q=definitionPointDocId:*&rows=0"
The sum ofq=dvObjectType:dataverses,...:datasetsand...:filesis 3715147.
curl "http://172.31.88.120:8983/solr/collection1/query?debug=queryq=*:*&rows=0"returns the number that is the sum of the 2 numbers above.This number of extra permission docs is growing, the longer the index is in place.
On a closer look:
none of [these] are literal duplicates; i.e., there are no datasets with multiple
definitionPointDocId:dataset_NNNNNpermission docs.
However, there multiple cases of, for example, both
definitionPointDocId:dataset_NNNNN
and
definitionPointDocId:dataset_NNNNN_draftpermission docs, when the only indexed dvobject document for it isid:dataset_NNNNN.
Seeing such cases for both datasets and files. Also seeing cases that are reverse of the above (i.e., both perm. documents exist when only an indexed draft document is present).
Meaning, likely, that in some cases we fail to remove the permission doc for the draft when we publish; and in some - create permission docs for published documents while they are still in draft.
For example, we may be creatingdefinitionPointDocId:file_MMMMMpermission documents for unpublished files, when the parent datasets has published versions.
This does not appear to corrupt the index in a way that would affect the accuracy of the results of searches. But the redundant permission docs are likely slowing down the lookups on such.
We should also experiment with dropping creating permission documents for public indexed dvobjects altogether, in combination with the new "avoid expensive solr join" mechanism (#10555). Still debating if that should be handled as a separate issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Solr count queries in the issue and compare published and draft permission documents with indexed dvobjects. Trace the indexing paths that create and remove permission documents, including the avoid-expensive-Solr-join mechanism from #10555. Done means redundant permission documents are no longer created or left behind, with the published and draft cases verified.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100