Cannot index deleted documents with Mango
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
Mango indexes cannot be used to index deleted documents, though it would sure be nice if they could.
## Expected Behavior
Given a Mango index like this:
```json
{
"index": {
"partial_filter_selector": {
"_deleted": {
"$exists": true
}
},
"fields": ["_deleted"]
},
"ddoc": "deleted",
"name": "deleted",
"type": "json"
}
```
A query against that index should return any deleted documents. For example, this request with curl:
```
$ curl -H 'content-type:application/json' $COUCH_URL/mango-delete-test/_find -d '{"selector":{"_deleted":{"$exists":true}},"use_index":"deleted"}'
{"docs":[
{"_id":"6231e1e26655d19f51ea23e183001081","_rev":"1-967a00dff5e02add41819138abb3284d","_deleted":true}
],
"bookmark": "nil"}
```
It should return deleted documents.
## Current Behavior
Indexing the `_deleted` field creates indexes that return no documents. For example, consider this query using the index from above:
```
$ curl -H 'content-type:application/json' $COUCH_URL/mango-delete-test/_find -d '{"selector":{"_deleted":{"$exists":true}},"use_index":"deleted"}'
{"docs":[
],
"bookmark": "nil"}
```
It returns nothing, but it's clear that Mango considers the index valid and indeed can detect that it has been requested.
Whereas a Map-Reduce view can be used to filter a changes feed to return changes pertaining to deleted documents, using a Mango index in this way results in an error:
```
$ curl "$COUCH_URL/mango-delete-test/_changes?view=deleted/deleted&filter=_view"
{"error":"error","reason":"{timeout,{gen_server,call,\n [couch_proc_manager,\n {get_proc,{doc,<<\"_design/deleted\">>,\n {1,\n [<<124,254,100,228,235,159,75,216,229,\n 48,135,104,37,85,76,42>>]},\n {[{<<\"language\">>,<<\"query\">>},\n {<<\"views\">>,\n {[{<<\"deleted\">>,\n {[{<<\"map\">>,\n {[{<<\"fields\">>,\n {[{<<\"_deleted\">>,\n <<\"asc\">>}]}},\n {<<\"partial_filter_selector\">>,\n {[{<<\"_deleted\">>,\n {[{<<\"$exists\">>,\n true}]}}]}}]}},\n {<<\"reduce\">>,<<\"_count\">>},\n {<<\"options\">>,\n {[{<<\"def\">>,\n {[{<<\"partial_filter_selector\">>,\n {[{<<\"_deleted\">>,\n {[{<<\"$exists\">>,\n true}]}}]}},\n {<<\"fields\">>,\n [<<\"_deleted\">>]}]}}]}}]}}]}}]},\n [],false,[]},\n {<<\"_design/deleted\">>,\n <<\"1-7cfe64e4eb9f4bd8e530876825554c2a\">>}},\n 5000]}}"}
```
As such, it is currently not possible to index deleted documents or to filter changes for deleted documents using a Mango index.
## Possible Solution
:woman_shrugging:
If a Map-Reduce view can be used to filter a changes feed for deleted (or un-deleted) documents, then perhaps Mango indexes can be applied in a similar way to achieve the same result.
## Steps to Reproduce (for bugs)
Here are the curl commands I used to produce this issue:
```
# create the db
$ curl -X PUT $COUCH_URL/mango-delete-test
# create and delete a document
$ curl -X POST -H 'content-type:application/json' $COUCH_URL/mango-delete-test -d '{}'
$ curl -X DELETE $COUCH_URL/mango-delete-test/{id}?rev={rev}
# create the index
$ curl -X POST -H 'content-type:application/json' $COUCH_URL/mango-delete-test/_index -d '{"index":{"partial_filter_selector":{"_deleted":{"$exists":true}},"fields":["_deleted"]},"ddoc":"deleted","name":"deleted","type":"json"}'
# query the index
$ curl -H 'content-type:application/json' $COUCH_URL/mango-delete-test/_find -d '{"selector":{"_deleted":{"$exists":true}},"use_index":"deleted"}'
```
That final request should yield a response that includes no documents.
## Context
It is occasionally important to find deleted documents en masse, such as while managing tombstones under conditions where other mitigation techniques are not available.
## Your Environment
* Version used: 2.1.1
* Browser Name and version: curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
* Operating System and version (desktop or mobile): Ubuntu 16.04.4 LTS
* Link to your project: N/A
Contributor guide
Research direction
Start by running the listed curl reproduction against the Mango _index and _find endpoints, then compare it with the Map-Reduce view and _changes?view=_view behavior described in the issue. Done means Mango can index deleted documents and use such an index to filter changes, with the reproduction requests returning the deleted document instead of an empty result or timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100