Mango bookmarks are unreliable when the last document of the previous page changes
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
[NOTE]: # ( Describe the problem you're encountering. )
[TIP]: # ( Do NOT give us access or passwords to your actual CouchDB! )
When the last document returned for the bookmark page is changed in a way that it does no longer match the selector of the query, results for the next bookmark page are affected. The first expected document for this page is skipped in this case.
## Steps to Reproduce
[NOTE]: # ( Include commands to reproduce, if possible. curl is preferred. )
1. Create around 25 documents with `_id` `ID-01` to `ID-25` and `flag: true`.
2. Run a query to retrieve the first 10 docs with `flag: true`.
`{"selector":{"flag", true}, "limit": "10", "bookmark": null}`
3. Run the query again for the next page by using the returned bookmark. Docs for `ID-11` to `ID-20` are returned.
4. Modify the last doc of the first page (`ID-10`). Set `flag: false`.
5. Run the previous query with the bookmark again. Docs for `ID-12` to `ID-21` are now returned.
```shell
#!/bin/sh
# Dependencies: curl, jq
# Auth needs to be disabled for the database.
DB=testcase
URL=http://localhost:5984/$DB
echo "Creating docs..."
for i in `seq 25`; do
id=`printf "ID-%02d" $i`
curl "$URL/$id" -X PUT -H 'Content-Type: application/json' -d '{"flag": true}'
done
echo
res=`curl "$URL/_find" -X POST -H 'Content-Type: application/json' -d '{"selector": {"flag": true}, "limit": 10}' --globoff -sS`
bookmark=`echo "$res" | jq .bookmark`
echo "--- RESPONSE (page 1) ---"
echo "$res"
echo "--- RESPONSE END ---"
echo
echo Bookmark: $bookmark
echo
q='{"selector": {"flag": true}, "limit": 10, "bookmark": '$bookmark'}'
echo "Query: $q"
res=`curl "$URL/_find" -X POST -H 'Content-Type: application/json' -d "$q" --globoff -sS`
echo "--- RESPONSE (page 2) ---"
echo "$res"
echo "--- RESPONSE END ---"
echo
rev=`curl "$URL/ID-10" | jq ._rev`
echo "Updating ID-10 (_rev: $rev)"
curl "$URL/ID-10" -X PUT -H 'Content-Type: application/json' -d '{"flag": false, "_rev": '$rev'}'
echo
echo "Query: $q"
res=`curl "$URL/_find" -X POST -H 'Content-Type: application/json' -d "$q" --globoff -sS`
echo "--- RESPONSE (page 2 after update) ---"
echo "$res"
echo "--- RESPONSE END ---"
echo
```
## Expected Behaviour
[NOTE]: # ( Tell us what you expected to happen. )
The results for page 2 should not be affected by changing the last document returned for page 1. I would expect that the documents with `ID-11` to `ID-20` are returned when using the bookmark.
## Your Environment
[TIP]: # ( Include as many relevant details about your environment as possible. )
[TIP]: # ( You can paste the output of curl http://YOUR-COUCHDB:5984/ here. )
```json
{"couchdb":"Welcome","version":"3.1.1","git_sha":"ce596c65d","uuid":"86ee0da8b516fc3b05dabfc95505136b","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
```
## Additional Context
[TIP]: # ( Add any other context about the problem here. )
We are using Mango bookmarks to update documents in batches during data migrations. During those migrations we noticed that some documents were not properly updated. Further investigation revealed that the same documents were affected when repeating the migrations against the same database state.
Contributor guide
Research direction
Start with the Mango _find bookmark query and reproduce the behavior using the inline curl/jq shell steps against CouchDB. Done means changing the last document from the first page does not skip the first expected document on the next page, so ID-11 through ID-20 are returned.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100