apache / apache/couchdb

/_scheduler/docs/_replicator/{doc_id} sometimes returns 500

Open
#1,000 1 comment 0 reactions 0 assignees View on GitHub
api replication
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

Occasionally while querying the `/_scheduler/docs/_replicator/{doc_id}` endpoint for a recently created replication, I get a 500 error:

HTTP/1.1 500 Internal Server Error
Content-Length: 70
Cache-Control: must-revalidate
Content-Type: application/json
Date: Thu, 16 Nov 2017 20:14:25 GMT
Server: CouchDB/2.1.0 (Erlang OTP/17)
X-Couch-Request-Id: 65913f4727
X-Couch-Stack-Hash: 3194022798
X-Couchdb-Body-Time: 0

{"error":"unknown_error","reason":"function_clause","ref":3194022798}

In the logs, I see something like this:

[notice] 2017-11-16T20:03:25.316906Z nonode@nohost <0.19112.66> -------- Replication `b9d7fa6c1c76ccbfa6549b93bf7c1e78` completed (triggered by `001415590d64afd6ff7c5c7e7c00fd3f`)
[notice] 2017-11-16T20:03:25.317197Z nonode@nohost <0.353.0> -------- couch_replicator_scheduler: Job {"b9d7fa6c1c76ccbfa6549b93bf7c1e78",[]} completed normally
[error] 2017-11-16T20:03:25.319390Z nonode@nohost <0.18804.66> 332478bac4 req_err(3194022798) unknown_error : function_clause
[<<"couch_replicator_httpd_util:update_db_name/1 L182">>,<<"couch_replicator_httpd:handle_scheduler_doc/3 L138">>,<<"chttpd:process_request/1 L295">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_p_do_apply/3 L237">>]
[notice] 2017-11-16T20:03:25.319694Z nonode@nohost <0.18804.66> 332478bac4 localhost:6002 172.17.0.1 undefined GET /_scheduler/docs/_replicator/001415590d64afd6ff7c5c7e7c00fd3f 500 ok 1

Based both on my observations, and the log output, I'm guessing it's a race condition of some sort; when the query happens at precisely the right moment with respect to the replication status update (or perhaps specifically a completion), the error occurs.

I have a pretty consistent reproduction case (estimate 90% of the time) in Go, as part of a much larger project. I tried reproducing it in Bash, and wasn't able to. If it's helpful to debug, I can attempt to produce a minimal reproduction case in Go, along with compilation and execution instructions.

Contributor guide

Open the contributing guide

Research direction

Start with the stack trace entries couch_replicator_httpd_util:update_db_name/1 at L182 and couch_replicator_httpd:handle_scheduler_doc/3 at L138 while querying /_scheduler/docs/_replicator/{doc_id}. Reproduce the failure around a recently completed replication, using the described Go case if available; done means the endpoint no longer returns a 500 during that status transition.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
api, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.