letsencrypt / letsencrypt/pebble
wfe.go `Order()` takes the order read lock recursively, which can wedge the whole DB
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 787
- Forks
- 176
- PR merge metrics
- No merged PRs in 30d
Description
I've been trying to diagnose some flaky mod_md tests which we've started running in the apache/httpd CI using pebble. It looks like pebble is getting wedged and stops responding in some conditions - but it's not easily reproducible. Claude has identified some locking issues - the first one looks trivially correct, I submitted a PR for that in #553
The second one is a bit more complex and looks plausible but I've not done anything with Go, so I apologise if this is all LLM hallucination. This is the Claude analysis verbatim:
Version: v2.10.1, also main (13f2ac3714)
WebFrontEndImpl.Order()read-locks the order and holds that lock until it returns, then callsorderForDisplay(), which read-locks the same order again:// wfe/wfe.go order.RLock() // 2069 orderAccountID := order.AccountID defer order.RUnlock() ... orderReq := wfe.orderForDisplay(order, request) // 2088 -> order.RLock() again (1864)
sync.RWMutexdoesn't allow recursive read locking. If another goroutine callsorder.Lock()between the twoRLock()s, the secondRLock()blocks behind that writer, and the writer waits for the firstRLock()to be released. That writer can be:
ca.CompleteOrder(), which takesorder.Lock()after issuing; orMemoryStore.GetOrderByID(), which takesorder.Lock()on every lookup.Once that has happened, any
GetOrderByID()for that order holdsm.RLock()and blocks inorder.GetStatus()'so.RLock(). Everym.Lock()then blocks (e.g.AddAccount()fornew-account), and newm.RLock()calls queue behind it. Pebble keeps running but stops completing requests.Seen with: Apache httpd's mod_md test suite, which polls an order immediately after finalizing it. Roughly one order in a few thousand hits it; CI jobs then hang until their timeout. Pebble log from one occurrence:
15:14:36 POST /finalize-order/ -> calling handler() 15:14:36 Order ip1aJF43... is fully authorized. Processing finalization 15:14:36 POST /my-order/ -> calling handler() <- never responds 15:14:36 Issued certificate serial 56bdbc63f6a03859 for order ip1aJF43... 15:15:13 POST /my-order/ -> calling handler() <- client retries, never responds 15:15:47 POST /sign-me-up -> calling handler() <- never responds; neither does any later new-account
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in wfe/wfe.go at WebFrontEndImpl.Order() and orderForDisplay(), then inspect the order locking in ca.CompleteOrder() and MemoryStore.GetOrderByID(). Trace the RLock/RUnlock sequence and reproduce the intermittent wedge through the mod_md tests using Pebble; done means the recursive read-lock deadlock is eliminated without leaving the order or store blocked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100