letsencrypt / letsencrypt/pebble

wfe.go `Order()` takes the order read lock recursively, which can wedge the whole DB

Open
#554 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
787
Forks
176
PR merge metrics
No merged PRs in 30d

Description

I've been trying to diagnose some flaky mod_md tests which we've started running in the apache/httpd CI using pebble. It looks like pebble is getting wedged and stops responding in some conditions - but it's not easily reproducible. Claude has identified some locking issues - the first one looks trivially correct, I submitted a PR for that in #553

The second one is a bit more complex and looks plausible but I've not done anything with Go, so I apologise if this is all LLM hallucination. This is the Claude analysis verbatim:

Version: v2.10.1, also main (13f2ac3714)

WebFrontEndImpl.Order() read-locks the order and holds that lock until it returns, then calls orderForDisplay(), which read-locks the same order again:

// wfe/wfe.go
order.RLock()                                    // 2069
orderAccountID := order.AccountID
defer order.RUnlock()
...
orderReq := wfe.orderForDisplay(order, request)  // 2088 -> order.RLock() again (1864)

sync.RWMutex doesn't allow recursive read locking. If another goroutine calls order.Lock() between the two RLock()s, the second RLock() blocks behind that writer, and the writer waits for the first RLock() to be released. That writer can be:

  • ca.CompleteOrder(), which takes order.Lock() after issuing; or
  • MemoryStore.GetOrderByID(), which takes order.Lock() on every lookup.

Once that has happened, any GetOrderByID() for that order holds m.RLock() and blocks in order.GetStatus()'s o.RLock(). Every m.Lock() then blocks (e.g. AddAccount() for new-account), and new m.RLock() calls queue behind it. Pebble keeps running but stops completing requests.

Seen with: Apache httpd's mod_md test suite, which polls an order immediately after finalizing it. Roughly one order in a few thousand hits it; CI jobs then hang until their timeout. Pebble log from one occurrence:

15:14:36 POST /finalize-order/ -> calling handler()
15:14:36 Order ip1aJF43... is fully authorized. Processing finalization
15:14:36 POST /my-order/ -> calling handler()        <- never responds
15:14:36 Issued certificate serial 56bdbc63f6a03859 for order ip1aJF43...
15:15:13 POST /my-order/ -> calling handler()        <- client retries, never responds
15:15:47 POST /sign-me-up -> calling handler()       <- never responds; neither does any later new-account

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in wfe/wfe.go at WebFrontEndImpl.Order() and orderForDisplay(), then inspect the order locking in ca.CompleteOrder() and MemoryStore.GetOrderByID(). Trace the RLock/RUnlock sequence and reproduce the intermittent wedge through the mod_md tests using Pebble; done means the recursive read-lock deadlock is eliminated without leaving the order or store blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.