keybase / keybase/client

Git push can indefinitely block "waiting for journal bytes to flush"

Open
#8,915 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

acked
Dominant language
Go
Stars
9.2k
Forks
1.3k
Avg merge
12h 58m
Merged PRs (30d)
56

Description

On one occasion:

$ git push kb master
Initializing Keybase... done.
Syncing with Keybase... done.
Counting and decrypting: 388 objects... done.
Preparing and encrypting: (100.00%) 388/388 objects... done.
Indexing hashes: (100.00%) 388/388 objects... done.
Indexing CRCs: (100.00%) 388/388 objects... done.
Indexing offsets: (100.00%) 388/388 objects... done.
Syncing encrypted data to Keybase: (0.00%) 0.00/20.89 KB...

...waited indefinitely at that "0.00%" twice, shortly after the repository in question, succeeding on later 3rd try.

Logs from the second failed instance look like:

2017-10-08T15:45:15.432523-05:00 ▶ [DEBU git(FBO bf3b6829) folder_branch_ops.go:1576] 3006 SetInitialHeadFromServer, revision=15 (unmerged) [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]
2017-10-08T15:45:15.432612-05:00 ▶ [DEBU git(PRE) prefetcher.go:218] 3007 Prefetching block by request from upstream component. Priority: 1073741824
2017-10-08T15:45:15.432683-05:00 ▶ [DEBU git(FBO bf3b6829) folder_branch_ops.go:1602] 3008 Head MD already set to revision 15 (unmerged), no need to set initial head again [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]
2017-10-08T15:45:15.432732-05:00 ▶ [DEBU git(FBO bf3b6829) folder_branch_ops.go:1604] 3009 SetInitialHeadFromServer, revision=15 (unmerged) done: <nil> [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]
2017-10-08T15:45:15.432777-05:00 ▶ [DEBU git(FBO bf3b6829) folder_branch_ops.go:1717] 300a getRootNode [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]
2017-10-08T15:45:15.432845-05:00 ▶ [DEBU git(FBO bf3b6829) folder_branch_ops.go:1750] 300c getRootNode done: NodeID(0xc42020c800) <nil> [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]
2017-10-08T15:45:15.432935-05:00 ▶ [DEBU git kbfs_ops.go:510] 300d Done: <nil> [tags:GITCMDID=7LXHNfvEcBbcSoPmRO5Skg,GITID=vF6E-T-EssErOZNECCPcsQ]
2017-10-08T15:45:15.433014-05:00 ▶ [DEBU git(FBO bf3b6829) folder_branch_ops.go:4201] 300e SyncAll [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]
2017-10-08T15:45:15.432802-05:00 ▶ [WARN git block_retrieval_queue.go:314] 300b Block missing for disk block cache metadata update
2017-10-08T15:45:15.433148-05:00 ▶ [DEBU git(FBO bf3b6829) folder_branch_ops.go:2769] 300f Blocking until squash finishes [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]
2017-10-08T15:45:15.433199-05:00 ▶ [DEBU git(FBO bf3b6829) folder_branch_ops.go:4208] 3010 SyncAll done: <nil> [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]
2017-10-08T15:45:15.433258-05:00 ▶ [DEBU git runner.go:440] 3011 Waiting for 21391 journal bytes to flush [tags:GITID=vF6E-T-EssErOZNECCPcsQ,GITCMDID=7LXHNfvEcBbcSoPmRO5Skg]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the journal flush wait in runner.go:440 and the preceding SyncAll path in folder_branch_ops.go, then compare it with the block-retrieval warning in block_retrieval_queue.go:314. Use the supplied logs to determine why the journal does not finish flushing and reproduce the intermittent Git push hang. Done means the push no longer blocks indefinitely and a regression test covers the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.