pimutils / pimutils/vdirsyncer

Server timeout when prefetching a large amount of refs

Open
#1,127 4 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
185
Avg merge
11h 49m
Merged PRs (30d)
1

Description

client: vdirsyncer-0.19.2, also reproduced with HEAD of main branch.
server: davmail-6.2.0
python: 3.11.9
OS: Linux (NixOS)

debug: <CIMultiDictProxy('Server': 'DavMail Gateway 6.2.0-3464', 'DAV': '1, calendar-access, calendar-auto-schedule, calendarserver-private-events, addressbook', 'Date': 'Fri, 26 Apr 2024 07:13:21 +0000
', 'Expires': 'Fri, 26 Apr 2024 07:13:21 +0000', 'Cache-Control': 'private, max-age=0', 'Transfer-Encoding': 'chunked', 'Content-Type': 'text/xml;charset=UTF-8', 'Connection': 'keep-alive')>
debug: <StreamReader>
error: Unknown error occurred for calendar_cisco/calendar:
error: Use `-vdebug` to see the full traceback.
debug:   File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/cli/tasks.py", line 72, in sync_collection
debug:     await sync.sync(
debug:   File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/sync/__init__.py", line 146, in sync
debug:     a_nonempty = await a_info.prepare_new_status()
debug:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug:   File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/sync/__init__.py", line 66, in prepare_new_status
debug:     async for href, item, etag in self.storage.get_multi(prefetch):
debug:   File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/storage/dav.py", line 529, in get_multi
debug:     await response.content.read()
debug:   File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 373, in read
debug:     block = await self.readany()
debug:             ^^^^^^^^^^^^^^^^^^^^
debug:   File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 395, in readany
debug:     await self._wait("readany")
debug:   File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 301, in _wait
debug:     with self._timer:
debug:   File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/helpers.py", line 735, in __exit__
debug:     raise asyncio.TimeoutError from None

A potential fix for the issue can be found in https://github.com/martiert/vdirsyncer/tree/batch_prefetch which batches the requests into chunks of 20

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in vdirsyncer/storage/dav.py at get_multi, reached from sync/init.py during prepare_new_status. Review the batch_prefetch branch and its proposed chunks of 20, then reproduce a sync with a large prefetch against the reported DavMail setup. Done means large prefetches no longer time out while reading the response.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.