pimutils / pimutils/vdirsyncer
Server timeout when prefetching a large amount of refs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 185
- Avg merge
- 11h 49m
- Merged PRs (30d)
- 1
Description
client: vdirsyncer-0.19.2, also reproduced with HEAD of main branch.
server: davmail-6.2.0
python: 3.11.9
OS: Linux (NixOS)
debug: <CIMultiDictProxy('Server': 'DavMail Gateway 6.2.0-3464', 'DAV': '1, calendar-access, calendar-auto-schedule, calendarserver-private-events, addressbook', 'Date': 'Fri, 26 Apr 2024 07:13:21 +0000
', 'Expires': 'Fri, 26 Apr 2024 07:13:21 +0000', 'Cache-Control': 'private, max-age=0', 'Transfer-Encoding': 'chunked', 'Content-Type': 'text/xml;charset=UTF-8', 'Connection': 'keep-alive')>
debug: <StreamReader>
error: Unknown error occurred for calendar_cisco/calendar:
error: Use `-vdebug` to see the full traceback.
debug: File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/cli/tasks.py", line 72, in sync_collection
debug: await sync.sync(
debug: File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/sync/__init__.py", line 146, in sync
debug: a_nonempty = await a_info.prepare_new_status()
debug: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug: File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/sync/__init__.py", line 66, in prepare_new_status
debug: async for href, item, etag in self.storage.get_multi(prefetch):
debug: File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/storage/dav.py", line 529, in get_multi
debug: await response.content.read()
debug: File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 373, in read
debug: block = await self.readany()
debug: ^^^^^^^^^^^^^^^^^^^^
debug: File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 395, in readany
debug: await self._wait("readany")
debug: File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 301, in _wait
debug: with self._timer:
debug: File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/helpers.py", line 735, in __exit__
debug: raise asyncio.TimeoutError from None
A potential fix for the issue can be found in https://github.com/martiert/vdirsyncer/tree/batch_prefetch which batches the requests into chunks of 20
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in vdirsyncer/storage/dav.py at get_multi, reached from sync/init.py during prepare_new_status. Review the batch_prefetch branch and its proposed chunks of 20, then reproduce a sync with a large prefetch against the reported DavMail setup. Done means large prefetches no longer time out while reading the response.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100