pimutils / pimutils/vdirsyncer

An "unsafe character" (+) seems to be derailing the sync process when syncing with Google.

Open
#865 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
185
Avg merge
11h 49m
Merged PRs (30d)
1

Description

Before you submit bug reports: https://vdirsyncer.pimutils.org/en/stable/contributing.html

Things to include in your bugreport:

  • Your vdirsyncer version
    0.16.8
  • Your Python version
    Python 3.7.9
  • Your operating system
    alpine docker container on ubuntu
  • Your config file
[general]
status_path = "/calendardata/.vdirsyncer/status/"

[pair calendar_test]
a = "my_calendar_local"
b = "my_calendar_remote"
collections = ["from a", "from b"]

[storage my_calendar_local]
type = "filesystem"
path = "/calendardata/test_calendar/"
fileext= ".ics"

[storage my_calendar_remote]
type = "google_calendar"
token_file = "/calendardata/.google_auth/"
client_id = "supersecretclientid"
client_secret = "supersercretsecret"
  • Use vdirsyncer -vdebug for debug output. The output is sensitive, but
    please attach at least the last few lines before the error (if applicable),
    censored as necessary. This is almost always the most useful information.

There was a debugging note right before the error regarding an "unsafe character" which is the + symbol translated into %2b. I included snippets of the debug output surrounding the offending character. Ellipses show gaps (that I would think are fairly irrelevant) cut out.

debug: Using 1 maximal workers.
Syncing calendar_test/calendar_name
debug: PROPFIND https://apidata.googleusercontent.com/caldav/v2/calendar_name/events/
debug: {'User-Agent': 'randomnumbers.apps.googleusercontent.com', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <D:propfind xmlns:D="DAV:">\n                <D:prop>\n                    <D:resourcetype/>\n                    <D:getcontenttype/>\n                    <D:getetag/>\n                </D:prop>\n            </D:propfind>\n            '
debug: Sending request...
debug: 207
debug: {'Date': 'Mon, 01 Feb 2021 23:05:42 GMT', 'Pragma': 'no-cache', 'Content-Type': 'text/xml; charset=UTF-8', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'Vary': 'Origin, X-Origin, Referer', 'Content-Encoding': 'gzip', 'Server': 'ESF', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Alt-Svc': 'h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'}
...
debug: b'<?xml version="1.0" encoding="UTF-8"?>\n<D:multistatus xmlns:D="DAV:" xmlns:caldav="urn:ietf:params:xml:ns:caldav" xmlns:cs="http://calendarserver.org/ns/" xmlns:ical="http://apple.com/ns/ical/">\n ... ... <D:response xmlns:carddav="urn:ietf:params:xml:ns:carddav" xmlns:cm="http://cal.me.com/_namespace/" xmlns:md="urn:mobileme:davservices">\n  <D:href>/caldav/v2/calendar_name/events/WEBEX-MEETING+CENTER-6.0686402-108582152.ics</D:href>\n  <D:propstat>\n   <D:status>HTTP/1.1 200 OK</D:status>\n   <D:prop>\n    <D:getcontenttype>text/calendar; component=vevent</D:getcontenttype>\n    <D:getetag>"63518914751"</D:getetag>\n    <D:resourcetype/>\n   </D:prop>\n  </D:propstat>\n </D:response>\n ... ...D:response xmlns:carddav="urn:ietf:params:xml:ns:carddav" xmlns:cm="http://cal.me.com/_namespace/" xmlns:md="urn:mobileme:davservices">\n  <D:href>/caldav/v2/calendar_name/events/WEBEX-MEETING+CENTER-6.0680272-245422177.ics</D:href>\n  <D:propstat>\n   <D:status>HTTP/1.1 200 OK</D:status>\n   <D:prop>\n    <D:getcontenttype>text/calendar; component=vevent</D:getcontenttype>\n    <D:getetag>"63531356273"</D:getetag>\n    <D:resourcetype/>\n   </D:prop>\n  </D:propstat>\n </D:response>\n ... ... /D:multistatus>\n'
...
debug: Normalized URL from '/caldav/v2/calendar_name/events/m6icp53ietetp7l1gqe15540lk%40google.com.ics' to '/caldav/v2/calendar_name/events/m6icp53ietetp7l1gqe15540lk@google.com.ics'
debug: Normalized URL from '/caldav/v2/calendar_name/events/6pftiq1k4nm3dt9866thuo3br0%40google.com.ics' to '/caldav/v2/calendar_name/events/6pftiq1k4nm3dt9866thuo3br0@google.com.ics'
debug: Normalized URL from '/caldav/v2/calendar_name/events/WEBEX-MEETING+CENTER-6.0686402-108582152.ics' to '/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0686402-108582152.ics'
debug: Normalized URL from '/caldav/v2/calendar_name/events/248h42p0kesa2tvhsrj4c04n3k%40google.com.ics' to '/caldav/v2/calendar_name/events/248h42p0kesa2tvhsrj4c04n3k@google.com.ics'
debug: Normalized URL from '/caldav/v2/calendar_name/events/6pj3gcj36dgm4b9n6phm2b9k74s3ebb275im6bb46hj68chh64s66p9ic8%40google.com.ics' to '/caldav/v2/calendar_name/events/6pj3gcj36dgm4b9n6phm2b9k74s3ebb275im6bb46hj68chh64s66p9ic8@google.com.ics'
...
debug: Normalized URL from '/caldav/v2/calendar_name/events/0478200E00074C5B7101A82E008700E029A129CFACD0017700001706C698DFBD5A36141BDE96A1B1058C02B.ics' to '/caldav/v2/calendar_name/events/0478200E00074C5B7101A82E008700E029A129CFACD0017700001706C698DFBD5A36141BDE96A1B1058C02B.ics'
debug: Normalized URL from '/caldav/v2/calendar_name/events/tooubg46aa4rfimr0up83370h8%40google.com.ics' to '/caldav/v2/calendar_name/events/tooubg46aa4rfimr0up83370h8@google.com.ics'
debug: Normalized URL from '/caldav/v2/calendar_name/events/WEBEX-MEETING+CENTER-6.0680272-245422177.ics' to '/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0680272-245422177.ics'
debug: Normalized URL from '/caldav/v2/calendar_name/events/0478200E00074C5B7101A82E008700E07E35C50B92D4017700001708697308B07EEDA4F95AE7E466CF804AD.ics' to '/caldav/v2/calendar_name/events/0478200E00074C5B7101A82E008700E07E35C50B92D4017700001708697308B07EEDA4F95AE7E466CF804AD.ics'
debug: Normalized URL from '/caldav/v2/calendar_name/events/cdh3ae3464p3abb66hhj4b9k64p30b9pc9j3cb9h6so66d35c9i38p9kco%40google.com.ics' to '/caldav/v2/calendar_name/events/cdh3ae3464p3abb66hhj4b9k64p30b9pc9j3cb9h6so66d35c9i38p9kco@google.com.ics'
...
debug: Already normalized: '/caldav/v2/calendar_name/events/0478200E00074C5B7101A82E0087001013269558E0D201770000170B92F3FD560EB8F48B07E8F4C3FCA120E.ics'
debug: Already normalized: '/caldav/v2/calendar_name/events/apaqr9f9rpo4pppefe9h9cp1ks@google.com.ics'
debug: Unsafe character: '%2B'
debug: Already normalized: '/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0686402-108582152.ics'
debug: Already normalized: '/caldav/v2/calendar_name/events/ik7bg09ub8cjs2p1i3nkdk3114@google.com.ics'
...
debug: Already normalized: '/caldav/v2/calendar_name/events/9s46nk3pb4nho69g592b5fd65g@google.com.ics'
debug: Already normalized: '/caldav/v2/calendar_name/events/3ffviaqu42ev1e2lj1proijedc@google.com.ics'
debug: Unsafe character: '%2B'
debug: Already normalized: '/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0680272-245422177.ics'
debug: Already normalized: '/caldav/v2/calendar_name/events/5608r6i8nq19msdp1anr100s3v@google.com.ics'
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n        <C:calendar-multiget xmlns:D="DAV:"\n            xmlns:C="urn:ietf:params:xml:ns:caldav">\n            <D:prop>\n                <D:getetag/>\n                <C:calendar-data/>\n            </D:prop>\n            <D:href>/caldav/v2/calendar_name/events/or57be33sferu9juai0vn13u20@google.com.ics</D:href>... ...\n<D:href>/caldav/v2/calendar_name/events/9s46nk3pb4nho69g592b5fd65g@google.com.ics</D:href>\n<D:href>/caldav/v2/calendar_name/events/3ffviaqu42ev1e2lj1proijedc@google.com.ics</D:href>\n<D:href>/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0680272-245422177.ics</D:href>\n<D:href>/caldav/v2/calendar_name/events/5608r6i8nq19msdp1anr100s3v@google.com.ics</D:href>\n<D:href>/caldav/v2/calendar_name/events/0aln4lncn4oml6n60ubesepo24@google.com.ics</D:href>\n<D:href>/caldav/v2/calendar_name/events/b4i7bp6ialrv59cv1cga2mgqao@google.com.ics</D:href>\n<D:href>/caldav/v2/calendar_name/events/02nmc16u3nvrbcu7drajeut5a9@google.com.ics</D:href>\n<D:href>/caldav/v2/calendar_name/events/C38E9895-D6BA-4C30-9FFF-270FF248EA7A.ics</D:href> ... ...       </C:calendar-multiget>'
...
debug: Sending request...
debug: 207
debug: {'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Content-Type': 'text/xml; charset=UTF-8', 'Date': 'Mon, 01 Feb 2021 23:06:00 GMT', 'Expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'Pragma': 'no-cache', 'Vary': 'Origin, X-Origin, Referer', 'Content-Encoding': 'gzip', 'Server': 'ESF', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Alt-Svc': 'h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'}
...
debug: b'<?xml version="1.0" encoding="UTF-8"?>\n<D:multistatus xmlns:D="DAV:" xmlns:caldav="urn:ietf:params:xml:ns:caldav" xmlns:cs="http://calendarserver.org/ns/" xmlns:ical="http://apple.com/ns/ical/">\n ... ...<D:response xmlns:carddav="urn:ietf:params:xml:ns:carddav" xmlns:cm="http://cal.me.com/_namespace/" xmlns:md="urn:mobileme:davservices">\n  <D:href>/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0686402-108582152.ics</D:href>\n  <D:status>HTTP/1.1 404 Not Found</D:status>\n </D:response>\n <D:response xmlns:carddav="urn:ietf:params:xml:ns:carddav" xmlns:cm="http://cal.me.com/_namespace/" xmlns:md="urn:mobileme:davservices">\n  <D:href>/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0680272-245422177.ics</D:href>\n  <D:status>HTTP/1.1 404 Not Found</D:status>\n </D:response>\n</D:multistatus>\n'
...
debug: Already normalized: '/caldav/v2/calendar_name/events/uec002tktnj5if16b4lvb7tsa8@google.com.ics'
debug: Already normalized: '/caldav/v2/calendar_name/events/tlp35ob9j4rl3fi8l2o4vjbp0g@google.com.ics'
debug: Already normalized: '/caldav/v2/calendar_name/events/ts4p2oqivvt4i2abj7qhsanhuk@google.com.ics'
debug: Unsafe character: '%2B'
debug: Already normalized: '/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0686402-108582152.ics'
debug: Skipping '/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0686402-108582152.ics', properties are missing.
debug: Unsafe character: '%2B'
debug: Already normalized: '/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0680272-245422177.ics'
debug: Skipping '/caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0680272-245422177.ics', properties are missing.
error: Unknown error occurred for calendar_test/calendar_name: /caldav/v2/calendar_name/events/WEBEX-MEETING%2BCENTER-6.0686402-108582152.ics
error: Use `-vdebug` to see the full traceback.
debug:   File "/usr/local/lib/python3.7/site-packages/vdirsyncer/cli/tasks.py", line 67, in sync_collection
debug:     partial_sync=pair.partial_sync
debug:   File "/usr/local/lib/python3.7/site-packages/vdirsyncer/sync/__init__.py", line 135, in sync
debug:     b_nonempty = b_info.prepare_new_status()
debug:   File "/usr/local/lib/python3.7/site-packages/vdirsyncer/sync/__init__.py", line 58, in prepare_new_status
debug:     if prefetch else ()):
debug:   File "/usr/local/lib/python3.7/site-packages/vdirsyncer/storage/dav.py", line 503, in get_multi
debug:     raise exceptions.NotFoundError(href)
error: 1 out of 12 tasks failed.
~                                                                                                                                                                                                                

Reading the output, it seems the href cant be found. Is that because the href is using the encoded character? Any ideas on how to resolve the issue?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with storage/dav.py, especially get_multi, and trace how sync/init.py prepares status and handles the encoded href. Reproduce the failure using a Google Calendar event whose href contains '+', then inspect the URL normalization and 404 handling shown in the traceback. Done means syncing those events no longer fails with NotFoundError while preserving normal href handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.