pimutils / pimutils/vdirsyncer
Avoid saving the entire item in memory for Items returned by filesystem storage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 185
- Avg merge
- 11h 49m
- Merged PRs (30d)
- 1
Description
During sync, all items that need to be copied from A to B are fetched from A at once. This alone makes RAM consumption of synchronization O(n) for initial sync, though it already is for other reasons (e.g. #546 not being implemented). I haven't found a fix for that, and I actually believe that it can be proven that the current sync semantics can't be done more efficiently wrt RAM consumption than in O(n).
However, we can at least reduce the constant factor by a lot. Currently Item is basically a wrapper around a string that contains the entire item content. This is not very memory-efficient, and pretty unnecessary for item content that is fetched from the local disk.
What I want to have is a subclass of Item that wraps a file on the local FS:
- It doesn't store the full item content, only the filepath and the etag, which are passed via the constructor.
- When accessing
raw, it opens the file, validates the etag against the one it has stored (!!) reads the entire content, closes and returns the content. uidis cached, as areidentandhash, but that part can just be inherited from the superclass
etag (=mtime) validation is important because I'm concerned about other programs modifying the file while synchronization is happening. It isn't foolproof but lowers the chance by a lot.
That new superclass is then used by filesystem storage's get method. However, that behavior should possibly be configurable to accomodate for situations where the old behavior is better (the underlying FS is slow, or badly cached). That means a new parameter for the filesystem storage.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the filesystem storage get method and the Item implementation, then trace how synchronization consumes fetched items. Done means filesystem-backed items retain only the path and etag until raw is accessed, validate the etag before reading, preserve the required caching behavior, and expose a configurable choice between lazy and eager loading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100