collective / collective/collective.exportimport
Support exporting a lot of content
- Dominant language
- Python
- Stars
- 19
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
Same as with the import (solved in https://github.com/collective/collective.exportimport/pull/4), the export of a lot of data would eat up all the memory on the machine. The python dict that is created holds the data during export before being written to file as json can be quite large if you choose to to include base64-encoded binary data.
I have the use-case to export 60GB of content in files.
Options:
* Export the blob-path and load each blob from the filesystem. That could be quite efficient.
* Use https://pypi.org/project/jsonlines as format and write one object at a time to the filesystem. This would rquire changes in the import since jsoinlines is not readable by json or ijson.
* Fake using jsonlines by writing one object at a time into a file but add a comma at the end of each line and wrap it in []. This would create valid json file the the import could read.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the import solution in the referenced pull request and tracing the current export path that builds the Python dictionary. Compare the proposed blob-path, JSON Lines, and streaming-valid-JSON approaches against the 60GB use case and the existing importer. Done means large exports no longer require holding all content in memory and remain importable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100