collective / collective/collective.exportimport

Support exporting a lot of content

Open
#17 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19
Forks
20
PR merge metrics
No merged PRs in 30d

Description

Same as with the import (solved in https://github.com/collective/collective.exportimport/pull/4), the export of a lot of data would eat up all the memory on the machine. The python dict that is created holds the data during export before being written to file as json can be quite large if you choose to to include base64-encoded binary data.

I have the use-case to export 60GB of content in files.

Options:

* Export the blob-path and load each blob from the filesystem. That could be quite efficient.
* Use https://pypi.org/project/jsonlines as format and write one object at a time to the filesystem. This would rquire changes in the import since jsoinlines is not readable by json or ijson.
* Fake using jsonlines by writing one object at a time into a file but add a comma at the end of each line and wrap it in []. This would create valid json file the the import could read.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the import solution in the referenced pull request and tracing the current export path that builds the Python dictionary. Compare the proposed blob-path, JSON Lines, and streaming-valid-JSON approaches against the 60GB use case and the existing importer. Done means large exports no longer require holding all content in memory and remain importable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.