Investigate possible export performance and memory usage improvements.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 598
- Forks
- 168
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 86
Description
I'd even be fine with merging this as is and promise to start on further refactoring right away with both ideas:
- Using a db iterator
- using qs.none()
Another thing (to think about later) that came to my mind: In the case of incremental, we now still include artifacts that are new to a repository version, but guaranteed to be in another (exported) repositories previous version. Could we safely skip them too?
Say the logic switching from:
$\displaystyle \bigcup_{\textrm{version}} \left( \textrm{version}.\textrm{artifacts} \setminus \textrm{prev}(\textrm{version}).\textrm{artifacts} \right)$
to
$\left( \displaystyle \bigcup_{\textrm{version}} \textrm{version}.\textrm{artifacts} \right) \setminus \left( \displaystyle \bigcup_{\textrm{version}} \textrm{prev}(\textrm{version}).\textrm{artifacts} \right)$
Originally posted by @mdellweg in https://github.com/pulp/pulpcore/pull/4161#pullrequestreview-1551702739
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the export implementation discussed in pull request #4161. Investigate the db iterator and qs.none() ideas, then assess whether incremental exports can safely exclude artifacts already present in earlier exported versions. Done means a clearly scoped performance change with export results preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100