aboutcode-org / aboutcode-org/federatedcode

Investigate git partial clone/sparse checkout

Đang mở
#82 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
6
Fork
2
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

As we are scaling up the deployment of many git repos, a partial clone (which is not a shallow clone) may help federatedcode clients to fetch just what they need.

The expected effects are ensuring each federatedcode client can clone, fetch and checkout exactly what they need out of the "fleet" of 1000's git repos and tens of millions of package PURLs, metadata, scans, SBOMs and more.

This is just a bunch of pointer to review and support the design of efficient partial sync of a decentralized PurlDB/vcio deployment through federatedcode and focusing only on a slice as the "packages of interest".

The git partial clone feature and sparse checkout features have been there for a while. See these doc:
- https://git-scm.com/docs/partial-clone
- https://git-scm.com/docs/git-sparse-checkout

The two are closely related to achieve expected effects, as explained most clearly by @mslinn :
- https://mslinn.com/git/600-partial-clone.html

> Sparse checkouts allow you to restrict the files and directories that Git can retrieve from the remote repository. When sparse checkout is used with partial cloning, the two features work together so that not only is the size of the working tree reduced, but the Git object database also reduced in size, so that only the requested objects are fetched from the remote repository, on demand.

See also:

- https://about.gitlab.com/blog/partial-clone-for-massive-repositories/
- https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/
- https://github.blog/open-source/git/bring-your-monorepo-down-to-size-with-sparse-checkout/
- https://www.canva.dev/blog/engineering/we-put-half-a-million-files-in-one-git-repository-heres-what-we-learned/
- https://stolee.dev/docs/git-merge-2022.pdf on git internals

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.