NVIDIA / NVIDIA/cccl

Inflated Repository Size Due to Committed `.sccache` Files

Open
#1,194 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.5k
Forks
486
Avg merge
2d 6h
Merged PRs (30d)
295

Description

Thanks to investigation from @ajschmidt8, we've identified a significant increase in the size of our repository, primarily attributed to `.sccache` files that were inadvertently committed to the repo's history. A fresh clone of the `cccl` repository is currently about 367MB, which is unusually large.

#### Investigation and Findings

1. **Clone and Size Assessment**:
```
git clone git@github.com:NVIDIA/cccl.git
cd cccl
du -h --max-depth 0 .
```
2. **Identification of Large Files**:
- Used a command (inspired by a [Stack Overflow answer](https://stackoverflow.com/a/42544963)) to list the largest files in the repo's history.

```
git rev-list --objects --all |
git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
sed -n 's/^blob //p' |
sort --numeric-sort --key=2 |
cut -c 1-12,41- |
$(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest
...
2f0daf262060 2.9MiB .sccache/9/e/9e37d362c1007202477b18f7855e55e71b05a5b46c55dd2c3bfbcc355994b163
a3d002f8cef0 3.3MiB .sccache/2/e/2ee8f6281e79e45202aae7b7cb89976ad27a69a7e49d905af549238f2dca0d33
9d720f55849d 3.3MiB .sccache/7/a/7a2c59e04a8f03e5675101cddcc59125d26b162b4100add1a6fe0c33a9a2fe96
27998b24b787 3.3MiB .sccache/0/2/02eabccdff6b33019c37f63bb59b1569ba8cf031e9fa1a8b247c4ab8a67b8903
cdf0c45e6c6a 3.3MiB .sccache/d/6/d662a4523ace3db5f5a08691f9b602b41aa268d5bdd4a6449f252d05acfd8432
288355260fea 3.4MiB .sccache/9/a/9ad06af3a8009bed920039eda66c3d837b280bfd1ae1727bb5627c3b1c8e7dee
663ca17550e3 3.8MiB .sccache/a/8/a86b008835722a40b1c00f11476e9c4481c69d049f11235b5c4eb26ac217d103
fd84d3bca1a1 5.1MiB libcudacxx/libcxxabi/test/test_demangle.pass.cpp
0f745de29822 6.9MiB .sccache/d/2/d263ca87a8a1b6deca13a2e0cdfd364634f59c1b01480ff69c92cb6ccddae1ad
08ed86aa1a9f 8.8MiB .sccache/d/8/d8797629aff35aa8b0d17d5b55673967eb7bb35b420a1d32ba331fed69a296a0
```
- Additionally, isolated the size contribution of `.sccache` files:
```
git rev-list --objects --all -- .sccache | git cat-file --batch-check="%(objectsize) %(rest)" | cut -d" " -f1 | paste -s -d + - | bc | numfmt --to=si
```
- Finding: **`.sccache` files contribute approximately 224MB to the repo size.**

#### Options for Resolution

1. **Do Nothing**:
- Pros: Simplest option, no action required.
- Cons: Continues the burden of large clone size and potential for slower operations.

2. **Rewrite History to Remove `.sccache` Files**:
- Pros: Significantly reduce repo size, faster clone and operations.
- Cons: Complex process; requires rewriting history, which can disrupt all current branches and open pull requests.
- Implications:
- All contributors will need to re-clone the repository or carefully rebase their work.
- Potential loss of historical context for some changes.
- Requires meticulous coordination and communication with all contributors.

#### Suggested Course of Action

Given the substantial size contribution of the `.sccache` files, it seems prudent to consider rewriting the repository's history to remove these files, despite the complexities involved. This action would provide long-term benefits in terms of repository manageability and performance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.