Remove large binaries from history
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 110
- PR merge metrics
- No merged PRs in 30d
Description
## The Issue
A bunch of large binaries from previous versions of the app are in the packfile. Most of these are from pre-native rewrite builds -- I assume that their value in the git history is pretty low, and they slow down your CI feedback loop considerably because a fresh clone takes quite a while due to file size (The .git folder is half a gig!)
### Steps to reproduce
Run this:
```
git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| sed -n 's/^blob //p' \
| sort --numeric-sort --key=2 \
| cut -c 1-12,41- \
| $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest
```
See screenshot below of large files
If you're on MacOS and do not have numfmt, install coreutils via brew
```
brew install coreutils
```
### Expected behaviour
Large binaries not in source control
### Actual behaviour
Large binaries in source control
## System Configuration
N/A
## Anything Else
Here is a step-by-step on how to clean up the history:
1. Do a fresh clone
```
git clone https://github.com/lbryio/lbry-android.git
cd lbry-android
```
2. Run this script from the project root to track all branches
```
#!/bin/bash
for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master`; do
git branch --track ${branch##*/} $branch
done
```
3. filter-branch on the unnecessary binaries from older versions
```
# Old SDK dependencies
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/libs/*.aar' --prune-empty -f -- --all
# index.android.bundle (two locations)
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch src/main/assets/index.android.bundle' --prune-empty -f -- --all
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/src/main/assets/index.android.bundle' --prune-empty -f -- --all
# Node package lock
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/package-lock.json' --prune-empty -f -- --all
# Old blockchain headers
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch src/main/assets/blockchain/headers' --prune-empty -f -- --all
```
4. Clean up the packfile
```
rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --prune=now
git gc --aggressive --prune=now
```
5. Force push all branches and all tags (need to turn branch protection off temporarily to do this)
```
git push origin --force --all
git push origin --force --tags
```
This takes the packfile size down from **537 MB to 11 MB**
**All other contributors will need to do a fresh clone of the repo after you force push. If they push from their old local copy, they will re-add all the files that were deleted from the history.**
## Screenshots

## Internal Use
### Acceptance Criteria
1.
2.
3.
### Definition of Done
- [ ] Tested against acceptance criteria
- [ ] Tested against the assumptions of the user story
- [ ] The project builds without errors
- [ ] Unit tests are written and passing
- [ ] Tests on devices/browsers listed in the issue have passed
- [ ] QA performed & issues resolved
- [ ] Refactoring completed
- [ ] Any configuration or build changes documented
- [ ] Documentation updated
- [ ] Peer Code Review performed
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with a fresh clone of lbry-android and review the listed app/libs/*.aar, asset bundles, app/package-lock.json, and blockchain/headers paths alongside the git filter-branch commands. Done means the unnecessary binaries are removed from all branches and tags, the repository packfile is reduced from about 537 MB toward 11 MB, and contributors are warned that fresh clones are required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, git
- Domain
- devops, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100