lbryio / lbryio/lbry-android

Remove large binaries from history

Open
#977 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
2.4k
Forks
110
PR merge metrics
No merged PRs in 30d

Description

## The Issue

A bunch of large binaries from previous versions of the app are in the packfile. Most of these are from pre-native rewrite builds -- I assume that their value in the git history is pretty low, and they slow down your CI feedback loop considerably because a fresh clone takes quite a while due to file size (The .git folder is half a gig!)

### Steps to reproduce

Run this:

```
git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| sed -n 's/^blob //p' \
| sort --numeric-sort --key=2 \
| cut -c 1-12,41- \
| $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest
```

See screenshot below of large files

If you're on MacOS and do not have numfmt, install coreutils via brew

```
brew install coreutils
```

### Expected behaviour
Large binaries not in source control

### Actual behaviour
Large binaries in source control

## System Configuration
N/A

## Anything Else

Here is a step-by-step on how to clean up the history:

1. Do a fresh clone
```
git clone https://github.com/lbryio/lbry-android.git
cd lbry-android

```

2. Run this script from the project root to track all branches

```
#!/bin/bash
for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master`; do
git branch --track ${branch##*/} $branch
done
```

3. filter-branch on the unnecessary binaries from older versions

```
# Old SDK dependencies
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/libs/*.aar' --prune-empty -f -- --all

# index.android.bundle (two locations)
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch src/main/assets/index.android.bundle' --prune-empty -f -- --all

git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/src/main/assets/index.android.bundle' --prune-empty -f -- --all

# Node package lock
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch app/package-lock.json' --prune-empty -f -- --all

# Old blockchain headers
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch src/main/assets/blockchain/headers' --prune-empty -f -- --all
```

4. Clean up the packfile

```
rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --prune=now
git gc --aggressive --prune=now
```

5. Force push all branches and all tags (need to turn branch protection off temporarily to do this)

```
git push origin --force --all
git push origin --force --tags
```

This takes the packfile size down from **537 MB to 11 MB**

**All other contributors will need to do a fresh clone of the repo after you force push. If they push from their old local copy, they will re-add all the files that were deleted from the history.**

## Screenshots

Screen Shot 2020-08-16 at 5 23 49 PM

## Internal Use

### Acceptance Criteria
1.
2.
3.

### Definition of Done
- [ ] Tested against acceptance criteria
- [ ] Tested against the assumptions of the user story
- [ ] The project builds without errors
- [ ] Unit tests are written and passing
- [ ] Tests on devices/browsers listed in the issue have passed
- [ ] QA performed & issues resolved
- [ ] Refactoring completed
- [ ] Any configuration or build changes documented
- [ ] Documentation updated
- [ ] Peer Code Review performed

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with a fresh clone of lbry-android and review the listed app/libs/*.aar, asset bundles, app/package-lock.json, and blockchain/headers paths alongside the git filter-branch commands. Done means the unnecessary binaries are removed from all branches and tags, the repository packfile is reduced from about 537 MB toward 11 MB, and contributors are warned that fresh clones are required.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, git
Domain
devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.