armbian / armbian/build

[Task / Epic]: HUGE repository cleanup ✨

Open
#6,820 7 comments 2 reactions 3 assignees Claimed by @ColorfulRhino View on GitHub
08 Discussion Help needed Task/To-Do
Dominant language
Shell
Stars
5.4k
Forks
3.1k
Avg merge
1d 9h
Merged PRs (30d)
204

Description

### Task description

### The Problem

Over the years, some older stuff was partially removed/not used anymore, but not fully cleaned up. Those files and leftovers in the code still live in the repository, leading to confusions (_what is this? is this still used? can this be deleted?_) and misleading search/`grep` results (e.g. for [packages/extras-buildpkgs/htop](https://github.com/armbian/build/tree/main/packages/extras-buildpkgs/htop/debian) or [packages/extras-buildpkgs/hostapd](https://github.com/armbian/build/tree/main/packages/extras-buildpkgs/hostapd/debian) which included changelogs and therefore lots of unrelated text).

Besides that, many blobs were added to the build repo, only some of them still remain. But even the deleted blobs still remain in the repo since Git saves all the history: The history size is huge, even though the current size of the `packages/blobs` folder is _only_ 55MB. This leads to a unnecesarily bloated repository, increasing the size for everybody.
I remember one person often having to visit a local library or university to download/update their Armbian repo since the size was too large on their slow or resticted internet connection at home.

**For comparison:**
- [Size of the whole U-Boot repository](https://api.github.com/repos/u-boot/u-boot) (since 2005!): ~ 228 MB
- [Size of the current Armbian/build repository](https://api.github.com/repos/Armbian/build): ~ 636 MB

I don't believe that Armbian/build is a bigger project than U-Boot, but it is amlost triple the size in MB. Vastly reducing the repo size (TODO: calculate actual size before/after blob purge) will make contributions more inclusive overall and save time on many occasions.

### The Solutions

Removing all known leftover code and moving all blobs to a separate blob repository, like already done with the Rockchip blobs in the [Armbian/rkbin](https://github.com/armbian/rkbin) repo. After this is done, purge the build repository's history from all the blobs (original idea by @rpardini I believe). The goal is to have a completely blobless Arbian/build repo while blobs are only pulled from other repositories.

### Task List

**Leftover code:**
- [x] Remove unused `packages/extras-buildpkgs/hostapd` plus its Realtek part and its traces https://github.com/armbian/build/pull/6830
- [x] Remove unused `packages/extras-buildpkgs/htop` and its traces --> [same PR](https://github.com/armbian/build/pull/6830)
- [x] Remove unused `packages/extras-buildpkgs/sunxi-tools` and its traces --> [same PR](https://github.com/armbian/build/pull/6830)
- [x] Remove old compiler torrents from `config/torrents/` https://github.com/armbian/build/pull/6829
- [ ] Remove 'external-toolchains' code path
- [x] https://github.com/armbian/build/issues/6821
- [ ] Check for other leftover stuff in the code (**HELP NEEDED: Please let me know or add to this task list if you know of any unused stuff 😄**)

**Blobs:**
- [ ] Create new repository for blobs (e.g. Armbian/blobs)
- [ ] Copy all existing blobs to the new blob repo
- [ ] Change all references in the code to point to the new repo for the blobs
- [ ] Remove all existing blobs from the Armbian/build repo
- [ ] Purge all blobs from the Armbian/build Git history by rewriting history (this is where the actual size reduction happens)
- [ ] Add an Actions workflow to check new PRs for blobs and if detected, kindly auto-remind them to commit the blobs to the blob repo instead of the build repo

_This task list will be extended with new findings. PRs solving specific tasks will be linked._

This task/story is open for ideas and discussions! 😄

---

### Some statistics for fun and to compare the impact of this cleanup:

| | Before | After | Difference |
|---------------|-----------|-------|------------|
| Lines of code | 6 458 280 | TBD | |
| # of files | 6503 | TBD | |
| Repo size | ~ 636 MB | TBD | |

Commands used:
- Lines of code: `git diff --shortstat 4b825dc642cb6eb9a060e54bf8d69288fbee4904`
- \# of files: `git ls-files | wc -l`
- Repo size: [Queried on GitHub](https://api.github.com/repos/Armbian/build)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.