bazelbuild / bazelbuild/rules_apple
Lazy hash computing in bundletool.py
- Dominant language
- Starlark
- Stars
- 593
- Forks
- 334
- Avg merge
- 16h 48m
- Merged PRs (30d)
- 9
Description
Currently, if `tree artifacts` is not enabled, `bundletool.py` will be used to bundle an iOS app. In `bundletool.py`, before writing each entry into the zip file [the md5 hash of the data will be computed](https://github.com/bazelbuild/rules_apple/blob/master/tools/bundletool/bundletool.py#L196) to check if different files will be written into the same `dest`. If we have large files in the bundle(for us, we have about 2GB of resource files), the hash computing will take some time to finish. However, in most cases, there won't be conflicting files, so time is wasted.
A simple fix is to lazy compute the md5 hash of each entry. Instead of `self._entry_hashes` we can have `self._entry_datas`. Then before we write an entry into the zip file, we check if there are already data for the same `dest`. If it's true, then we calculate the hash value of the two entries to check if they are the same.
We did a simple test with this fix. Before the fix, it cost around 40 seconds to finish the bundling. After it cost around 22 seconds.
I can think about two downsides of this method:
1. memory consumption of bundling will be increased.
2. for cases where the same file will be written multiple times into the zip file, lazy hash computing may actually take more time.
If the lazy hash computing is acceptable, I'd like to create a PR. If there is a better solution, I'd be happy to try it too.
Contributor guide
Research direction
Start in tools/bundletool/bundletool.py around the entry-writing code near line 196 and trace how _entry_hashes detects conflicting destinations. Compare the lazy-data approach against large bundles and repeated destinations; done means preserving conflict behavior while avoiding unnecessary hashing, with the stated memory and performance trade-offs evaluated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, mobile-dev
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100