bazelbuild / bazelbuild/rules_apple

Lazy hash computing in bundletool.py

Open
#1,641 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
593
Forks
334
Avg merge
16h 48m
Merged PRs (30d)
9

Description

Currently, if `tree artifacts` is not enabled, `bundletool.py` will be used to bundle an iOS app. In `bundletool.py`, before writing each entry into the zip file [the md5 hash of the data will be computed](https://github.com/bazelbuild/rules_apple/blob/master/tools/bundletool/bundletool.py#L196) to check if different files will be written into the same `dest`. If we have large files in the bundle(for us, we have about 2GB of resource files), the hash computing will take some time to finish. However, in most cases, there won't be conflicting files, so time is wasted.

A simple fix is to lazy compute the md5 hash of each entry. Instead of `self._entry_hashes` we can have `self._entry_datas`. Then before we write an entry into the zip file, we check if there are already data for the same `dest`. If it's true, then we calculate the hash value of the two entries to check if they are the same.

We did a simple test with this fix. Before the fix, it cost around 40 seconds to finish the bundling. After it cost around 22 seconds.

I can think about two downsides of this method:

1. memory consumption of bundling will be increased.
2. for cases where the same file will be written multiple times into the zip file, lazy hash computing may actually take more time.

If the lazy hash computing is acceptable, I'd like to create a PR. If there is a better solution, I'd be happy to try it too.

Contributor guide

Open the contributing guide

Research direction

Start in tools/bundletool/bundletool.py around the entry-writing code near line 196 and trace how _entry_hashes detects conflicting destinations. Compare the lazy-data approach against large bundles and repeated destinations; done means preserving conflict behavior while avoiding unnecessary hashing, with the stated memory and performance trade-offs evaluated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, mobile-dev
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.