Unity-Technologies / Unity-Technologies/UnityDataTools

Support repacking Archives (Lz4 compression configuration)

Open
#64 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
821
Forks
71
Avg merge
3h 13m
Merged PRs (30d)
9

Description

With #62 the archive command is exposing the blocks and directory section of the archive metadata.
This task is to add an option for "repacking" an existing Archive file (e.g. an AssetBundle) into a freshly compressed Archive that has the same content.

Unity already has AssetBundle.RecompressAssetBundleAsync which can handle some transformation, for example compressed to uncompressed.

However currently there is no way to influence how files are split into individual blocks for the block based compression (LZ4 and LZ4HC). Unity always uses 128KB block size and will span over file boundaries.

This task is to add a command for repacking an archive. It could take the following configuration parameters:

  • more variability for the block sizes (max block size uncompressed (input) data, max block size compressed data)
  • how file boundaries are handled (span or always split)
  • padding between files
  • minimum compression factor (at what ratio to fallback to uncompressed) - avoiding on a block by block based an extra layer of compression when the data is already compressed.

This would allow some optimization, including picking configurations specifically fine tuned to the target platform's characteristics.

In my understanding so long as the Block and Directory data structures are properly populated then the Unity Runtime will be able to load the archive data properly, e.g that it already could support reading a wider range of blocking.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the archive command and the block and directory metadata exposed by #62. Define how an existing Archive is read and repacked while preserving its content, then specify configuration for block sizes, file boundaries, padding, and compression fallback; done means Unity can load the resulting archive.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, unity
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.