bevyengine / bevyengine/bevy

Bevy needs a better asset-hosting solution for example assets

Open
#13,875 19 comments 0 reactions 4 assignees Claimed by @mockersf View on GitHub
A-Build-System C-Examples C-Feature D-Complex D-Domain-Agnostic S-Needs-Design X-Needs-SME
Dominant language
Rust
Stars
48.2k
Forks
4.8k
Avg merge
3d 16h
Merged PRs (30d)
171

Description

## What problem does this solve or what need does it fill?

As seen in #13671, adding or replacing new assets for Bevy's examples is shockingly controversial and high risk.

In the absence of other constraints, assets for examples should be:

- representative of real world use
- attractive, to showcase Bevy in a good light
- high quality, to avoid confusing testers and users about Bevy problems vs broken assets
- varied, to help catch unexpected issues

Without technical constraints, I would be very happy to have several hundred GB of test assets, updated on a whim whenever we have a new feature or find a better asset to replace something subpar.

However, because these assets are committed directly in Git, any assets (or changes to assets!) that we add are necessarily cloned with the repository. This wastes bandwidth and disk space for users, but also starts running into concerns with Github's hosting, which has a soft limit on repo size of [5 GB](https://www.reddit.com/r/github/comments/xn8y97/comment/ips9cxh/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button).

As an additional wrinkle, the way these assets are stored in the `assets` folder means that users commonly get confused when testing the examples locally, as the whole repo needs to be cloned to do this easily, rather than just copy-pasting code. See #13645.

## What options have you considered?

### Use Git LFS via Github

These are large files! We're using git! We should use [Git LFS](https://git-lfs.com/), right?

Eh, maybe. There are two broad problems here:

1. Git LFS has a reputation for instability and frustration. I haven't encountered these in my projects, but they're pervasive enough to worry me.
2. Git LFS hosting is [quite expensive](https://docs.github.com/en/billing/managing-billing-for-git-large-file-storage/upgrading-git-large-file-storage) on Github. By default there's [1 free GB of storage and bandwidth](https://docs.github.com/en/billing/managing-billing-for-git-large-file-storage/about-billing-for-git-large-file-storage#about-billing-for-git-large-file-storage) which is hilariously, unusably low for Bevy.

To use this freely, we'd probably want 10 data packs, putting us at 500 GiB of storage and 500 GiB/month of bandwidth.
At the current $5/month per data pack, that's $50/month. Not absurd, but not fun.

The amount of bandwidth required is *very* hard to estimate though: Bevy has a ton of users and contributors, and each of them will want to pull down the repo and examples.

If Github decides to offer us in-kind support for this, or some form of deal, I think this is worth considering. Otherwise, probably not.

### Use Git LFS on a different host

With a bit more devops work, we should be able to configure Git LFS to use an alternate backend. [Preliminary research](https://nickb.dev/blog/backblaze-b2-as-a-cheaper-alternative-to-githubs-git-lfs/) suggests that this should be meaningfully cheaper.

It also opens us up to accepting in-kind donations from alternate hosting companies (hi get in touch?).

There's open questions here about the level of engineering required, and the contributor / user experience though.

### No assets in the repo

Alternatively, we could avoid hosting assets in the repository at *all*, and instead simply download them on demand from Bevy-controlled servers. In the long term, this would likely share infrastructure with the Bevy Marketplace (or whatever we call our Unity Asset Store equivalent).

Appealingly, this means we have full control over cost / backing, and don't need to fuss with Git LFS at all. Users trying out example have things "just work" on copy-paste although there's weirdness with "please wait, downloading assets" and we don't waste nearly as much bandwidth copying over *all* of the assets for users that only want to try out a handful of examples.

This involves the most infrastructure work (but probably most stuff we want anyways), and we will need to take careful plan to ensure this doesn't cripple our ability to automatically test examples in CI.

## Additional context

This issue was prompted by @superdump raising these concerns [on Discord](https://discord.com/channels/691052431525675048/743663924229963868/1251887322287570995).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.