Efficient incremental bundle updates

Nice-to-have use-case: Serving a bundle of many pages for offline use. Think, e.g., downloading a large slice of Wikipedia.

Associated requirement: Updating bundles without needing to re-write the entire bundle, so that (e.g.) a single 100 kB response changing, being added or deleted doesn't result in needing to also re-write the 10 GB of 100,000 unchanged responses.

Non-goal: Updating only part of a single large response incrementally.

The alternative is to accept that updating a bundle can't be done incrementally; presumably the work-around would be to instead serve many bundles that are small enough that re-writing them isn't painful, and instead suffer the pain of dealing with many bundles instead of just one. Even in that case, I'd like to see the use case & requirement written down as something that was explicitly declined as part of a trade-off, rather than just not considered.

This is almost supported by the spec as is, but blocked by two things: first, how the index can't be placed last in the stream (because the responses section must be the last); and, second, CBOR does not allow padding around responses (though the spec, as written, doesn't break if padding is added - the section is never parsed as CBOR and finding responses is all done with offsets).

If I squint and imagine that these are non-issues, to create an incrementally-updatable bundle, place the index section after the responses section, and make sure that the section lengths are written as bignums with plenty of leading bits; to update the bundle append the new responses to the end of the responses section and then append the new index section and re-write lengths/offsets as needed.

There's a security consideration here, though: the above procedue will leave old responses in the bundle, which could be an information leak. Zeroing redundant responses out is a partial solution, but still leaks response length; aligning responses to, e.g., 1 KiB boundaries would reduce the length leak. The need for padding would also mean that a bundle wouldn't be a valid CBOR document.

The updater could keep track of zeroed-out ranges and opportunistically place new responses there, but a dumber append-only strategy should work pretty well in most situations, trading implementation simplicity & reduced state for (uncompressed) bundle size.

Implementations might also get clever with opportunistic use of sparse files, so there should be at least a SHOULD on using zero bytes specifically to enable that.

Start by reading the bundle specification's index and responses sections, then trace how section lengths, offsets, and CBOR validity are defined. Compare the proposed incremental-update model with the security concerns around retained responses and padding. Done means the specification records a resolved design and its implementation requirements.