astral-sh / astral-sh/tar-codec
Expose policy-validated effective TAR metadata before projection
- Dominant language
- Rust
- Stars
- 9
- Forks
- 3
- Avg merge
- 9m
- Merged PRs (30d)
- 3
Description
## Use case
RustFS Snowball accepts TAR request bodies from untrusted clients. `TarArchive` is the right policy boundary for this input because it validates PAX limits, duplicate records, global member metadata, allowed vendor namespaces, and non-UTF-8 vendor values before yielding a member.
Snowball also needs effective member metadata that is TAR-specific:
- the effective modification time; and
- precedence-resolved `minio.*` PAX values used for object metadata and version IDs.
## Current gap
`TarArchive::next_member` applies `DecodePolicy::check_member` and then projects the `MemberFrame` into a format-neutral `archive_trait::Member`. That projection retains path, position, size, executable state, and payload, but it does not expose the effective PAX state or modification time.
Using `tar_framing::logical::TarReader` directly exposes `PaxState::effective_record`, including local-over-global precedence and deletion tombstones, but it bypasses the policy checks owned by `tar-codec`. A downstream consumer would therefore have to duplicate those checks or inspect raw records without an authoritative precedence decision. Neither is a good security boundary.
This is adjacent to, but distinct from, #118: that issue concerns byte-preserving paths and link targets, while this use case needs policy-validated effective metadata.
## Requested direction
Would a lending, TAR-specific member view or cursor in the `tar-codec` decode layer fit the intended architecture? The exact API shape is up to the maintainers, but the useful properties would be:
- it runs the same `DecodePolicy` checks and fusing behavior as `TarArchive`;
- it exposes effective standard metadata such as modification time;
- it exposes precedence-resolved values, including deletion, for vendor namespaces accepted by the configured policy;
- it does not require consumers to reimplement PAX precedence or inspect unvalidated raw records; and
- it preserves the existing borrowed payload model and leaves the format-neutral `archive_trait::Member` projection unchanged.
RustFS's candidate Snowball policy rejects global PAX extensions, duplicate PAX records, non-UTF-8 vendor values, and every vendor namespace except `minio`. The API itself should remain correct for policies that do allow global PAX state.
## Non-goals
- This is not a request to relax validation of the two-block TAR end marker. RustFS will retain strict `MissingEndMarker` handling and classify the known footerless minio-go shape only at its own authenticated, complete-request, exact-member-boundary gate.
- This is not a request for legacy GNU sparse or other vendor-specific interpretation in `tar-codec`.
## Reproducer and compatibility evidence
RustFS PR [rustfs/rustfs#7157](https://github.com/rustfs/rustfs/pull/7157) adds deterministic raw and S2-compressed request-body fixtures generated with minio-go v7.3.0. Its compatibility tests demonstrate both the policy-valid decode and the metadata lost at the current projection boundary.
Contributor guide
Research direction
Start by reading TarArchive::next_member, DecodePolicy::check_member, MemberFrame, and the TarReader/PaxState::effective_record path described in the issue. Trace where policy validation and fusing occur before the archive_trait::Member projection. Done means a TAR-specific view exposes policy-validated effective modification time and vendor metadata, including precedence and deletion, while preserving borrowed payloads and the existing projection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100