astral-sh / astral-sh/tar-codec

Expose policy-validated effective TAR metadata before projection

Open
#120 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9
Forks
3
Avg merge
9m
Merged PRs (30d)
3

Description

## Use case

RustFS Snowball accepts TAR request bodies from untrusted clients. `TarArchive` is the right policy boundary for this input because it validates PAX limits, duplicate records, global member metadata, allowed vendor namespaces, and non-UTF-8 vendor values before yielding a member.

Snowball also needs effective member metadata that is TAR-specific:

- the effective modification time; and
- precedence-resolved `minio.*` PAX values used for object metadata and version IDs.

## Current gap

`TarArchive::next_member` applies `DecodePolicy::check_member` and then projects the `MemberFrame` into a format-neutral `archive_trait::Member`. That projection retains path, position, size, executable state, and payload, but it does not expose the effective PAX state or modification time.

Using `tar_framing::logical::TarReader` directly exposes `PaxState::effective_record`, including local-over-global precedence and deletion tombstones, but it bypasses the policy checks owned by `tar-codec`. A downstream consumer would therefore have to duplicate those checks or inspect raw records without an authoritative precedence decision. Neither is a good security boundary.

This is adjacent to, but distinct from, #118: that issue concerns byte-preserving paths and link targets, while this use case needs policy-validated effective metadata.

## Requested direction

Would a lending, TAR-specific member view or cursor in the `tar-codec` decode layer fit the intended architecture? The exact API shape is up to the maintainers, but the useful properties would be:

- it runs the same `DecodePolicy` checks and fusing behavior as `TarArchive`;
- it exposes effective standard metadata such as modification time;
- it exposes precedence-resolved values, including deletion, for vendor namespaces accepted by the configured policy;
- it does not require consumers to reimplement PAX precedence or inspect unvalidated raw records; and
- it preserves the existing borrowed payload model and leaves the format-neutral `archive_trait::Member` projection unchanged.

RustFS's candidate Snowball policy rejects global PAX extensions, duplicate PAX records, non-UTF-8 vendor values, and every vendor namespace except `minio`. The API itself should remain correct for policies that do allow global PAX state.

## Non-goals

- This is not a request to relax validation of the two-block TAR end marker. RustFS will retain strict `MissingEndMarker` handling and classify the known footerless minio-go shape only at its own authenticated, complete-request, exact-member-boundary gate.
- This is not a request for legacy GNU sparse or other vendor-specific interpretation in `tar-codec`.

## Reproducer and compatibility evidence

RustFS PR [rustfs/rustfs#7157](https://github.com/rustfs/rustfs/pull/7157) adds deterministic raw and S2-compressed request-body fixtures generated with minio-go v7.3.0. Its compatibility tests demonstrate both the policy-valid decode and the metadata lost at the current projection boundary.

Contributor guide

Open the contributing guide

Research direction

Start by reading TarArchive::next_member, DecodePolicy::check_member, MemberFrame, and the TarReader/PaxState::effective_record path described in the issue. Trace where policy validation and fusing occur before the archive_trait::Member projection. Done means a TAR-specific view exposes policy-validated effective modification time and vendor metadata, including precedence and deletion, while preserving borrowed payloads and the existing projection.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.