dotnet / dotnet/runtime

[API Proposal]: Add reproducible/deterministic mode to TarWriter for normalized metadata

Open
#132,049 4 comments 0 reactions 0 assignees View on GitHub
api-suggestion area-System.Formats.Tar
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Background and motivation

Work on dotnet/sdk#55689 exposed two separate sources of non-reproducible data when creating PAX archives with `TarWriter`:

1. **Process-dependent PAX header names.** `TarHeader.GenerateExtendedAttributeName()` writes names such as `/PaxHeaders./`, and global extended headers use `/GlobalHead..`. These names are serialized into the tar stream, so writing the same `PaxTarEntry` in two different processes produces different archive bytes.
2. **Host filesystem metadata captured by the path overload.** On Unix, `TarWriter.WriteEntry(string sourcePath, ...)` reads `mtime`, mode, uid, gid, uname, and gname from the source filesystem. The ownership fields vary between machines and builder accounts even when the file contents are identical.

These are related but distinct behaviors.

The SDK PR does **not** use `WriteEntry(string sourcePath, ...)` for container layers. `Layer.FromDirectory` manually constructs `PaxTarEntry` instances, sets their timestamps and modes, opens files only for their content, and calls `WriteEntry(TarEntry)`. Manually constructed POSIX entries already default to `uid=0`, `gid=0`, `uname=""`, and `gname=""`, so the builder account is not captured by that code path. The write-through stream in dotnet/sdk#55689 specifically normalizes the process-dependent `PaxHeaders.` name.

The runtime API should therefore cover both TarWriter-owned sources of nondeterminism:

- use process-independent names for PAX extended and global extended headers;
- provide deterministic defaults for metadata captured by `WriteEntry(string sourcePath, ...)`;
- allow callers to provide a stable modification timestamp, commonly derived from `SOURCE_DATE_EPOCH`;
- retain explicit caller-provided metadata when writing a manually constructed `TarEntry`.

Entry order, file content, links, and file modes remain intentional inputs controlled by the caller. `TarWriterOptions.HardLinkMode` already controls whether filesystem hard-link identity affects the archive.

In OCI image scenarios, each layer tar archive is hashed. Any process ID, host ownership value, or filesystem timestamp serialized into the archive changes the layer digest and prevents registries from deduplicating otherwise identical layers.

### API Proposal

```csharp
namespace System.Formats.Tar;

public sealed class TarWriterOptions
{
// Existing properties
public TarEntryFormat Format { get; set; }
public TarHardLinkMode HardLinkMode { get; set; }

///
/// Gets or sets whether TarWriter should avoid process- and host-dependent metadata.
///
/// When enabled:
/// - PAX extended and global extended header names do not contain the process ID or TMPDIR.
/// - Entries created by WriteEntry(string sourcePath, ...) use uid=0, gid=0,
/// uname="", and gname="", unless overridden below.
/// - Entries created by WriteEntry(string sourcePath, ...) use
/// OverrideModificationTime, or UnixEpoch when no override is provided.
///
/// Explicit metadata on entries passed to WriteEntry(TarEntry) is preserved. The
/// process-independent PAX header naming still applies to those entries.
///
public bool Deterministic { get; set; }

///
/// Gets or sets the modification timestamp used for entries created from filesystem paths.
/// In deterministic mode, null means DateTimeOffset.UnixEpoch.
/// Outside deterministic mode, null preserves the source filesystem timestamp.
///
public DateTimeOffset? OverrideModificationTime { get; set; }

///
/// Gets or sets the user ID used for entries created from filesystem paths.
/// In deterministic mode, null means 0.
/// Outside deterministic mode, null preserves the source filesystem value.
///
public int? OverrideUid { get; set; }

///
/// Gets or sets the group ID used for entries created from filesystem paths.
/// In deterministic mode, null means 0.
/// Outside deterministic mode, null preserves the source filesystem value.
///
public int? OverrideGid { get; set; }

///
/// Gets or sets the user name used for entries created from filesystem paths.
/// In deterministic mode, null means an empty string.
/// Outside deterministic mode, null preserves the source filesystem value.
///
public string? OverrideUName { get; set; }

///
/// Gets or sets the group name used for entries created from filesystem paths.
/// In deterministic mode, null means an empty string.
/// Outside deterministic mode, null preserves the source filesystem value.
///
public string? OverrideGName { get; set; }
}
```

The exact property names are open for discussion. The important part of the proposal is that deterministic mode covers the PAX header name generated internally by `TarWriter`, not only ownership metadata captured by the filesystem-path overload.

### API Usage

```csharp
// Example 1: Reproducible archive directly from filesystem paths
using var layerStream = new MemoryStream();
var options = new TarWriterOptions
{
Format = TarEntryFormat.Pax,
Deterministic = true,
OverrideModificationTime = DateTimeOffset.FromUnixTimeSeconds(sourceDateEpoch)
};

using (var writer = new TarWriter(layerStream, options, leaveOpen: true))
{
writer.WriteEntry("/app/config.json", "config.json");
writer.WriteEntry("/app/app.exe", "app.exe");
}

// The source file modes are preserved, while process ID, builder ownership and
// filesystem mtime no longer affect the archive bytes.
```

```csharp
// Example 2: SDK-style manually constructed entries
using var layerStream = new MemoryStream();
var options = new TarWriterOptions
{
Format = TarEntryFormat.Pax,
Deterministic = true
};

using (var writer = new TarWriter(layerStream, options, leaveOpen: true))
{
var entry = new PaxTarEntry(TarEntryType.RegularFile, "app.exe")
{
DataStream = File.OpenRead("/app/app.exe"),
ModificationTime = DateTimeOffset.FromUnixTimeSeconds(sourceDateEpoch),
Uid = configuredContainerUid
};

writer.WriteEntry(entry);
}

// Explicit entry metadata is retained, but the generated PAX extended-header name
// is stable and no post-processing stream is required.
```

```csharp
// Example 3: Custom ownership for path-based writes
var options = new TarWriterOptions
{
Format = TarEntryFormat.Pax,
Deterministic = true,
OverrideModificationTime = DateTimeOffset.FromUnixTimeSeconds(sourceDateEpoch),
OverrideUid = 0,
OverrideGid = 0,
OverrideUName = "root",
OverrideGName = "root"
};
```

### Alternative Designs

**Design 1: Boolean deterministic mode plus override properties (proposed)**

- One switch covers TarWriter-generated process-dependent names and deterministic defaults for path-based writes.
- Override properties support `SOURCE_DATE_EPOCH`, root ownership, and other reproducible policies.
- Explicitly constructed `TarEntry` metadata remains under caller control.

**Design 2: Enum-based metadata mode**

```csharp
public enum TarMetadataMode
{
Preserve,
Deterministic,
NormalizeToRoot,
Custom
}
```

A separate option would still be needed for the modification timestamp, and the deterministic mode would still need to define process-independent PAX header naming.

**Design 3: Separate options for each behavior**

```csharp
public bool UseDeterministicPaxHeaderNames { get; set; }
public bool NormalizeFileSystemOwnership { get; set; }
public DateTimeOffset? OverrideModificationTime { get; set; }
```

This is more explicit, but makes the common reproducible-archive scenario easier to configure incompletely.

**Design 4: SOURCE_DATE_EPOCH environment-variable support**

`TarWriter` could read `SOURCE_DATE_EPOCH` directly and enable deterministic behavior automatically. This follows the reproducible-builds convention but introduces implicit environment-dependent library behavior. Passing the parsed timestamp through `TarWriterOptions` is more explicit.

**Design 5: Continue constructing entries manually and post-process the stream**

This is possible today and is what dotnet/sdk#55689 does. It works, but requires every caller to understand TarWriter's internal PAX naming and filesystem metadata behavior.

### Risks

**Breaking changes:** None. Existing behavior remains the default. The proposed behavior is opt-in.

**Performance:** Deterministic path-based writes skip user and group database lookups. This should be neutral or faster.

**Platform behavior:**

- Windows path-based writes already default to uid=0, gid=0, empty uname/gname.
- Unix path-based writes change only when deterministic mode or an explicit override is selected.
- Process-independent PAX header naming applies consistently across platforms.

**PAX-specific concerns:**

- Both regular extended header names and global extended header names must be process-independent.
- Ownership values must be normalized in both standard header fields and PAX extended attributes when applicable.
- Explicit ownership on a manually constructed `TarEntry` must not be discarded.
- Tests should compare raw archive bytes across different process IDs and path-based source ownership values.

**Timestamp concerns:**

- `WriteEntry(string sourcePath, ...)` currently captures filesystem `mtime`, so ownership normalization alone cannot promise byte-identical output.
- A stable timestamp must be part of deterministic mode, either Unix epoch or an explicit value such as `SOURCE_DATE_EPOCH`.

**Container registry implications:**

- Enabling deterministic mode changes existing layer digests once.
- Subsequent builds from the same content and intentional metadata produce stable digests and can be deduplicated.

**Compatibility with dotnet/sdk#55689:**

- The SDK workaround was introduced for the process ID in generated PAX extended-header names.
- The SDK already manually constructs `PaxTarEntry` objects, so it does not capture the builder's uid, gid, uname, or gname through the path overload.
- Native process-independent PAX header naming would allow the SDK to remove `PaxHeaderNameNormalizingStream`.
- The path-based ownership and timestamp options remain valuable for `TarFile.CreateFromDirectory` and other callers that use filesystem-path APIs.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing TarWriterOptions, TarWriter.WriteEntry, and TarHeader.GenerateExtendedAttributeName(), including both path-based and TarEntry overloads. Compare raw archive bytes across process IDs, filesystem ownership, and timestamps, while checking that explicit TarEntry metadata remains unchanged. Done means deterministic PAX names and stable path-based metadata with caller-provided overrides.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.