hashgraph / hashgraph/solo-weaver

feat(manifests): add archives: support to external-files.yaml parser

Open
#1,084 0 comments 0 reactions 0 assignees View on GitHub
New Feature Story
Dominant language
Go
Stars
3
Forks
0
Avg merge
3d 5h
Merged PRs (30d)
47

Description

**Epic:** #502 — Network Upgrade Workflow (Execute Phase)
**Parent:** #408

## Context

The HIP external-files spec now includes an `archives:` top-level key (sibling to `files:`) for verifiable archive download and extraction. The motivating case is TSS WRAPS proving keys (~2 GB `wraps-v1.0.0.tar.gz` containing 4 members) that must be staged atomically before the node starts.

This story adds the parser and validation support in `pkg/manifests`. Download and install execution are handled by #537 and #538 respectively (updated to include archives).

## Scope

### New types

- `Archive` struct: `url`, `format`, `algorithm`, `checksum`, `contentType`, `destination`, `optional`, `phase`, `contents`
- `ArchiveMember` struct: `path`, `algorithm`, `checksum`, `size` (optional), `destination` (optional override)
- `ArchiveFormat` enum: `tar.gz`, `tar`, `zip` — explicit, never inferred from URL

### Add `Archives` field to `ExternalFiles`

```go
type ExternalFiles struct {
Header `yaml:",inline"`
Files []ExternalFile `yaml:"files,omitempty"`
Archives []Archive `yaml:"archives,omitempty"`
}
```

### Parse-time validation rules

- `url`, `format`, `algorithm`, `checksum`, `destination`, `phase.download`, `phase.install` are required and non-empty
- `format` must be one of `tar.gz | tar | zip`
- `url` must use a recognised scheme (`https://`, `s3://`, `gcs://`)
- `destination` must begin with a recognised marker prefix followed by `/` (same closed set as `files[]`)
- `contents` is required and must declare at least one entry
- `contents[].path` must be relative: no leading `/`, no `.` or `..` segment, no drive letter, no backslash separator. Paths must be unique within a single `archives[]` entry
- `contents[].algorithm` and `contents[].checksum` are required and non-empty
- **Global destination uniqueness**: every resolved install path — across all `archives[]` members and all `files[]` entries — must be unique

### URL scheme validation for `files[]`

While here, add URL scheme validation to `files[]` entries as well. The HIP specifies `https://`, `s3://`, `gcs://` as the allowed set. Currently `files[]` only checks non-empty. Add an `allowedURLSchemes` closed set and accessor (following the `allowedBucketSchemes` / `allowedDestinationPrefixes` pattern).

### Tests

- Happy-path archive with all fields populated
- Archive with optional per-member `destination` override
- Validation failures for each rule above
- Global destination uniqueness across files + archive members
- Path safety: reject `..`, leading `/`, backslash
- Lenient: unknown fields in archives silently ignored (HIP-1494)

## Acceptance Criteria

- [ ] `ParseExternalFiles` accepts manifests with `archives:` entries
- [ ] All parse-time validation rules enforced with `ValidationError`
- [ ] URL scheme validation added to both `files[]` and `archives[]`
- [ ] Global destination uniqueness across files and archive members
- [ ] Path safety enforced on `contents[].path`
- [ ] Tests cover happy path, all validation failures, and lenient parsing

Contributor guide

Open the contributing guide

Research direction

Start at ParseExternalFiles in pkg/manifests and read the existing allowedBucketSchemes and allowedDestinationPrefixes patterns. Run the existing manifest parser tests, then use the listed archive, URL, path-safety, uniqueness, and lenient-parsing cases as coverage; done means all acceptance criteria pass with ValidationError for invalid manifests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.