Beginning script at specific point in manifest.json
- Dominant language
- Rust
- Stars
- 3
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
@jwodder just filing an Issue to brainstorm:
When running the `s3invsync`, it takes quite some time to resume much further in the `manifest.json` where I would like a subsequent backup to continue -- I have been using the `--path-filter` as a human-readable mechanism, but I'm curious if you have any thoughts on how I can get `s3invsync` to start at a specific row/delimiter
I understand that it must at least do a regex comparison per row; however, wondering if there could be some sort of `f.seek()` behavior where a pointer could be set to start somewhere specific.
Curious to get your thoughts here -- for context, here is a sample command for what I am running, where the `PREFIX` is determined from an array of processes on MIT Engaging Cluster
```
s3invsync --path-filter "zarr/${PREFIX}[a-z0-9-]*" --allow-new-nonempty --ignore-errors all \
s3://linc-brain-mit-prod-us-east-2/linc-brain-mit-prod-us-east-2/production-configuration/ \
/orcd/data/linc/001/s3lincbrain/
```
Cc @kabilar @yarikoptic @satra
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how s3invsync reads manifest.json and applies --path-filter. The issue does not identify files, tests, or a settled behavior; first clarify the desired starting-point semantics and whether resuming at a row or delimiter is supported. Done would require an agreed design and corresponding implementation and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cli, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100