maximilien / maximilien/weave-cli
feat: Enhanced Ingestion Pipeline with Resume/Checkpointing
Open
Nobody has claimed this yet.
enhancement
feature
p2
phase-2
- Dominant language
- Go
- Stars
- 34
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Summary
Enhance the ingestion pipeline with resilience features for production workloads.
Scope
- Add resume from failures capability
- Implement checkpointing
- Add deduplication
- Add validation before insert
- Track batch metrics
- Improve error reporting
- Add dry-run mode
- Export metrics to JSON/YAML
- Test resilient ingestion
Commands
weave stack ingest Docs data/ --resume
weave stack ingest Docs data/ --dry-run
weave stack ingest Docs data/ --validate
weave stack ingest Docs data/ --report metrics.json
Success Criteria
- Enhanced ingestion handles failures gracefully
- Checkpointing works correctly
- Metrics export functional
Priority: Phase 2 - Week 2
Estimated: 2-3 days
Assignee: TBD
Related
Can be developed in parallel with cloud deployments.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the weave stack ingest command and trace the existing ingestion path. Review how input batches and inserts are currently handled, then define tests for resume/checkpointing, deduplication, validation, dry-run, and metrics export. Done means the listed commands work and failure recovery is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100