maximilien / maximilien/weave-cli

feat: batch file ingestion (weave docs create-batch)

Open
#38 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement stale
Dominant language
Go
Stars
34
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Problem

There is no native multi-file ingestion command. Client0 wrote a 255-line bash script to
process 20 PDFs sequentially with retry, Milvus restart, and skip/resume logic. This
complexity belongs inside weave-cli.

Proposed Solution

New subcommand `weave docs create-batch`:

```bash
weave docs create-batch AuctionListings "data/pdfs/*-catalogue.pdf"
--milvus-local
--embedding text-embedding-3-small
--skip-existing
--timeout 30m
--delay 10s
--max-retries 3
--log-file logs/ingestion.log
--checkpoint-file .ingestion-checkpoint.json
```

Key Features

  1. Glob expansion — already in v0.9.27
  2. Configurable delay between files (`--delay 10s`) — prevents VDB memory buildup
  3. Retry with backoff (`--max-retries 3`, `--retry-delay 30s`) — auto-retry failed files
  4. Checkpoint/resume (`--checkpoint-file`) — save state after each file, resume on crash
  5. Structured log file (`--log-file`) — timestamped append-mode log
  6. Auto-create image collection — if `--image-collection` doesn't exist, create it
  7. Batch summary at completion:
    ```
    ✅ Batch complete: 9/9 succeeded, 0 failed, 2 skipped
    Duration: 1h 23m | Log: logs/ingestion.log
    ```

Checkpoint File Format

```json
{
"collection": "AuctionListings",
"started": "2026-02-17T10:00:00Z",
"completed": [
{"file": "2017-catalogue.pdf", "chunks": 28, "at": "2026-02-17T10:12:33Z"}
],
"failed": [],
"skipped": []
}
```

Client0 Impact

Replaces the entire 255-line bash wrapper. Two `weave docs create-batch` calls replace the
entire pipeline script.

Priority

P1 — v0.9.29 target (~10 hours)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing weave docs create subcommand and the v0.9.27 glob-expansion path. Read how one file is ingested, how Milvus and image collections are handled, and where CLI options are defined. Done means the new command supports the listed delay, retry, checkpoint, logging, skip/resume, collection, and summary behaviors with tests for the batch flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.