borgbackup / borgbackup/borg

repo-info --stats: report repository size (and optionally dedup stats) read-only, with --json

Open
#10,329 2 comments 0 reactions 0 assignees View on GitHub
cmd: info
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

## Problem

Borg 2 has no read-only way to learn how large a repository is.

- `repo-info --json` reports id, location, `last_modified`, encryption and cache paths, no size.
- `info --json` (archive level) reports `nfiles` and `original_size` only. Its help says: "for the repository-wide deduplicated size, use borg compact --stats."
- `compact --stats` does report it ("Repository size is 402 kB in 8 objects", plus source data size, deduplicated size, compression factor), but only while compacting: it needs the exclusive lock, rewrites packs, and `compact --dry-run --stats` answers "Ignoring --stats. It is not supported when using --dry-run." The numbers are also only printed as rounded INFO log lines, so a program has to parse "402 kB" out of a `log_message`.
- `repo-list --format "{size}"` is per-archive original size, not repository size.

For anything that wants to show or monitor repository size without changing the repository (a UI, a Prometheus exporter, a cron report), the only options today are store-level measurements (`du`, `rclone size`, walking the borgstore backend), which measure files, not Borg objects, and which cannot be expressed uniformly across `file:`, `sftp:`, `rest:`, `rclone:` and `s3:` stores.

This is the read-only half of #8796 (closed with "use `compact --stats -v`") and overlaps the monitoring wish in #2934.

## Observation: the data is already there and cheap

Measured on 2.0.0b23 and 2.0.0b24 (identical):

1. **Repository size** needs nothing but the chunk index. `Repository.list()` yields `(chunk_id, storage_size)` from the `index/` fragments under a shared lock, without loading a pack. Summing it reproduces `compact --stats`'s "Repository size" byte for byte (402434 bytes = "402 kB in 8 objects"; `du` of `packs/` is ~5 % higher through pack headers). On a small scratch repository this takes 15 ms.

```python
from borg.logger import setup_logging; setup_logging()
from borg.repository import Repository
from borg.helpers import Location
with Repository(Location(url), exclusive=False) as r:
sizes = [s for _, s in r.list()]
print(len(sizes), sum(sizes))
```

2. **Source data size, file count and deduplicated size** are what `CompactCmd.analyze_archives()` computes, and since the archive reference caches landed (`cache/referenced_by_archive_` in the store, `get_archive_references(..., cached=True)`), an unchanged archive is not even opened again. A read-only variant is exactly the `dry_run` path of that method (`store=False`, no cache cleanup).

So a read-only `repo-info --stats` is a re-use of two existing code paths, not new bookkeeping.

## Proposal

Add `--stats` to `repo-info`:

```
borg repo-info --stats [--json]
```

Text output (same wording as `compact --stats`, so users recognise it):

```
Repository size: 402 kB in 8 objects
Source data size: 700 kB in 2 files (2 archives)
Deduplicated size: 400 kB
Deduplication factor: 0.57
Compression factor: 1.00
```

JSON: add a `stats` object with exact byte values, alongside the existing keys, so `repo-info --json` stays backwards compatible without the flag (values illustrative):

```json
{
"repository": {"id": "...", "location": "...", "last_modified": "..."},
"encryption": {...},
"cache": {...},
"stats": {
"repository_size": 402434,
"object_count": 8,
"source_size": 700436,
"source_files": 2,
"archive_count": 2,
"deduplicated_size": 400304
}
}
```

Semantics:

- `repository_size` / `object_count`: sum and count over the chunk index (what `compact --stats` prints after compaction). Always available, shared lock, no pack access.
- `source_size`, `source_files`, `archive_count`, `deduplicated_size`: from the read-only analysis of all archives, using the per-archive reference caches when present. Archives without a cache are scanned (as `compact --dry-run` does today) but nothing is written to the store.
- No `compaction_saved`, since nothing is compacted.
- Requires a shared lock only, so it can run while a backup is in progress, unlike `compact`.

If the archive analysis is considered too expensive for `repo-info`, a smaller first step would already help most callers: `repository_size` and `object_count` only (index sum), with the dedup fields behind a second flag (`--stats --analyze`) or left to `compact --stats`. Alternatively, make `compact --dry-run --stats` honour `--stats` instead of ignoring it; that gives the same numbers read-only, but still under the exclusive lock and without JSON.

## Why not parse `compact --stats -v`

- It changes the repository, so a monitor cannot call it on its own schedule.
- It takes the exclusive lock.
- The values are rounded to three significant digits and only reachable through INFO `log_message` lines.
- Borg 1's `info --json` exposed `cache.stats` with exact `unique_csize`, `unique_size`, `total_size`; tools built on that lost the field with Borg 2 and currently have no replacement.

## Environment

borg 2.0.0b23 and 2.0.0b24, borgstore 0.6.1, Python 3.12, Linux; measurements above on a `file:` store; the same `repo-info` runs against `rest:` stores in production.

Contributor guide

Open the contributing guide

Research direction

Start with the repo-info command and Repository.list() for the shared-lock repository size and object count. Then read CompactCmd.analyze_archives() and its dry_run path to understand the optional archive analysis and reference caches. Done means read-only text and JSON stats with exact values, while preserving existing repo-info JSON output without --stats.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.