Basekick-Labs / Basekick-Labs/arc

Tamper-evident audit log: hash chain + verification endpoint

Open
#703 4 comments 0 reactions 0 assignees View on GitHub
compliance design enhancement help wanted security
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Context

Arc Enterprise has real audit logging (`internal/audit/`, gated on `FeatureAuditLogging`), but the audit log is a **plain mutable SQLite table**. Anyone who can write to the SQLite file can silently edit or delete audit rows, and the retention loop deletes old entries with no record that a deletion happened.

This matters because the audit log is the artifact that is supposed to *prove* nothing happened. Right now it is the softest target in the system: Parquet data files are immutable and require a full rewrite to alter, but the log that would record that rewrite can be edited with one `UPDATE`.

Regulated buyers (AMS2750 aerospace pyrometry, 21 CFR Part 11, Nadcap) ask a specific question: *"What prevents an IT administrator from backdating or altering a record?"* Today our honest answer is "filesystem permissions." That is not a sufficient answer for these standards, and it is the single most valuable gap to close for the regulated-industry segment.

## Current state

`internal/audit/audit.go:118` — schema has no integrity columns:

```sql
CREATE TABLE IF NOT EXISTS audit_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
event_type TEXT NOT NULL,
actor TEXT,
method TEXT NOT NULL,
path TEXT NOT NULL,
database_name TEXT,
measurement TEXT,
status_code INTEGER,
ip_address TEXT,
user_agent TEXT,
duration_ms INTEGER,
detail TEXT
);
```

`grep -rniE "hash|sha256|hmac|sign|chain|prev" internal/audit/` returns nothing but a comment about vacuuming.

## Goal

Make any modification, insertion, backdating, or deletion of audit entries **cryptographically detectable**, and expose an endpoint that proves it on demand.

## Proposed design (open to counter-proposals — this is a `design` issue)

**1. Hash chain columns**

Add `prev_hash TEXT` and `entry_hash TEXT` to `audit_logs`. On each insert:

```
entry_hash = SHA-256(canonical(entry) || prev_hash)
```

`canonical(entry)` must be a deterministic serialization of the entry fields. **Pick an unambiguous encoding and document it** — a naive `field1|field2|field3` join is vulnerable to delimiter injection if any field can contain the delimiter (a `path` or `user_agent` certainly can). Length-prefixing each field, or a canonical-JSON form with sorted keys, both work. State the choice in a doc comment; this is a security boundary.

**2. Verification endpoint**

`GET /api/v1/audit/verify` walks the chain and returns the first break with its entry ID, or confirms integrity. Must be admin-auth'd + license-gated like the rest of the audit API. For large tables it needs a bounded/resumable mode rather than walking millions of rows in one request — propose an approach.

**3. Checkpoint sealing**

Periodically write a checkpoint (head hash + entry count + timestamp, signed) to **object storage**, not to the same SQLite file.

This part is what makes the scheme actually defensible. If the checkpoint lives next to the chain, an attacker with DB access rewrites both and the chain verifies clean. The checkpoint must land somewhere the DB admin cannot rewrite — object storage, ideally with Object Lock (see the companion issue). Please treat "where does the checkpoint live and who can write it" as a first-class design question, not an implementation detail.

**4. Retention pruning must not look like tampering**

`cleanupOldEntries()` (`internal/audit/audit.go:301`) deletes rows older than `RetentionDays`. With a hash chain, that breaks verification — legitimately, but indistinguishably from an attack.

Pruning must write a **tombstone** into the chain recording what was removed: range of IDs, count, the hash of the last pruned entry, and the timestamp of the prune. Verification then treats a tombstone as a valid chain link rather than a break.

**5. Fix the unbatched DELETE while you're in here**

`internal/audit/audit.go:308` currently issues a single unbounded `DELETE FROM audit_logs WHERE timestamp < ?`. On a large audit table this holds the SQLite write lock for seconds and blocks ingest file registration and auth token updates. Per the SQLite Review Checklist in `.claude/CLAUDE.md`, this must be chunked at 1000 rows with `LIMIT`, an `ORDER BY` aligned with `idx_audit_timestamp`, and a `ctx.Err()` check between batches. This code is being rewritten for tombstones anyway.

## Design questions to resolve before coding

- Canonical serialization format (delimiter-injection-safe — say which and why)
- Checkpoint storage target, signing key management, and rotation
- Verification performance on multi-million-row tables (resumable? sampled? background?)
- Behavior on unclean shutdown mid-write — can the chain end in a torn state, and how is that distinguished from tampering?
- Migration for existing deployments with populated `audit_logs` (genesis entry? chain starts at upgrade?)
- Clustered deployments: one chain per node, or a merged chain? (Per-node is almost certainly right — say so explicitly and justify it.)

## Acceptance criteria

- [ ] Design written up in `docs/progress/` and agreed **before** implementation
- [ ] `prev_hash` / `entry_hash` populated on every entry
- [ ] `GET /api/v1/audit/verify` detects modification, deletion, and insertion — with a test for each of the three
- [ ] Retention pruning writes tombstones; verification passes across a prune
- [ ] Retention DELETE batched per SQLite Review Checklist
- [ ] Checkpoints written to object storage
- [ ] Migration path for existing audit tables
- [ ] Every new config key has `v.SetDefault()` in `internal/config/config.go`
- [ ] Release notes + `docs/progress/2026-01-08-arc-enterprise-product-definition.md` updated

## Notes for contributors

Read `.claude/CLAUDE.md` first — particularly the SQLite Review Checklist and the configuration-matrix review process. This touches `*sql.DB` on a hot path, so the SQLite checklist applies in full.

**Please open the design discussion in this issue before writing code.** The crypto and the tombstone semantics are where this succeeds or fails; the Go is comparatively easy. Happy to review a design comment quickly.

Related: companion issue on S3 Object Lock / WORM storage.

Contributor guide

Open the contributing guide

Research direction

Read .claude/CLAUDE.md first, especially the SQLite Review Checklist, then inspect internal/audit/audit.go and the existing audit API. Use the listed design questions and the companion Object Lock issue to frame the design discussion before coding. Done initially means an agreed design in docs/progress/covering canonicalization, checkpoints, verification, pruning, migration, and deployment behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
api, cloud, cryptography, databases, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.