demml / demml/scouter

Trace latency: move maintenance to leases and add recent Compact

Open
#287 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
13
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Parent: #281

# Goal

Move Delta table maintenance under Postgres-coordinated leases and add recent bin-pack compaction where it is supported by the Delta writer stack.

This reduces small-file pressure without letting multiple pods compact, vacuum, or refresh table state in conflicting ways.

# Scope

- Add lease-managed maintenance tasks for DataFusion/Delta tables.
- Keep nightly Z-ORDER where it already makes sense.
- Add a separate recent bin-pack Compact task for active partitions if the pinned Delta implementation supports it.
- Validate support for `OptimizeType::Compact` and target-size configuration in the pinned Delta crate before assuming the API exists.
- Do not rely on `delta.autoOptimize.optimizeWrite=true` unless the pinned Rust Delta implementation actually honors it.
- Add refresh-origin tracking so background refreshes, write-commit refreshes, maintenance refreshes, and any accidental request-path refreshes can be distinguished.
- Ensure maintenance covers trace spans, trace summaries, GenAI trace tables, Bifrost datasets, dispatch records if applicable, and eval scenarios.

# High-level design

Small files multiply footer reads and bloom/index page checks. Recent bin-pack compaction is useful because daytime writes create small files before nightly Z-ORDER has a chance to cluster them.

Maintenance should be lease-driven. In multi-pod deployments, one pod should own a maintenance action at a time, while other pods keep serving from their current snapshots and pick up the new Delta version through background refresh.

# Acceptance criteria

- Maintenance tasks are protected by Postgres leases.
- Compact support is verified against the pinned Delta crate before implementation.
- If Compact is unsupported, the issue records the supported alternative instead of silently implementing a no-op.
- Refresh-origin metrics exist and show user requests are not waiting on maintenance refreshes.
- Benchmarks show file-count and object-store request-count impact for recent partitions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.