Basekick-Labs / Basekick-Labs/arc

feat: Change default sort order to DESC for time-series optimization

Open
#155 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

Change the default sort order from `time ASC` to `time DESC` for both ingest and compaction.

## Rationale

Time-series data loses value as time passes:
- What happened to a sensor **1 minute ago** is more important than what happened **1 month ago**
- Most queries want "newest first" (`ORDER BY time DESC LIMIT N`)
- Dashboards, alerts, and monitoring all show recent data first
- Debugging typically starts with "what just happened?"

Current default (`time ASC`) optimizes for the less common query pattern.

## Changes Required

### 1. Ingest (`internal/ingest/arrow_writer.go`)
Change default sort keys from `["time"]` to `["time:desc"]` or equivalent

### 2. Compaction (`internal/compaction/job.go`)
Change `ORDER BY time` to `ORDER BY time DESC` in compaction query

### 3. Configuration
- Update `arc.toml` defaults
- Document the change in release notes
- Consider migration path for existing data

## Migration Considerations

- Existing data sorted ASC will remain ASC until re-compacted
- New data will be sorted DESC
- Mixed ASC/DESC files in same partition could affect query planning
- Option: Add a one-time "re-sort" compaction flag

## Performance Impact

Based on testing (see #153):
- `ORDER BY time DESC LIMIT N` will become the fast path (~120ms)
- `ORDER BY time ASC LIMIT N` will become slower (~650ms)
- This matches actual usage patterns better

## Related

- #153 - Performance: ORDER BY time DESC is ~2x slower than ASC

## Target

Next minor version (not current release)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.