Basekick-Labs / Basekick-Labs/arc
feat: Change default sort order to DESC for time-series optimization
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
Change the default sort order from `time ASC` to `time DESC` for both ingest and compaction.
## Rationale
Time-series data loses value as time passes:
- What happened to a sensor **1 minute ago** is more important than what happened **1 month ago**
- Most queries want "newest first" (`ORDER BY time DESC LIMIT N`)
- Dashboards, alerts, and monitoring all show recent data first
- Debugging typically starts with "what just happened?"
Current default (`time ASC`) optimizes for the less common query pattern.
## Changes Required
### 1. Ingest (`internal/ingest/arrow_writer.go`)
Change default sort keys from `["time"]` to `["time:desc"]` or equivalent
### 2. Compaction (`internal/compaction/job.go`)
Change `ORDER BY time` to `ORDER BY time DESC` in compaction query
### 3. Configuration
- Update `arc.toml` defaults
- Document the change in release notes
- Consider migration path for existing data
## Migration Considerations
- Existing data sorted ASC will remain ASC until re-compacted
- New data will be sorted DESC
- Mixed ASC/DESC files in same partition could affect query planning
- Option: Add a one-time "re-sort" compaction flag
## Performance Impact
Based on testing (see #153):
- `ORDER BY time DESC LIMIT N` will become the fast path (~120ms)
- `ORDER BY time ASC LIMIT N` will become slower (~650ms)
- This matches actual usage patterns better
## Related
- #153 - Performance: ORDER BY time DESC is ~2x slower than ASC
## Target
Next minor version (not current release)
Contributor guide
Assessment
This issue has not been assessed yet.