Basekick-Labs / Basekick-Labs/arc
Performance: ORDER BY time DESC is ~2x slower than ASC
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Summary
Queries with `ORDER BY time DESC` are approximately 2x slower than `ORDER BY time ASC` when using `LIMIT`.
## Observed Behavior
```sql
-- ASC: ~465ms
SELECT * FROM production.cpu
WHERE time > now() - INTERVAL '7 days'
ORDER BY time ASC
LIMIT 10000
-- DESC: ~890ms (nearly 2x slower)
SELECT * FROM production.cpu
WHERE time > now() - INTERVAL '7 days'
ORDER BY time DESC
LIMIT 10000
```
## Root Cause
Parquet files are sorted by `time ASC` during ingest (default sort key). This means:
- **ASC + LIMIT**: DuckDB reads the first N rows from already-sorted data and stops early
- **DESC + LIMIT**: DuckDB must scan more data to find the last N rows, then reverse sort
## Impact
Most time-series dashboards want **newest data first** (`ORDER BY time DESC`), which is the slower path.
## Potential Solutions
1. **Add configurable sort order** - Allow `time:desc` as default sort key in config
```toml
[ingest]
default_sort_keys = "time:desc"
```
2. **Dual-sorted files** - Store data sorted both ways (increases storage ~2x)
3. **Reverse-time column** - Add a computed column like `_reverse_time = MAX_TIMESTAMP - time` that's indexed
4. **Query optimizer hints** - Detect DESC+LIMIT patterns and use different read strategy
5. **Parquet row group ordering** - Investigate if DuckDB can read row groups in reverse order efficiently
## Questions for Discussion
- What's the typical query pattern for Arc users? (ASC vs DESC)
- Is the storage overhead of dual-sorting acceptable?
- Are there DuckDB-specific optimizations we can leverage?
## Environment
- Arc version: latest (feature/tiered-storage branch)
- DuckDB: embedded
- Storage: Local + S3 tiered storage
Contributor guide
Assessment
This issue has not been assessed yet.