Basekick-Labs / Basekick-Labs/arc

Performance: ORDER BY time DESC is ~2x slower than ASC

Open
#153 2 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Summary

Queries with `ORDER BY time DESC` are approximately 2x slower than `ORDER BY time ASC` when using `LIMIT`.

## Observed Behavior

```sql
-- ASC: ~465ms
SELECT * FROM production.cpu
WHERE time > now() - INTERVAL '7 days'
ORDER BY time ASC
LIMIT 10000

-- DESC: ~890ms (nearly 2x slower)
SELECT * FROM production.cpu
WHERE time > now() - INTERVAL '7 days'
ORDER BY time DESC
LIMIT 10000
```

## Root Cause

Parquet files are sorted by `time ASC` during ingest (default sort key). This means:

- **ASC + LIMIT**: DuckDB reads the first N rows from already-sorted data and stops early
- **DESC + LIMIT**: DuckDB must scan more data to find the last N rows, then reverse sort

## Impact

Most time-series dashboards want **newest data first** (`ORDER BY time DESC`), which is the slower path.

## Potential Solutions

1. **Add configurable sort order** - Allow `time:desc` as default sort key in config
```toml
[ingest]
default_sort_keys = "time:desc"
```

2. **Dual-sorted files** - Store data sorted both ways (increases storage ~2x)

3. **Reverse-time column** - Add a computed column like `_reverse_time = MAX_TIMESTAMP - time` that's indexed

4. **Query optimizer hints** - Detect DESC+LIMIT patterns and use different read strategy

5. **Parquet row group ordering** - Investigate if DuckDB can read row groups in reverse order efficiently

## Questions for Discussion

- What's the typical query pattern for Arc users? (ASC vs DESC)
- Is the storage overhead of dual-sorting acceptable?
- Are there DuckDB-specific optimizations we can leverage?

## Environment

- Arc version: latest (feature/tiered-storage branch)
- DuckDB: embedded
- Storage: Local + S3 tiered storage

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.