crate / crate/cratedb-guide

Feedback on /connect/general: Add insights into HTTP vs. PG bulk ingesting

Open
#572 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
3
Forks
4
Avg merge
3d 18h
Merged PRs (30d)
1

Description

### Documentation feedback


- Page title: General information
- Page URL: https://cratedb.com/docs/guide/connect/general.html
- Source: https://github.com/crate/cratedb-guide/blob/main/docs/connect/general.md

---

### About

@WalBeh was conducting a few orientation flights using [inserter](https://github.com/WalBeh/inserter). Insights are shared below.

### Summary

Grundsätzlich is pg/http fast gleich auf. Was allerdings bemerkenswert ist, ist dass bei ähnlicher insert Leistung die Last mit psql am Server ca. 20-25% niedriger ist.

### Details

Clear picture over 9 minutes:

```
┌─────────────────────┬──────────────────┬────────────────┐
│ │ HTTP │ PG │
├─────────────────────┼──────────────────┼────────────────┤
│ effective rec/cpu/s │ 8,985 │ 11,922 │
├─────────────────────┼──────────────────┼────────────────┤
│ p90 rec/s │ 331,381 │ 413,609 │
├─────────────────────┼──────────────────┼────────────────┤
│ rejected │ 1,684,833 (1.1%) │ 210,696 (0.3%) │
├─────────────────────┼──────────────────┼────────────────┤
│ queue avg │ 149.8 │ 33.9 │
├─────────────────────┼──────────────────┼────────────────┤
│ queue p95 │ 200 (wall) │ 186 │
├─────────────────────┼──────────────────┼────────────────┤
│ active threads avg │ 31.0 │ 18.6 │
├─────────────────────┼──────────────────┼────────────────┤
│ latency avg │ 109ms │ 83ms │
└─────────────────────┴──────────────────┴────────────────┘
```

My judgment: PG is definitively better for bulk inserts — 33% more throughput per CPU with 75% less queue pressure. Over a longer run the gap widens because HTTP degrades under sustained pressure (queue avg 150 = constantly near the wall, latency creeps up, rejections compound).

The HTTP effective dropped from ~10,650 (2min) to 8,985 (9min) — that's the server struggling under sustained saturation. PG stayed stable at ~11,900 regardless of duration.

For the throttling feature: I'd implement it on both, but it matters more for HTTP since PG naturally runs cooler. The queue data makes it clear — HTTP is slamming the queue at 75% capacity on average, while PG sits at 17%. Throttling would bring HTTP closer to PG's behavior.

Regarding HTTP causing higher CrateDB load: that's expected and not a bug in our code. The server does more work per request with HTTP (parse headers, decompress gzip,
deserialize JSON body) vs PG wire (binary parameters, no decompression). We confirmed this with thread pool monitoring — same records, same batches, but HTTP keeps 31/32 write
threads busy (load 31) while PG only uses 19/32 (load 21).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.