PostgREST / PostgREST/postgrest

[Question] Postgrest overhead w.r.t direct DB query

Open
#2,005 8 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

perf question
Dominant language
Haskell
Stars
27.7k
Forks
1.2k
Avg merge
1d 9h
Merged PRs (30d)
54

Description

Environment
  • PostgreSQL version: AWS RDS 13.3 (2 vCPU, 8GB Mem)
  • PostgREST version: Docker (postgrest/postgrest:v8.0.0)
# 4 Pods
resources:
    limits:
        cpu: 1500m
        ephemeral-storage: 1Gi
        memory: 256Mi
    requests:
        cpu: 750m
        ephemeral-storage: 250Mi
        memory: 256Mi
  • Operating system: AWS EKS (1.15)
Description of issue

We ran some load tests comparing Postgrest against direct DB queries, and saw that the overhead is significant. The setup is described below.

create table if not exists perf_test (
    id         bigserial,
    name       text,
    value      jsonb,
    created_at timestamp with time zone default now() not null
);

create index if not exists idx_id
    on perf_test (id);

Table has ~ 2M rows.

Tests were executed from K8s using Locust.

Benchmark DB query

def execute_query(self):
    id =  random.randint(1, 2000000)
    id2 = id + 20
    self.client.execute_query("select * from perf_test where id between {0} and {1}".format(id, id2))

Benchmark Postgrest request

def get_perf_test(self):
    id = random.randint(1, 2000000)
    id2 = id + 20
    headers = {'Accept-Encoding': 'gzip', 'Connection': 'Keep-Alive', 'Keep-Alive': 'timeout=5, max=2000'}
    self.client.get(f"/perf_test?id=geq.{id}&id=le.{id2}", name="/perf_test", headers=headers)

The DB queries are able to scale up to 2K TPS with a latency (P95) of ~ 2ms. The corresponding Postgrest requests are topping off at ~ 10ms (P95) with a TPS of ~50.

Is this amount of overhead expected?
Are there any recommended configurations / best practices to optimize throughput and latency of Postgrest?
Are there any benchmarks that you have done with this kind of comparison?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied PostgreSQL schema, direct-query benchmark, and PostgREST request run through Locust on the stated Kubernetes setup. Review the reported latency and throughput difference, then identify the expected overhead, relevant configuration or best practices, and comparable benchmarks. The issue does not define a code change or concrete acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, haskell, kubernetes, postgresql
Domain
api, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.