finagle / finagle/finagle-postgres

Cursors and Performance

Open
#14 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Scala
Stars
81
Forks
36
PR merge metrics
No merged PRs in 30d

Description

We are running a PostgreSQL database with Infobright and very large data sets. We are running some rather involved queries against the data, which can return millions of rows, which we need to process as quickly as possible. Using the normal PostgreSQL JDBC driver is way too slow. I tried running our query through this Finagle 'driver' and the performance was even worse.

We get the best performance using the native libpg C library and cursors. It seems to me that we should be able to get pretty close to that in terms of performance from finagle-postgres... but I don't see the path towards that.

Is there "missing implementation" for cursors or streaming results from large queries? I would be willing to help in the implementation here if someone can point me in the right direction.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file, test, or entry point is named. Start by tracing how finagle-postgres handles large query results and comparing that behavior with PostgreSQL cursor and streaming semantics; done should include a clearly scoped implementation plan and measurements against the native libpq approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, scala
Domain
backend-api-design, databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.