trinodb / trinodb/trino-python-client

Empty results on large queries when using fetchall()

Open
#44 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
439
Forks
206
PR merge metrics
No merged PRs in 30d

Description

Hello there.

I've found that these two examples will not always return the same thing on large queries:

  1. fetchall
cur.execute(large_query)
rows = cur.fetchall()
print(rows)
  1. iterator
for r in cur.execute(large_query):
  print(r)

The fetchall approach ends up returning an empty list every time I use it when running large queries, but the iterator approach prints the result successfully

Does anyone know why this may be happening?

I don't know how I should supply a working example to debug this problem, as my queries all contain domain logic of my company. Nevertheless, I can say that I'm connecting around 8 Postgres databases with 100k rows per table and equal schemas, and I union the tables from all databases together with UNION views, like this:

CREATE OR REPLACE VIEW hive.default."table_unioned_view" AS 
SELECT 'database1' AS database, "other_column" FROM database1.public.table
UNION ALL
SELECT 'database2' AS database, "other_column" FROM database2.public.table
UNION ALL
SELECT 'database3' AS database, "other_column" FROM database3.public.table;

And the big query that I'm running uses these type of views

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the cursor.execute/fetchall path with cursor iteration using a minimal large-query example, since the report cannot provide a reproducible query. Check whether the behavior depends on the UNION views across the eight PostgreSQL databases; done means identifying and reproducing the cause of the differing results.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres, python
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.