duckdb / duckdb/duckdb-java

DuckDB adds Ducklake and Postgres. Cannot generate parquet after a while

Open
#423 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
127
Forks
80
Avg merge
13h 49m
Merged PRs (30d)
48

Description

Using Java language

DuckDB adds Ducklake and Postgres.
I'm using Druid to manage connections in Java. Initially, I was able to write metadata to Postgres and generate Parquet files on disk, but after a while, the generation stopped. What's the cause?

·····

The technology stack is: Spring Boot + MyBatis + DruidSource + DuckDB + DuckLake + Postgres.

DruidSource is used to manage DuckDB. DuckDB connections are managed with a data connection pool.

At the beginning of the project, I could generate Parquet files using INSERT INTO statements and view the corresponding metadata in Postgres.

However, after a few hours, the corresponding metadata in Postgres could no longer be viewed, but there were no errors.

I simulated queries using two machines.

Initially, the two machines could share data for queries because the queries were performed through Postgres, and the files were written to disk, generating Parquet files.

However, after a few hours, when one machine wrote data, the Parquet files could not be generated, and the other machine could not read it. I suspect that the data was being written directly to the memory of the current machine.

What is causing this? Is the Postgres connection to DuckDB disconnected?

How can this be avoided?

····

Sample Code :
[DuckDbContextHolder.java](https://github.com/user-attachments/files/22917540/DuckDbContextHolder.java)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.