Bulk and ad-hoc parquet export APIs
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start by reviewing the existing table-partition-to-Parquet export path, then compare the requested SQL COPY ... TO ... WITH FORMAT PARQUET and /exp?query=...&format=parquet entry points. The work is complete when both bulk and ad-hoc exports support the stated Parquet options and produce downloadable, chronologically ordered files without using the WAL.
Written by the indexing model from the issue text.
Description
Currently, Parquet files can be exported from QuestDB by converting a table's partitions to Parquet format, and copying them out. This goes via the WAL, so can impact ingestion performance.
Instead, we will add two more means to export data in parquet format:
- bulk export:
COPY ... TO ... WITH FORMAT PARQUET;- This supports exporting a table or arbitrary query to parquet
- The parquet format can be configured (row group sizes, compression codecs)
- The data can be newly partitioned
- ad-hoc export:
/exp?query=...&format=parquet- This is similar to the above option, but works via the REST API
- This allows you to download smaller datasets in parquet format, instead of CSV.
Chronologically-ordered parquet files can be ingested very quickly into QuestDB. This feature will make it much simpler to get data in and out of the database, and hook it up to other tools such as Pandas/Polars/DuckDB.
- Dominant language
- No language data
- Stars
- 7
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from questdb/roadmap
-
enterprise security
-
open source SQL
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
open source SQL
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
enterprise open format storage
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
enterprise high availability
Difficulty 5/5 Over a week Newbie friendliness 25/100
Similar issues
-
area/sessions comp/agent P2 tool/skills type/perf
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
NousResearch/hermes-agent#117788 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
code-quality refactoring
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
github/gh-aw-firewall#8816 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
getgrav/grav-plugin-api#45 ·