questdb / questdb/questdb

PostgreSQL ingestion in a partitioned database affects partition names

Open
#2,846 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
17.3k
Forks
1.6k
Avg merge
5d 10h
Merged PRs (30d)
28

Description

Describe the bug

Hi,

I am using QuestDB to store message/username data from a Twitch Chat. I have some data from previous streams which I uploaded via the web interface/csv upload functionality. My table is partitioned by day, and looking at the directory structure, all the folder names are correct.

The issue arose when I began using the PostgreSQL wire protocol to add data during the livestreams. Now, the folder names are messed up and have decimals at the end of the date rather than the HH:MM:SS format. I implemented this on October 7th, which is when the filenames changed.
image

image
I am trying to drop the most recent partition due to some bad data, but entering ALTER TABLE chat_data DROP PARTITION LIST '2022-12-06' in the web interface returns the error message:

image

Entering ALTER TABLE chat_data DROP PARTITION LIST '2022-12-06.988' instead gives the error message:

image

It seems that the decimal is affected by when the first piece of data was added to the partition - the first piece of data that was ingested on 2022-12-06 happened around 98.8% of the way through the day (UTC time).

I can't rename the folder in the file structure without damaging the database integrity (I tried and had to restore via a saved csv), nor can I just delete the folder without causing the same issue.

Is there a way to drop this partition?

To reproduce
  1. Create a table partitioned by day
  2. Use the PostgreSQL Wire Protocol to append data in batches
Expected Behavior

I should be able to drop the partition using ALTER TABLE chat_data DROP PARTITION LIST '2022-12-06' or ALTER TABLE chat_data DROP PARTITION LIST '2022-12-06.988'

Environment
- **QuestDB version**: 6.5.3
- **OS** Windows 11 Home:
- **Browser** Firefox 107.0.1:
Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue by creating a day-partitioned table and appending batches through the PostgreSQL wire protocol, then compare the resulting partition names with CSV upload behavior. Investigate the partition naming and DROP PARTITION handling described in the report. Done means the affected partition can be dropped using the documented date form without damaging the database.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.