questdb / questdb/py-questdb-client
Add ability for user to set `create_table` and `table_partition` when saving data to QDB
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 71
- Forks
- 14
- Avg merge
- 1h 7m
- Merged PRs (30d)
- 1
Description
Hi,
I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.
I use the below code to save new and old data.
If the table exists - it appends to it. If it does not, it creates it and appends to it.
import pandas as pd
from questdb.ingress import Sender
df = pd.DataFrame({
'id': pd.Categorical(['toronto1', 'paris3']),
'temperature': [20.0, 21.0],
'humidity': [0.5, 0.6],
'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})
conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
sender.dataframe(df, table_name='sensors', at='timestamp')
This causes a few issue:
- upon creation it always creates a partition by
DAY
Feature request: add a parameter which is by default DAY but can be changed by the user to be MONTH, NONE or other partition frequency
- It silently creates a table by default, even if one does not exist.
Feature request: add a parameter which is by defaultTrue(to create a table) but can be also set toFalsesuch that if a table does not exist, it will not create one.
This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the questdb.ingress.Sender.dataframe entry point and trace how missing tables are created and partitioning is selected. Verify behavior for existing and missing tables using the DataFrame example. Done means callers can choose create_table and table_partition while retaining the current defaults and controlling whether missing tables are created.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- database
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100