questdb / questdb/py-questdb-client

Add ability for user to set `create_table` and `table_partition` when saving data to QDB

Open
#90 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
71
Forks
14
Avg merge
1h 7m
Merged PRs (30d)
1

Description

Hi,

I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.

I use the below code to save new and old data.
If the table exists - it appends to it. If it does not, it creates it and appends to it.

import pandas as pd
from questdb.ingress import Sender

df = pd.DataFrame({
    'id': pd.Categorical(['toronto1', 'paris3']),
    'temperature': [20.0, 21.0],
    'humidity': [0.5, 0.6],
    'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})

conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    sender.dataframe(df, table_name='sensors', at='timestamp')

This causes a few issue:

  1. upon creation it always creates a partition by DAY

Feature request: add a parameter which is by default DAY but can be changed by the user to be MONTH, NONE or other partition frequency

  1. It silently creates a table by default, even if one does not exist.
    Feature request: add a parameter which is by default True (to create a table) but can be also set to False such that if a table does not exist, it will not create one.

This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the questdb.ingress.Sender.dataframe entry point and trace how missing tables are created and partitioning is selected. Verify behavior for existing and missing tables using the DataFrame example. Done means callers can choose create_table and table_partition while retaining the current defaults and controlling whether missing tables are created.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.