ni / ni/datastore-python

Data Frame Library Support needed to integrate in Pythons Data Science Ecosystem.

Open
#114 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 14h
Merged PRs (30d)
6

Description

The current API is low level resulting in a high learning curve and manual work in order to leverage commonly used data science libraries in python like Pandas, Polars etc. I have not come across a python customer that did not use data frame libraries for post processing, data analysis and automating report generation.

Some of the pain points:

  1. It requires lots of iterating over objects, no table / column level operations.
  2. Hard to merge / join different query results (tables).
  3. Need for using separate clients (Data vs MetaData)
  4. Need for separate calls to read measurement data.
  5. String based queries

To better enable customer workflow's and reduce the learning curve our API needs options to return data frames. Using something like pyArrow would allow customers to choose their data frame library of choice.

Data Frames support will significantly reduce effort needed for adoption of MDS and integration with existing workflows at customers.

AB#3752380

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points. Start by mapping the current Python API's data and metadata clients, measurement reads, and string-based queries, then assess a PyArrow-based return path for Pandas and Polars. Done should include an agreed scope and acceptance criteria for data frame support.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
api, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.