apache / apache/arrow-adbc

Implement ingestion of list types for SQLite

Open
#2,212 10 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
C#
Stars
627
Forks
217
Avg merge
17h
Merged PRs (30d)
57

Description

### What feature or improvement would you like to see?

In https://github.com/apache/arrow-adbc/issues/2066 @skarakuzu and I outlined a use case for streaming LIST data into PostgreSQL and SQLite via ADBC. Support for PostgreSQL was [recently added](https://github.com/apache/arrow-adbc/pull/2153). (Thanks, @paleolimbot!)

Would it be justifiable to add LIST support for SQLite as well? Our use case is embedded scientific data collection scenarios, where it is not feasible to deploy PostgreSQL but we'd like a compatible-as-possible data model. In #2066 had floated an idea of using a SQLite JSON or JSONB column for this, as SQLite is generally loose about types and lacks any array/list type.

We would be happy to try to contribute this if that is feasible.

Contributor guide

Open the contributing guide

Research direction

Start by reading issue #2066 and the PostgreSQL implementation in pull request #2153 to understand the existing LIST-ingestion approach. Then determine how SQLite JSON or JSONB columns should represent LIST data and what tests would demonstrate compatible ingestion; done means the design is agreed and SQLite support is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.