GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

Use ibis DuckDB backend to access pandas dataframe instead of ibis pandas backend

Open
#1,702 1 comment 0 reactions 0 assignees View on GitHub
type: feature request
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

**Is your feature request related to a problem? Please describe.**
A feature request - DVT currently uses a pandas backend that has been removed in Ibis 10.0 (we are currently on Ibis 5.1). You can use DuckDB to access data stored in Pandas dataframes. There are some performance benefits and some feature changes. Ibis explains why in this [article](https://ibis-project.org/posts/farewell-pandas). In version 9.0, ibis introduced the concept of a default backend and they chose [DuckDB](https://ibis-project.org/posts/why-duckdb/).
**What version of DVT are you using?**
8.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.