apache / apache/datafusion

Sub-optimal `information_schema` table retrieval logic

Open
#11,865 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Presently the `information_schema.tables` builder serially loads all tables when constructing the output https://github.com/apache/datafusion/blob/bddb6415a50746d2803dd908d19c3758952d74f9/datafusion/core/src/catalog_common/information_schema.rs#L93-L102

In our case those are Delta tables with the implications that:
- Each load (likely) results in network request(s) to an object store, so hitting many of them in series will result in slow-down (see https://github.com/splitgraph/seafowl/issues/589 for an example)
- Since we already have the table name the only reason table loading happens is to fetch the table type, which in case of Delta tables is hard-coded https://github.com/delta-io/delta-rs/blob/aa28d730e1d69ed419f2dc22404c5bbab8e98647/crates/core/src/delta_datafusion/mod.rs#L700

### Describe the solution you'd like

It seems that loading the full `TableProvider` for each table is an overkill since we only ever want to know the table types.
In addition it would be preferable to have a bulk load method, in case when the table type is not hard-coded and must be fetched from an external source.

In principle this could be achieved by having a method on the schema provider that returns `Vec`, since `TableSource` also has the table type.

This gets further complicated having in mind that `information_schema.columns` and `information_schema.views` also do this serial table loading, but in their case it's table schema and table definition that's fetched. `TableSource` does have the former, but not the later.

Moreover, to get a Delta table's schema you really want to [load](https://github.com/delta-io/delta-rs/blob/aa28d730e1d69ed419f2dc22404c5bbab8e98647/crates/core/src/table/mod.rs#L316) it to get the latest snapshot (unless you also keep track of it someplace else) which brings us back to the initial problem.

So maybe something like a new trait along the lines of
```rust
pub trait TableInfoProvider {
/// Get a reference to the schema for this table
fn schema(&self) -> SchemaRef;

/// Get the type of this table for metadata/catalog purposes.
fn table_type(&self) -> TableType;

/// Get the create statement used to create this table, if available.
fn get_table_definition(&self) -> Option<&str> {
None
}
}

impl TableInfoProvider for T
where
T: TableProvider,
{
fn schema(&self) -> SchemaRef {
self.schema()
}

fn table_type(&self) -> TableType {
self.table_type()
}

fn get_table_definition(&self) -> Option<&str> {
self.get_table_definition()
}
}
```

and then also adding a new method on the `SchemaProvider` to fetch these in bulk (for all or some specific ones):
```rust
async fn table_infos(&self, tables: Option>) -> Vec;
```

These would then allow for an option to pre-fetch these in bulk, or just default to calling `SchemaProvider::table` on each one (so effectively same as now).

### Describe alternatives you've considered

If I know that all the tables are Delta tables make a custom `information_schema.tables` builder that just returns the hard-coded table type, though this doesn't help with `columns` and `views`.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/core/src/catalog_common/information_schema.rs around lines 93-102, then inspect the SchemaProvider and TableProvider APIs used by the information_schema.tables, columns, and views builders. The issue needs a settled API design for fetching table metadata in bulk and tests showing that the metadata builders no longer require serial full table loads.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.