apache / apache/beam

HCatalogTableProvider supports filter pushdown

Open
#19,499 0 comments 0 reactions 0 assignees View on GitHub
bug dsl hcatalog io java P3 sql
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Hi,

I'm using HCatalogTableProvider while doing SqlTransform.query. The query is something like "select * from `hive`.`table_name` limit 10". Despite of the limit clause, the data source still reads much more rows (the data of Hive table are files on S3), even more than the number of rows in one file (or partition).

 

Some more details:
- It is running on Flink.
- I actually implemented my own HiveTableProvider because HCatalogBeamSchema only supports primitive types. However, the table provider works when I query a small table with ~1k rows.

Imported from Jira [BEAM-6874](https://issues.apache.org/jira/browse/BEAM-6874). Original Jira may contain additional context.
Reported by: Near.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.