apache / apache/hudi

Hudi configs cannot take effect during or after the table has been created

Open
#17,328 1 comment 0 reactions 1 assignee Assigned to @yihua View on GitHub
from-jira priority:critical type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

{code:java}
The following `SET` statements cannot take effect when they are put before the createTable statements or put into the options clause of the createTable statement.

# These set statements must be run before the create table statement.
SET hoodie.metadata.enable =true;
SET hoodie.metadata.index.column.stats.enable = true;
SET hoodie.enable.data.skipping = true;

CREATE DATABASE IF NOT EXISTS tpcds_hudi_1tb LOCATION
's3a://performance-benchmark-datasets-us-west-2/jenkins/benchmarks/read/tpcds_hudi_1tb';
USE tpcds_hudi_1tb;

CREATE TABLE tpcds_hudi_1tb.store_sales
USING hudi
OPTIONS (
type = 'mor',
primaryKey = 'ss_ticket_number',
precombineField = 'ss_sold_date_sk',
partitionPathComponentPrefixes = 'ss_sold_date_sk'
)
LOCATION 's3a://performance-benchmark-datasets-us-west-2/jenkins/benchmarks/read/tpcds_hudi_1tb/store_sales';

# For partition pruning
# Trigger partition pruning:
select count(1) from tpcds_hudi_1tb.store_sales where ss_sold_date_sk = 2450816;
# Without partition prunning:
select count(1) from tpcds_hudi_1tb.store_sales;

# For data skipping
# Both the following queries will skip files.
select count(1) from tpcds_hudi_1tb.store_sales where ss_customer_sk = 325;
select count(1) from tpcds_hudi_1tb.store_sales where ss_customer_sk = 100;
# To compare, we need to recreate the table, with `hoodie.enable.data.skipping` set to false.

{code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8665
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9109
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.