apache / apache/hudi

External Hudi table is changed to Managed after enabling schema on read

Open
#14,292 1 comment 0 reactions 0 assignees View on GitHub
area:schema type:bug type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

Here’s a clean, GitHub-friendly version using Markdown code fences and collapsible details for better readability:

```markdown
### Verifying Schema-on-Read Behavior in Apache Hudi

#### 1. Create a Hudi table (initially without schema-on-read)

```sql
CREATE TABLE IF NOT EXISTS trips_quickstart1 (
ts BIGINT,
uuid STRING,
rider STRING,
driver STRING,
fare DOUBLE,
city STRING
) USING HUDI
PARTITIONED BY (city)
LOCATION 's3a:///trips_quickstart1'
TBLPROPERTIES (
primaryKey = 'uuid',
preCombineField = 'ts'
);
```

Initial table metadata:

```sql
DESCRIBE EXTENDED trips_quickstart1;
```

```text
+--------------------+---------+-------+
| col_name|data_type|comment|
+--------------------+---------+-------+
| _hoodie_commit_time| string| null|
|_hoodie_commit_seqno| string| null|
| _hoodie_record_key| string| null|
|_hoodie_partition...| string| null|
| _hoodie_file_name| string| null|
| ts| bigint| null|
| uuid| string| null|
| rider| string| null|
| driver| string| null|
| fare| double| null|
| city| string| null|
+--------------------+---------+-------+

Table type: EXTERNAL
Provider: hudi
Location: s3a:///trips_quickstart1
```

#### 2. Enable schema-on-read

```sql
SET hoodie.schema.on.read.enable=true;
```

```text
+-------------------------------+-----+
| key|value|
+-------------------------------+-----+
|hoodie.schema.on.read.enable | true|
+-------------------------------+-----+
```

#### 3. Re-check table metadata after enabling schema-on-read

```sql
DESCRIBE EXTENDED trips_quickstart1;
```

```text
+--------------------+---------+-------+
| col_name|data_type|comment|
+--------------------+---------+-------+
| _hoodie_commit_time| string| null|
|_hoodie_commit_seqno| string| null|
| _hoodie_record_key| string| null|
|_hoodie_partition...| string| null|
| _hoodie_file_name| string| null|
| ts| bigint| null|
| uuid| string| null|
| rider| string| null|
| driver| string| null|
| fare| double| null|
| city| string| null|
+--------------------+---------+-------+

Table type: MANAGED <-- Now reported as MANAGED when schema-on-read is enabled
Provider: hudi
```

**Key observation**: With `hoodie.schema.on.read.enable=true`, Spark reports the Hudi table as `MANAGED` instead of `EXTERNAL`. This is expected behavior in recent Hudi versions when schema evolution/promotion features are active.
```

Paste the above directly into GitHub — it renders cleanly with proper syntax highlighting and collapsible sections if needed.

### Environment

**Hudi version:** 0.15.0
**Query engine:** Spark
**Relevant configs:** `hoodie.schema.on.read.enable`

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the SQL CREATE TABLE and DESCRIBE EXTENDED sequence on Hudi 0.15.0 with Spark, first without and then with hoodie.schema.on.read.enable=true. Trace the table-type classification used by DESCRIBE EXTENDED and determine whether the external table should remain EXTERNAL; done means the behavior is clarified and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.