apache / apache/hudi

Return num_affected_rows from sql INSERT statement

Open
#15,583 0 comments 0 reactions 0 assignees View on GitHub
area:sql from-jira priority:medium type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently when running spark sql DML, in order to check how many rows were affected, users need to get to the commit stats using hudi cli or stored procedure.

We can improve user experience by returning num_affected_rows after INSERT INTO command, so that spark sql users can easily see how many rows were inserted without the need to go to the commits itself.

num_affected_rows can be extracted in writer itself form commitMetadata

Example:
{code:java}
spark.sql("""
create table test_mor (id int, name string)
using hudi
tblproperties (primaryKey = 'id', type='mor');
""")

spark.sql(
"""
INSERT INTO test_mor
VALUES 
(1, "a"),
(2, "b"),
(3, "c"),
(4, "d"),
(5, "e"),
(6, "f"),
(7, "g")
""").show()

returns:
+-----------------+
|num_affected_rows|
+-----------------+
|                7|
+-----------------+
{code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5243
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Spark SQL INSERT INTO path into the writer and its commitMetadata, where the issue says num_affected_rows can be extracted. Use the provided test_mor INSERT example to establish the expected result. Done means INSERT INTO returns a one-row result containing num_affected_rows equal to the inserted row count.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.