Return num_affected_rows from sql INSERT statement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently when running spark sql DML, in order to check how many rows were affected, users need to get to the commit stats using hudi cli or stored procedure.
We can improve user experience by returning num_affected_rows after INSERT INTO command, so that spark sql users can easily see how many rows were inserted without the need to go to the commits itself.
num_affected_rows can be extracted in writer itself form commitMetadata
Example:
{code:java}
spark.sql("""
create table test_mor (id int, name string)
using hudi
tblproperties (primaryKey = 'id', type='mor');
""")
spark.sql(
"""
INSERT INTO test_mor
VALUES
(1, "a"),
(2, "b"),
(3, "c"),
(4, "d"),
(5, "e"),
(6, "f"),
(7, "g")
""").show()
returns:
+-----------------+
|num_affected_rows|
+-----------------+
| 7|
+-----------------+
{code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5243
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Spark SQL INSERT INTO path into the writer and its commitMetadata, where the issue says num_affected_rows can be extracted. Use the provided test_mor INSERT example to establish the expected result. Done means INSERT INTO returns a one-row result containing num_affected_rows equal to the inserted row count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100