apache / apache/paimon

[Bug] Failed to execute delete with an append table by a timestamp condition statement, error "failed to parse timestamp datatype value"

Open
#4,364 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.8.2

### Compute Engine

spark 3.4.1

### Minimal reproduce step

1. create a append table in spark-sql
CREATE TABLE paimon.events.event_test (
id varchar(100),
time timestamp,
created_at timestamp,
new_session varchar(600),
source_tmp varchar(6000),
time_dt date
) partitioned by (time);

2. insert some data to the table in spark-sql
insert into paimon.events.event_test values('111', cast('2024-10-01 01:00:00' as timestamp), cast('2024-10-01 01:00:00' as timestamp), '111', '111', cast('2024-10-01' as date));
insert into paimon.events.event_test values('111', cast('2024-10-01 06:00:00' as timestamp), cast('2024-10-02 06:00:00' as timestamp), '111', '111', cast('2024-10-01' as date));
insert into paimon.events.event_test values('111', cast('2024-10-01 12:00:00' as timestamp), cast('2024-10-01 12:00:00' as timestamp), '111', '111', cast('2024-10-01' as date));
insert into paimon.events.event_test values('111', cast('2024-10-01 18:00:00' as timestamp), cast('2024-10-01 18:00:00' as timestamp), '111', '111', cast('2024-10-01' as date));
insert into paimon.events.event_test values('111', cast('2024-10-02 01:00:00' as timestamp), cast('2024-10-02 01:00:00' as timestamp), '111', '111', cast('2024-10-02' as date));
insert into paimon.events.event_test values('111', cast('2024-10-02 06:00:00' as timestamp), cast('2024-10-02 06:00:00' as timestamp), '111', '111', cast('2024-10-02' as date));
insert into paimon.events.event_test values('111', cast('2024-10-02 12:00:00' as timestamp), cast('2024-10-02 12:00:00' as timestamp), '111', '111', cast('2024-10-02' as date));
insert into paimon.events.event_test values('111', cast('2024-10-02 18:00:00' as timestamp), cast('2024-10-02 18:00:00' as timestamp), '111', '111', cast('2024-10-02' as date));
insert into paimon.events.event_test values('111', cast('2024-10-03 01:00:00' as timestamp), cast('2024-10-03 01:00:00' as timestamp), '111', '111', cast('2024-10-03' as date));
insert into paimon.events.event_test values('111', cast('2024-10-03 01:00:01' as timestamp), cast('2024-10-03 01:00:01' as timestamp), '111', '111', cast('2024-10-03' as date));

3. execute select in spark-sql, it would execute successfully
select * from paimon.events.event_test where time = '2024-10-03 01:00:01';

4. execute delete in spark-sql, it would raise an error
delete from paimon.events.event_test where time = '2024-10-03 01:00:01';

![img_v3_02fu_9beaba0f-caaa-4d2a-a12d-0bb91727628g](https://github.com/user-attachments/assets/06a0b13e-fa52-461c-9d56-ce7a106eeec0)

### What doesn't meet your expectations?

it could execute select successfully, and it failed to execute delete with the same condition statement.

### Anything else?

I made a test with a primary table, both select and delete with timestamp datatype condition could be executed successfully.

I also made a test with a append table, both select and delete with date datatype condition could be executed successfully.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure in Spark SQL using the append-table schema and timestamp inserts from the issue, then compare the successful SELECT with the failing DELETE path. Done means DELETE accepts the same timestamp condition on an append table without the parse-timestamp error, while the existing primary-table and date-condition cases continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.