[Feature] consumer-id in batch reading from spark
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
When storing logs into append-only bucked table and batch reading from spark,
Is there any way to prevent data removal from compaction without turning on write-only or making snapshot-retention longer?
If there isn't, like streaming `consumer-id`, making a temporary snapshot prevention is needed.
### Solution
setting a spark sql hint for storing consumer-id and retention
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by tracing Spark batch reading, compaction, and the existing streaming consumer-id behavior; done would define and implement a Spark SQL hint that preserves the required snapshots through the requested retention period.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100