[Bug]: Table output - bad performance in transaction-based pipelines
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### Apache Hop version?
2.10.0
### Java version?
17.0.2
### Operating system
Linux
### What happened?
Following discussion #4678, here's a use case:
1. Set a transform that loads some thousands of rows (i.e. with a Table input from a database, or with a Text file input from a text file)
2. Link this transform to a Table output, and set it to truncate the table (in any case, not just if a row comes in)
Start the pipeline in the default local run configuration, without transactions enabled: the execution time is somehow reasonable.
Start the pipeline [in a run configuration that includes transactions](https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/native-local-pipeline-engine.html): the pipeline works as expected, and it's correctly rollbacked in case of errors, but the execution times worsen a lot (even 60 times compared to the previous case).
For a data sample, you can try to download the [GeoLite2 City CSV Database from Maxmind](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data/) (you need to signup to Maxmind, the GeoLite2 database is free of charge).
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
Contributor guide
Research direction
Reproduce the slowdown with a Table input or Text file input feeding a Table output configured to truncate, comparing the default local run configuration with one that includes transactions. Start by tracing the Table output and transaction-enabled pipeline execution paths; use the GeoLite2 City CSV sample if available. Done means transaction-enabled execution no longer has the reported severe slowdown while rollback behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, linux
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100