apache / apache/hop

[Bug]: Table output - bad performance in transaction-based pipelines

Open
#4,680 1 comment 0 reactions 0 assignees View on GitHub
awaiting triage P2 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### Apache Hop version?

2.10.0

### Java version?

17.0.2

### Operating system

Linux

### What happened?

Following discussion #4678, here's a use case:
1. Set a transform that loads some thousands of rows (i.e. with a Table input from a database, or with a Text file input from a text file)
2. Link this transform to a Table output, and set it to truncate the table (in any case, not just if a row comes in)

Start the pipeline in the default local run configuration, without transactions enabled: the execution time is somehow reasonable.

Start the pipeline [in a run configuration that includes transactions](https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/native-local-pipeline-engine.html): the pipeline works as expected, and it's correctly rollbacked in case of errors, but the execution times worsen a lot (even 60 times compared to the previous case).

For a data sample, you can try to download the [GeoLite2 City CSV Database from Maxmind](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data/) (you need to signup to Maxmind, the GeoLite2 database is free of charge).

### Issue Priority

Priority: 2

### Issue Component

Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Reproduce the slowdown with a Table input or Text file input feeding a Table output configured to truncate, comparing the default local run configuration with one that includes transactions. Start by tracing the Table output and transaction-enabled pipeline execution paths; use the GeoLite2 City CSV sample if available. Done means transaction-enabled execution no longer has the reported severe slowdown while rollback behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, linux
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.