apache / apache/hop

[Task]: Problem with transaction pipeline

Open
#4,346 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage Database P3 Pipelines
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### What needs to happen?

![ApachHOP_01](https://github.com/user-attachments/assets/80e56a96-a590-4086-8ace-695259dcd86d)
Hi,

In the example on the screenshot, the insertion into the database is performed one insert per row.

INSERT INTO "BI2_ACCRUAL_CUBE_6860" ("ITEM_ID", "ETOF_ID", "ISD_ID", "CARRIER_ID", ... "COST_USD")
VALUES ( '111111111111'::Integer, '111111111111'::Integer, '111111111111'::Integer, .... '1.11111111111111111111')
...........
INSERT INTO "BI2_ACCRUAL_CUBE_6860" ("ITEM_ID", "ETOF_ID", "ISD_ID", "CARRIER_ID", ... "COST_USD")
VALUES ( '2222222222222'::Integer, '2222222222222'::Integer, '2222222222222222'::Integer, .... '2.222222222222222')
and so on 35 thousand times in our example.

The transaction pipeline ignores batch insertion. And this is correct, since there should be one commit at the end.
But for the Vertika DB, inserting each row as a separate insert leads to the creation of a huge number of files and an error: "to many ROS containers". What makes it impossible to use transaction pipeline with Vertica DB.
Can you suggest any solution that will allow transaction pipelines to work normally with the Vertica database?

### Issue Priority

Priority: 3

### Issue Component

Component: Database, Component: Pipelines

Contributor guide

Open the contributing guide

Research direction

No file or test is named in the issue. Start by reproducing the 35,000-row transaction pipeline against Vertica and trace how database inserts are generated; done means a transaction pipeline can work with Vertica without creating too many ROS containers while retaining the final commit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.