apache / apache/beam

Add Apache Ignite IO

Open
#18,522 0 comments 0 reactions 0 assignees View on GitHub
ideas io new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

Hi all,
{color:#14892c}I tried to connect Apache Ignite(In-memory) via the beam's sdk:org.apache.beam.sdk.io.jdbc.JdbcIO
Here, i am not sure if the JdbcIO sdk only is provided for some specific Database: MySQL(disk), postgreSQL(disk)?{color}
my java test code is as follows:
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;

import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.jdbc.JdbcIO;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;

public class BeamtoJDBC {
public static void main(String[] args) {
Integer[] value=new Integer[] {1,2,3,4,5};
List\> dataList = new ArrayList<\>();
int n=value.length;
int count=0;
for (int i=0; i\> data=p.apply("create data with time",Create.of(dataList));

data.apply(JdbcIO.\>write()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
.create("org.apache.ignite.IgniteJdbcDriver", "jdbc:ignite://localhost:11211/")
)
.withPreparedStatementSetter(new JdbcIO.PreparedStatementSetter\>() {
public void setParameters(KV element, PreparedStatement query)
throws SQLException {
query.setInt(1, element.getKey());
query.setInt(2, element.getValue());
}
})
);
p.run();

}
}

{color:#d04437}my error message is:
" InvocationTargetException: org.apache.beam.sdk.util.UserCodeException: java.sql.SQLException: Cannot create PoolableConnectionFactory
(Failed to establish connection.): Failed to get future result due to waiting timed out. "{color}

{color:#14892c}I would like to know whether the connection between beam and ignite is feasible or not?{color}

Thanks

Rick

Imported from Jira [BEAM-3073](https://issues.apache.org/jira/browse/BEAM-3073). Original Jira may contain additional context.
Reported by: Ricklin.

Contributor guide

Open the contributing guide

Research direction

Start with the provided Java example and the JdbcIO DataSourceConfiguration connection setup, then investigate the reported connection timeout with Apache Ignite's JDBC driver and URL. Done would require a clear project-supported path for connecting Beam's JdbcIO to Ignite, but the issue does not name Beam files or tests to update.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.