apache / apache/paimon

[Bug] java api can not read data

Open
#2,252 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

paimon-bundle-0.6

### Compute Engine

java api

### Minimal reproduce step

Here is th official website example:

Identifier identifier = Identifier.create("default", "test");
Options options = new Options();
options.set("warehouse", "hdfs://hadoop100:8020/paimon/hive");
options.set("metastore", "hive");
options.set("uri", "thrift://hadoop100:9083");
options.set("hive-conf-dir", "E:\\tmp\\debzium_test\\src\\main\\resources");

CatalogContext context = CatalogContext.create(options);
Catalog catalog = CatalogFactory.createCatalog(context);

int[] projection = new int[] {0, 1};

Table table = catalog.getTable(identifier);

ReadBuilder readBuilder = table.newReadBuilder();
//.withFilter(Lists.newArrayList(notNull, predicate));

List splits = readBuilder.newScan().plan().splits();

TableRead read = readBuilder.newRead();

RecordReader reader = read.createReader(splits);

reader.forEachRemaining(System.out::println);

### What doesn't meet your expectations?

Here is my example code:

Options options = new Options();
options.set("warehouse", "hdfs://hadoop100:8020/paimon/hive");
options.set("metastore", "hive");
options.set("uri", "thrift://hadoop100:9083");
options.set("hive-conf-dir", "E:\\tmp\\debzium_test\\src\\main\\resources");

CatalogContext context = CatalogContext.create(options);
Catalog catalog = CatalogFactory.createCatalog(context);

Identifier identifier = Identifier.create("default", "test");

Table table = catalog.getTable(identifier);

ReadBuilder readBuilder = table.newReadBuilder();
//.withFilter(Lists.newArrayList(notNull, predicate));
List splits = readBuilder.newScan().plan().splits();

TableRead read = readBuilder.newRead();

RecordReader reader = read.createReader(splits);

//reader.forEachRemaining(row -> System.out.println(row));

RecordReader.RecordIterator iterator = reader.readBatch();

InternalRow next = iterator.next();
long userId = next.getLong(0);
long itemId = next.getLong(1);
BinaryString behavior = next.getString(2);
BinaryString dt = next.getString(3);
BinaryString hh = next.getString(4);
System.out.println("user_id:" + userId + " item_id:" + itemId + " behavior:" + behavior + " dt:" + dt + " hh:" + hh);

### Anything else?

no

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the Java Catalog, Table, ReadBuilder, TableRead, and RecordReader example in the report, capturing the exception or incorrect result that is currently missing. Compare the working website example with the custom readBatch and iterator path; done means the cause is identified and Java API reads return the expected table data.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.