googleapis / googleapis/java-bigtable-hbase

Scan filter set in dataflow to read bigtable not honoured during execution

Open
#4,010 0 comments 0 reactions 0 assignees View on GitHub
api: bigtable
Dominant language
Java
Stars
184
Forks
184
Avg merge
10h
Merged PRs (30d)
6

Description

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

- Search the issues already opened: https://github.com/googleapis/java-bigtable-hbase/issues
- Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform

If you are still having issues, please include as much information as possible:

#### Environment details

1. Specify the API at the beginning of the title. General, Core, and Other are also allowed as types
2. OS type and version: Unix
3. Java version: 8
4. Version(s): bigtable-hbase-beam:2.6.5

#### Steps to reproduce

1. Create a dataflow which scans the bigtable to read rows. Add a family filter criteria to scan only specific families of the table.
2. Execute the dataflow and check the column of each cell read.
3. The dataflow will read from other column families as well apart from the column family filters set.
4. This is not just true for family filter but rather every filter except the start and stop row filters

#### Code example

```
Scan scan = new Scan();
scan.addFamily(StringHelper.utf8Bytes("i"));
scan.addFamily(StringHelper.utf8Bytes("j"));
scan.setCacheBlocks(false);
String startRowKey = getStartRowKey();
if (!startRowKey.isEmpty()) {
scan.setStartRow(StringHelper.utf8Bytes(startRowKey));
}

String stopRowKey = getStopRowKey();
if (!stopRowKey.isEmpty()) {
scan.setStopRow(StringHelper.utf8Bytes(stopRowKey));
}
CloudBigtableScanConfiguration config = new CloudBigtableScanConfiguration.Builder()
.withInstanceId(options.getBigtableInstanceId()).withProjectId(options.getBigtableProjectId()).withTableId(options.getBigtableTableId()).withScan(scan).build();

Pipeline p = Pipeline.create(options);
p.apply(Read.from(CloudBigtableIO.read(config))).apply(ParDo.of(new KeyedByMaster(getConfigurationForTable(options)))).apply(TextIO.write().to(options.getOutputLocation()));
DataflowOptions options = DataFlowOptionsCreator.create(dumpName, args);

CloudBigtableScanConfiguration config = new CloudBigtableScanConfiguration.Builder().withInstanceId(options.getBigtableInstanceId())
p.run();
```

Only the startRow and stopRow from the configuration is copied over to the Scan object created.
#### Stack trace
```
void initializeScanner() throws IOException {
Configuration config = source.getConfiguration().toHBaseConfig();
Scan scan =
new Scan()
.withStartRow(source.getConfiguration().getZeroCopyStartRow())
.withStopRow(source.getConfiguration().getZeroCopyStopRow())
.setMaxVersions(Integer.MAX_VALUE);
scanner =
connection
.getTable(TableName.valueOf(source.getConfiguration().getTableId()))
.getScanner(scan);
}

connection = ConnectionFactory.createConnection(config);
```

#### External references such as API reference guides

- ?

#### Any additional information below
This is a bug introduced with the bigtable-hbase-beam 2.x upgrade and is breaking a lot of dataflows while upgrading from bigtable-hbase-beam 1.x to bigtable-hbase-beam 2.x

Following these steps guarantees the quickest resolution possible.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.