googleapis / googleapis/java-bigtable-hbase

Scan filter set in dataflow to read bigtable not honoured during execution

Aperta
#4,010 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
api: bigtable
Lingua principale
Java
Stelle
184
Fork
184
Merge medio
10h
PR unite (30g)
6

Descrizione

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

- Search the issues already opened: https://github.com/googleapis/java-bigtable-hbase/issues
- Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform

If you are still having issues, please include as much information as possible:

#### Environment details

1. Specify the API at the beginning of the title. General, Core, and Other are also allowed as types
2. OS type and version: Unix
3. Java version: 8
4. Version(s): bigtable-hbase-beam:2.6.5

#### Steps to reproduce

1. Create a dataflow which scans the bigtable to read rows. Add a family filter criteria to scan only specific families of the table.
2. Execute the dataflow and check the column of each cell read.
3. The dataflow will read from other column families as well apart from the column family filters set.
4. This is not just true for family filter but rather every filter except the start and stop row filters

#### Code example

```
Scan scan = new Scan();
scan.addFamily(StringHelper.utf8Bytes("i"));
scan.addFamily(StringHelper.utf8Bytes("j"));
scan.setCacheBlocks(false);
String startRowKey = getStartRowKey();
if (!startRowKey.isEmpty()) {
scan.setStartRow(StringHelper.utf8Bytes(startRowKey));
}

String stopRowKey = getStopRowKey();
if (!stopRowKey.isEmpty()) {
scan.setStopRow(StringHelper.utf8Bytes(stopRowKey));
}
CloudBigtableScanConfiguration config = new CloudBigtableScanConfiguration.Builder()
.withInstanceId(options.getBigtableInstanceId()).withProjectId(options.getBigtableProjectId()).withTableId(options.getBigtableTableId()).withScan(scan).build();

Pipeline p = Pipeline.create(options);
p.apply(Read.from(CloudBigtableIO.read(config))).apply(ParDo.of(new KeyedByMaster(getConfigurationForTable(options)))).apply(TextIO.write().to(options.getOutputLocation()));
DataflowOptions options = DataFlowOptionsCreator.create(dumpName, args);

CloudBigtableScanConfiguration config = new CloudBigtableScanConfiguration.Builder().withInstanceId(options.getBigtableInstanceId())
p.run();
```

Only the startRow and stopRow from the configuration is copied over to the Scan object created.
#### Stack trace
```
void initializeScanner() throws IOException {
Configuration config = source.getConfiguration().toHBaseConfig();
Scan scan =
new Scan()
.withStartRow(source.getConfiguration().getZeroCopyStartRow())
.withStopRow(source.getConfiguration().getZeroCopyStopRow())
.setMaxVersions(Integer.MAX_VALUE);
scanner =
connection
.getTable(TableName.valueOf(source.getConfiguration().getTableId()))
.getScanner(scan);
}

connection = ConnectionFactory.createConnection(config);
```

#### External references such as API reference guides

- ?

#### Any additional information below
This is a bug introduced with the bigtable-hbase-beam 2.x upgrade and is breaking a lot of dataflows while upgrading from bigtable-hbase-beam 1.x to bigtable-hbase-beam 2.x

Following these steps guarantees the quickest resolution possible.

Thanks!

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia tracciando initializeScanner() e il modo in cui CloudBigtableScanConfiguration viene convertito nello Scan utilizzato dal dataflow. Verifica che i filtri oltre a startRow e stopRow vengano conservati, quindi riproduci l’esempio e conferma che l’esecuzione legga solo le famiglie di colonne selezionate.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
data-engineering, databases
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.