googleapis / googleapis/java-bigtable-hbase
Scan filter set in dataflow to read bigtable not honoured during execution
- Lingua principale
- Java
- Stelle
- 184
- Fork
- 184
- Merge medio
- 10h
- PR unite (30g)
- 6
Descrizione
Thanks for stopping by to let us know something could be better!
**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":
- Search the issues already opened: https://github.com/googleapis/java-bigtable-hbase/issues
- Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform
If you are still having issues, please include as much information as possible:
#### Environment details
1. Specify the API at the beginning of the title. General, Core, and Other are also allowed as types
2. OS type and version: Unix
3. Java version: 8
4. Version(s): bigtable-hbase-beam:2.6.5
#### Steps to reproduce
1. Create a dataflow which scans the bigtable to read rows. Add a family filter criteria to scan only specific families of the table.
2. Execute the dataflow and check the column of each cell read.
3. The dataflow will read from other column families as well apart from the column family filters set.
4. This is not just true for family filter but rather every filter except the start and stop row filters
#### Code example
```
Scan scan = new Scan();
scan.addFamily(StringHelper.utf8Bytes("i"));
scan.addFamily(StringHelper.utf8Bytes("j"));
scan.setCacheBlocks(false);
String startRowKey = getStartRowKey();
if (!startRowKey.isEmpty()) {
scan.setStartRow(StringHelper.utf8Bytes(startRowKey));
}
String stopRowKey = getStopRowKey();
if (!stopRowKey.isEmpty()) {
scan.setStopRow(StringHelper.utf8Bytes(stopRowKey));
}
CloudBigtableScanConfiguration config = new CloudBigtableScanConfiguration.Builder()
.withInstanceId(options.getBigtableInstanceId()).withProjectId(options.getBigtableProjectId()).withTableId(options.getBigtableTableId()).withScan(scan).build();
Pipeline p = Pipeline.create(options);
p.apply(Read.from(CloudBigtableIO.read(config))).apply(ParDo.of(new KeyedByMaster(getConfigurationForTable(options)))).apply(TextIO.write().to(options.getOutputLocation()));
DataflowOptions options = DataFlowOptionsCreator.create(dumpName, args);
CloudBigtableScanConfiguration config = new CloudBigtableScanConfiguration.Builder().withInstanceId(options.getBigtableInstanceId())
p.run();
```
Only the startRow and stopRow from the configuration is copied over to the Scan object created.
#### Stack trace
```
void initializeScanner() throws IOException {
Configuration config = source.getConfiguration().toHBaseConfig();
Scan scan =
new Scan()
.withStartRow(source.getConfiguration().getZeroCopyStartRow())
.withStopRow(source.getConfiguration().getZeroCopyStopRow())
.setMaxVersions(Integer.MAX_VALUE);
scanner =
connection
.getTable(TableName.valueOf(source.getConfiguration().getTableId()))
.getScanner(scan);
}
connection = ConnectionFactory.createConnection(config);
```
#### External references such as API reference guides
- ?
#### Any additional information below
This is a bug introduced with the bigtable-hbase-beam 2.x upgrade and is breaking a lot of dataflows while upgrading from bigtable-hbase-beam 1.x to bigtable-hbase-beam 2.x
Following these steps guarantees the quickest resolution possible.
Thanks!
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia tracciando initializeScanner() e il modo in cui CloudBigtableScanConfiguration viene convertito nello Scan utilizzato dal dataflow. Verifica che i filtri oltre a startRow e stopRow vengano conservati, quindi riproduci l’esempio e conferma che l’esecuzione legga solo le famiglie di colonne selezionate.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- java
- Ambito
- data-engineering, databases
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100