googleapis / googleapis/java-bigtable-hbase

Reverse timestamp issue with cell timestamps

Aperta
#2,193 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
api: bigtable type: docs
Lingua principale
Java
Stelle
184
Fork
184
Merge medio
10h
PR unite (30g)
6

Descrizione

This is a request to document some limitations we came across with Bigtable, and a question about the use of `bigtable-beam-import` to copy data from HBase to Bigtable and vice-versa.

### Documentation requests
In our HBase table we do Put operations with a reversed timestamp, i.e.,
```
Put p = new Put(row, column, qualifier, Long.MAX_VALUE - timeNowMillis, value);
table.put(p);
```
We do this to enforce a particular ordering, and this has worked fine with HBase.

When we used the `bigtable-hbase-1.x` client to do the same Put in Bigtable, the subsequent Get results all contained `Long.MAX_VALUE` timestamp. We traced it down to the `com.google.cloud.bigtable.hbase.util.TimestampConverter` which internally converts the time, and doesn't handle things well when the Put timestamp exceeds `TimestampConverter.HBASE_EFFECTIVE_MAX_TIMESTAMP`. We are exploring ways to fix this.

I think it will be useful if you document this issue in [CBT Docs](https://cloud.google.com/bigtable/docs/hbase-differences), especially considering the [HBase suggestions](https://hbase.apache.org/book.html#schema.casestudies.log_timeseries.revts) around this which others may be following.

### Questions

Will you make the `TimestampConverter.HBASE_EFFECTIVE_MAX_TIMESTAMP` public? Because one fix we are exploring is to use `HBASE_EFFECTIVE_MAX_TIMESTAMP - timeNowMillis` instead of `Long.MAX_VALUE - timeNowMillis` in our Puts; we are vary of computing it ourselves in case `TimestampConverter.FACTOR` changes.

Also, how does this work while using `bigtable-beam-import`:

a) Say, I have an HBase table with cells with timestamp in milliseconds. And I export this to sequencefiles using HBase's `Export`. When I import this sequencefile into Bigtable using `bigtable-beam-import`, will it do the `hbase2bigtable()` translation on the sequencefile cell timestamps?

b) If I export the data in Bigtable using `bigtable-beam-import` `export`, will it do the reverse translation using `bigtable2hbase()`? i.e., can I expect millisecond timestamps in the exported sequencefile or will it be microsecond timestamps?

c) How do these work if the HBase table had cells created with reverse timestamps, i.e., `Long.MAX_VALUE - timeNowMillis`?

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start by reviewing the CBT Docs HBase differences page, TimestampConverter, and the bigtable-beam-import hbase2bigtable() and bigtable2hbase() paths. Confirm how reversed and millisecond timestamps are handled for import and export, then document the limitation and answer the public-constant question.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java
Ambito
databases, documentation
Tipo di issue
Documentazione
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.