googleapis / googleapis/java-bigtable-hbase

Reverse timestamp issue with cell timestamps

Open
#2,193 3 comments 0 reactions 0 assignees View on GitHub
api: bigtable type: docs
Dominant language
Java
Stars
184
Forks
184
Avg merge
10h
Merged PRs (30d)
6

Description

This is a request to document some limitations we came across with Bigtable, and a question about the use of `bigtable-beam-import` to copy data from HBase to Bigtable and vice-versa.

### Documentation requests
In our HBase table we do Put operations with a reversed timestamp, i.e.,
```
Put p = new Put(row, column, qualifier, Long.MAX_VALUE - timeNowMillis, value);
table.put(p);
```
We do this to enforce a particular ordering, and this has worked fine with HBase.

When we used the `bigtable-hbase-1.x` client to do the same Put in Bigtable, the subsequent Get results all contained `Long.MAX_VALUE` timestamp. We traced it down to the `com.google.cloud.bigtable.hbase.util.TimestampConverter` which internally converts the time, and doesn't handle things well when the Put timestamp exceeds `TimestampConverter.HBASE_EFFECTIVE_MAX_TIMESTAMP`. We are exploring ways to fix this.

I think it will be useful if you document this issue in [CBT Docs](https://cloud.google.com/bigtable/docs/hbase-differences), especially considering the [HBase suggestions](https://hbase.apache.org/book.html#schema.casestudies.log_timeseries.revts) around this which others may be following.

### Questions

Will you make the `TimestampConverter.HBASE_EFFECTIVE_MAX_TIMESTAMP` public? Because one fix we are exploring is to use `HBASE_EFFECTIVE_MAX_TIMESTAMP - timeNowMillis` instead of `Long.MAX_VALUE - timeNowMillis` in our Puts; we are vary of computing it ourselves in case `TimestampConverter.FACTOR` changes.

Also, how does this work while using `bigtable-beam-import`:

a) Say, I have an HBase table with cells with timestamp in milliseconds. And I export this to sequencefiles using HBase's `Export`. When I import this sequencefile into Bigtable using `bigtable-beam-import`, will it do the `hbase2bigtable()` translation on the sequencefile cell timestamps?

b) If I export the data in Bigtable using `bigtable-beam-import` `export`, will it do the reverse translation using `bigtable2hbase()`? i.e., can I expect millisecond timestamps in the exported sequencefile or will it be microsecond timestamps?

c) How do these work if the HBase table had cells created with reverse timestamps, i.e., `Long.MAX_VALUE - timeNowMillis`?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.