facebook / facebook/rocksdb

SST files written using SstFileWriter don't work with TtlDB.ingestExternalFiles

Open
#7,299 3 comments 1 reaction 1 assignee Claimed by @riversand963 View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

When `ingestExternalFiles` is called on `TtlDB` with files written using `SstFileWriter`, the call returns without issue but any subsequent queries to the DB will fail with (often inscrutable) errors. This is because `TtlDB` requires a timestamp to be appended to each row whereas SST files from `SstFileWriter` don't have this by default. When querying `TtlDB`, the `get` call attempts to strip a timestamp from the end of the fetched value and this causes two kinds of issues:
- Since the last 32 bytes of the record don't carry a timestamp, interpreting them as a timestamp causes [date validation failures](https://github.com/facebook/rocksdb/blob/fdf882ded218344c136c97daf76dfb59e4bc155f/utilities/ttl/db_ttl_impl.cc#L224)
- If timestamp validation passes by chance, downstream record deserialization breaks because the last 32 bytes of the record have been stripped away

I was wondering if it would be possible to:
- Add a mode to `SstFileWriter` that appends timestamps to records when writing
- Add a validation step to `ingestExternalFiles` for `TtlDB` that ensures that the SST files being ingested carry timestamps on each record. (This might be infeasible to check on a per record basis at ingestion time but maybe there's another way?)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.