SST files written using SstFileWriter don't work with TtlDB.ingestExternalFiles
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
When `ingestExternalFiles` is called on `TtlDB` with files written using `SstFileWriter`, the call returns without issue but any subsequent queries to the DB will fail with (often inscrutable) errors. This is because `TtlDB` requires a timestamp to be appended to each row whereas SST files from `SstFileWriter` don't have this by default. When querying `TtlDB`, the `get` call attempts to strip a timestamp from the end of the fetched value and this causes two kinds of issues:
- Since the last 32 bytes of the record don't carry a timestamp, interpreting them as a timestamp causes [date validation failures](https://github.com/facebook/rocksdb/blob/fdf882ded218344c136c97daf76dfb59e4bc155f/utilities/ttl/db_ttl_impl.cc#L224)
- If timestamp validation passes by chance, downstream record deserialization breaks because the last 32 bytes of the record have been stripped away
I was wondering if it would be possible to:
- Add a mode to `SstFileWriter` that appends timestamps to records when writing
- Add a validation step to `ingestExternalFiles` for `TtlDB` that ensures that the SST files being ingested carry timestamps on each record. (This might be infeasible to check on a per record basis at ingestion time but maybe there's another way?)
Contributor guide
Assessment
This issue has not been assessed yet.