[Feature] Support appending to a normally closed TsFile in C++ and C APIs
- Dominant language
- Java
- Stars
- 203
- Forks
- 104
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 33
Description
### Motivation
The Java implementation can reopen a normally closed (complete) TsFile for appending through:
`RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File)`
This method detects a complete file, locates the start of `FileMetadata` from the footer, truncates the separator marker and all tail metadata, recovers the existing schema/chunk metadata, and then allows new data to be appended. Closing the writer generates a new metadata section and tail magic.
The C++ implementation currently supports recovery and continued writing only when the file is incomplete or its tail is damaged. When the tail magic is valid, `RestorableTsFileIOWriter::self_check()` treats the file as complete, sets `can_write_ = false`, and closes the write handle.
As a result, users of the C++ API and the C wrapper cannot append to a TsFile that was closed normally, even though the equivalent workflow is available in Java.
### Expected behavior
Provide a supported C++ API equivalent to the Java completed-file append workflow, and expose it through the C API as well.
The workflow should:
1. Validate that the input is a compatible TsFile.
2. Detect whether the file is complete by checking the tail magic.
3. For a complete file, read the footer metadata size and locate the start of `FileMetadata`.
4. Safely truncate the separator marker, metadata/index section, footer size, and tail magic while retaining all completed chunk groups.
5. Recover the existing tree/table schema, alignment information, chunk metadata, statistics, and per-device last timestamps.
6. Allow users to append data using the recovered schema.
7. Allow registration of supported new schemas after recovery, with clear behavior for:
- a new device or measurement in the tree model;
- a new table;
- attempts to replace or evolve an existing table/aligned schema.
8. On close, write a new metadata section, footer, and tail magic so that the resulting file is complete and readable.
9. Reject out-of-order timestamps relative to recovered data.
The operation should preserve the original file when validation or recovery fails before truncation. The API documentation should make the destructive/in-place nature of a successful append-open explicit.
### Current C++ behavior
For an incomplete or damaged-tail file, the following path already works:
`RestorableTsFileIOWriter::open(path, true)`
→ recover schema and chunk metadata
→ `TsFileWriter::init(&restorable_writer)`
→ append data
→ close and rebuild metadata
For a normally closed file, `open()` returns successfully but `can_write()` is false, so it cannot be passed to `TsFileWriter::init()`.
### Suggested tests
- Append to a normally closed tree-model TsFile and verify old and new rows.
- Append to a normally closed table-model TsFile.
- Preserve recovered encoding/compression/data types and alignment.
- Register a new device and a new non-aligned measurement after recovery.
- Verify documented behavior for aligned-series and existing-table schema evolution.
- Reject duplicate or incompatible schemas.
- Reject timestamps not newer than the recovered last timestamp.
- Handle empty, incomplete, incompatible, encrypted, and large-metadata files.
- Verify that a no-op append followed by close produces a valid file without losing data.
- Add coverage for the C wrapper API.
### Related implementation
- Java: `RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File)`
- C++: `RestorableTsFileIOWriter::self_check()`
- C++ recovery integration: `TsFileWriter::init(RestorableTsFileIOWriter*)`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.