apache / apache/tsfile

[Feature] Support appending to a normally closed TsFile in C++ and C APIs

Open
#923 2 comments 0 reactions 1 assignee Claimed by @Adarsh-Me View on GitHub
C c++ feature help wanted
Dominant language
Java
Stars
203
Forks
104
Avg merge
2d 6h
Merged PRs (30d)
33

Description

### Motivation

The Java implementation can reopen a normally closed (complete) TsFile for appending through:

`RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File)`

This method detects a complete file, locates the start of `FileMetadata` from the footer, truncates the separator marker and all tail metadata, recovers the existing schema/chunk metadata, and then allows new data to be appended. Closing the writer generates a new metadata section and tail magic.

The C++ implementation currently supports recovery and continued writing only when the file is incomplete or its tail is damaged. When the tail magic is valid, `RestorableTsFileIOWriter::self_check()` treats the file as complete, sets `can_write_ = false`, and closes the write handle.

As a result, users of the C++ API and the C wrapper cannot append to a TsFile that was closed normally, even though the equivalent workflow is available in Java.

### Expected behavior

Provide a supported C++ API equivalent to the Java completed-file append workflow, and expose it through the C API as well.

The workflow should:

1. Validate that the input is a compatible TsFile.
2. Detect whether the file is complete by checking the tail magic.
3. For a complete file, read the footer metadata size and locate the start of `FileMetadata`.
4. Safely truncate the separator marker, metadata/index section, footer size, and tail magic while retaining all completed chunk groups.
5. Recover the existing tree/table schema, alignment information, chunk metadata, statistics, and per-device last timestamps.
6. Allow users to append data using the recovered schema.
7. Allow registration of supported new schemas after recovery, with clear behavior for:
- a new device or measurement in the tree model;
- a new table;
- attempts to replace or evolve an existing table/aligned schema.
8. On close, write a new metadata section, footer, and tail magic so that the resulting file is complete and readable.
9. Reject out-of-order timestamps relative to recovered data.

The operation should preserve the original file when validation or recovery fails before truncation. The API documentation should make the destructive/in-place nature of a successful append-open explicit.

### Current C++ behavior

For an incomplete or damaged-tail file, the following path already works:

`RestorableTsFileIOWriter::open(path, true)`
→ recover schema and chunk metadata
→ `TsFileWriter::init(&restorable_writer)`
→ append data
→ close and rebuild metadata

For a normally closed file, `open()` returns successfully but `can_write()` is false, so it cannot be passed to `TsFileWriter::init()`.

### Suggested tests

- Append to a normally closed tree-model TsFile and verify old and new rows.
- Append to a normally closed table-model TsFile.
- Preserve recovered encoding/compression/data types and alignment.
- Register a new device and a new non-aligned measurement after recovery.
- Verify documented behavior for aligned-series and existing-table schema evolution.
- Reject duplicate or incompatible schemas.
- Reject timestamps not newer than the recovered last timestamp.
- Handle empty, incomplete, incompatible, encrypted, and large-metadata files.
- Verify that a no-op append followed by close produces a valid file without losing data.
- Add coverage for the C wrapper API.

### Related implementation

- Java: `RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File)`
- C++: `RestorableTsFileIOWriter::self_check()`
- C++ recovery integration: `TsFileWriter::init(RestorableTsFileIOWriter*)`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.