apache / apache/arrow-js

RecordBatchStreamWriter.write() silently drops batches on schema mismatch

Open
#388 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
21h 18m
Merged PRs (30d)
8

Description

`RecordBatchStreamWriter.write()` silently discards `RecordBatch` payloads whose schema does not match the writer's current schema. No error is thrown, no warning is logged, and `write()` returns `undefined` — identical to a successful write. This makes it very easy to lose data without any indication that something went wrong.

## Current behavior

With the default `autoDestroy: true`:

1. The first `write(batch)` call establishes the writer's schema.
2. If a subsequent `write(batch)` is called with a batch whose schema differs, the writer silently calls `this.close()` and returns — the batch is never written.
3. The caller receives no indication that data was dropped.

The relevant code path is in `src/ipc/writer.ts`:

```typescript
if (schema && !compareSchemas(schema, this._schema)) {
if (this._started && this._autoDestroy) {
return this.close(); // ← batch silently dropped, no error
}
this.reset(this._sink, schema);
}
```

Additionally, the `autoDestroy` option in `RecordBatchStreamWriterOptions` has an empty JSDoc comment, so this behavior is undocumented.

## Expected behavior

`write()` should throw an error when a batch's schema does not match the writer's schema, e.g.:

```
Error: RecordBatch schema does not match the writer's schema.
Expected: [id: Int32, name: Utf8]
Received: [x: Float64, y: Float64]
```

This is consistent with how other Arrow implementations handle this case — for example, PyArrow raises `ArrowInvalid` if you attempt to write a batch with a mismatched schema.

## Reproducer

```typescript
import {
Field, Float64, Int32, makeData,
RecordBatch, RecordBatchStreamWriter,
Schema, Struct, tableFromIPC, Utf8,
} from 'apache-arrow';

const schemaA = new Schema([
new Field('id', new Int32()),
new Field('name', new Utf8()),
]);
const schemaB = new Schema([
new Field('x', new Float64()),
new Field('y', new Float64()),
]);

// Build batch A (3 rows)
const batchA = new RecordBatch(schemaA, makeData({
type: new Struct(schemaA.fields), length: 3, nullCount: 0,
children: [
makeData({ type: new Int32(), data: new Int32Array([1, 2, 3]) }),
makeData({ type: new Utf8(), data: Buffer.from('foobarbaz'), valueOffsets: new Int32Array([0, 3, 6, 9]) }),
],
}));

// Build batch B (2 rows, different schema)
const batchB = new RecordBatch(schemaB, makeData({
type: new Struct(schemaB.fields), length: 2, nullCount: 0,
children: [
makeData({ type: new Float64(), data: new Float64Array([1.1, 2.2]) }),
makeData({ type: new Float64(), data: new Float64Array([3.3, 4.4]) }),
],
}));

const writer = new RecordBatchStreamWriter(); // autoDestroy defaults to true
writer.write(batchA); // establishes schema
writer.write(batchB); // silently dropped — no error thrown

const table = tableFromIPC(writer.toUint8Array(true));
console.log(table.numRows); // 3 — only batchA was written
console.log(table.batches.length); // 1 — batchB was silently lost
```

## Additional notes

- The `autoDestroy: false` path has different but also surprising behavior: instead of dropping the batch, it calls `reset()` which silently switches the writer to the new schema. This may be the intended behavior for the multi-stream use case (see `stream-writer-tests.ts` line 109), but it would benefit from documentation.
- All existing tests that use `autoDestroy` explicitly set it to `false`, so the default `true` path was effectively untested for schema mismatches.

Thank you,

Rusty

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.