apache / apache/iggy

bug(connectors): `FlatBuffer` encoder ignores schemas and silently produces incompatible payloads

Open
#3,430 0 comments 0 reactions 1 assignee Claimed by @Standing-Man View on GitHub
bug connectors rust
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

The connectors `FlatBuffer` encoding path currently reports successful encoding
but does not produce payloads compatible with the configured FlatBuffers schema.

This affects `JSON`, `Text`, `Raw`, `Proto`, and `Avro` to `FlatBuffer` encoding.

### Critical issues

#### 1. The encoder ignores the configured FlatBuffers schema
`FlatBufferEncoderConfig` exposes:

- `schema_path`
- `root_table_name`
- `include_paths`
- `preserve_unknown_fields`

However, none of these fields affect encoding.

`encode_json_to_flatbuffer()` creates an `ad-hoc` root vector/string instead of
encoding the configured root table. Therefore, external consumers generated
from the configured `.fbs` schema cannot read the produced payload.

#### 2. JSON field types and structure are lost during encoding

For JSON objects, every non-string field is serialized into a string before
being placed in the buffer.

This loses FlatBuffers field semantics for:

- integers and floating-point values
- booleans
- nested objects
- arrays
- enums, structs, tables, and unions

For example, `{"active": true, "age": 30}` is encoded using the strings
`"true"` and `"30"` rather than schema-defined boolean and integer fields.

The current buffer also stores key/value offset values inside a generic
`vector` instead of building the configured FlatBuffers table.

#### 3. ......

### Expected behavior
- The encoder must load and validate the configured `.fbs` schema.
- `JSON` fields must be encoded according to the configured root table and field
types.
- .....

### Contribution

- [x] I'm willing to submit a pull request to fix this bug

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.