bug(connectors): `FlatBuffer` encoder ignores schemas and silently produces incompatible payloads
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
### Description
The connectors `FlatBuffer` encoding path currently reports successful encoding
but does not produce payloads compatible with the configured FlatBuffers schema.
This affects `JSON`, `Text`, `Raw`, `Proto`, and `Avro` to `FlatBuffer` encoding.
### Critical issues
#### 1. The encoder ignores the configured FlatBuffers schema
`FlatBufferEncoderConfig` exposes:
- `schema_path`
- `root_table_name`
- `include_paths`
- `preserve_unknown_fields`
However, none of these fields affect encoding.
`encode_json_to_flatbuffer()` creates an `ad-hoc` root vector/string instead of
encoding the configured root table. Therefore, external consumers generated
from the configured `.fbs` schema cannot read the produced payload.
#### 2. JSON field types and structure are lost during encoding
For JSON objects, every non-string field is serialized into a string before
being placed in the buffer.
This loses FlatBuffers field semantics for:
- integers and floating-point values
- booleans
- nested objects
- arrays
- enums, structs, tables, and unions
For example, `{"active": true, "age": 30}` is encoded using the strings
`"true"` and `"30"` rather than schema-defined boolean and integer fields.
The current buffer also stores key/value offset values inside a generic
`vector` instead of building the configured FlatBuffers table.
#### 3. ......
### Expected behavior
- The encoder must load and validate the configured `.fbs` schema.
- `JSON` fields must be encoded according to the configured root table and field
types.
- .....
### Contribution
- [x] I'm willing to submit a pull request to fix this bug
Contributor guide
Assessment
This issue has not been assessed yet.