Milvus Source Connector: Critical Data Type & Embedding Edge Cases

Following are the vital boundary edge cases which need to be covered in the source connector.

1. BinaryVector uses bits, code treated it as bytes

BinaryVector dimension is measured in bits. The BinaryVector and Int8Vector had been sharing one branch, using dim directly as the byte count. So for dim=128, it tried to split data into 128-byte chunks and was way too big. Most rows produced nothing. The fix splits them apart, converting bits to bytes for BinaryVector only.

2. SparseFloatVector merged all rows into one JSON blob

SparseFloatArray.contents is a list: one binary entry per row. It passed the whole array into a single JSON builder, so all rows got merged together. Two rows with became one and the first value was overwritten. The fix iterates contents and builds one JSON per row.

3. Batch delete only kept the first primary key

A DeleteRequest with 100 PKs was stuffed into one Delete event as a List. The emitter then called list.get(0) and only the first PK ever turned into a tombstone record. The other 99 were silently dropped. The fix will do one event per PK.

4. Null bitmap was completely ignored

Milvus 2.5+ sends a valid_data boolean array alongside nullable fields. false means null. It never read it, so nulls came through as zero values 0 for ints, "" for strings, indistinguishable from real data. The fix reads the bitmap and replaces those positions with actual nulls.