Vector `file` source preserves BOM as U+FEFF char and CRLF trailing CR in EVERY line

Vector version: 0.58.0

Severity: Major (silent data corruption in every line, not just header)

Description:

Vector file source uses line_delimiter (default " ") but does not strip UTF-8 BOM or trailing CR from any line. The bug affects EVERY line in the file, not just the first.

Source code (confirmed):
lib/file-source/src/file_watcher/mod.rs:247:

match read_until_with_max_size(
    reader.as_mut(),
    file_position,
    self.line_delimiter.as_ref(),
    &mut self.buf,
    self.max_line_bytes,
).await

The line_delimiter is passed verbatim. No BOM stripping. No CR stripping. The full bytes between line boundaries (or file boundary, for the first line) become the raw_line.

Reproduction (Vector 0.58.0, file with BOM + 5 CRLF lines):

File bytes:

EF BB BF line0_col1<TAB>line0_col2<CR><LF>
          line1_col1<TAB>line1_col2<CR><LF>
          line2_col1<TAB>line2_col2<CR><LF>
          line3_col1<TAB>line3_col2<CR><LF>
          line4_col1<TAB>line4_col2<CR><LF>

Vector output (5 events):

• Line 0: msg length 25, starts with U+FEFF (BOM), ends with CR
• Line 1: msg length 22, no BOM, ends with CR
• Line 2: msg length 22, no BOM, ends with CR
• Line 3: msg length 22, no BOM, ends with CR
• Line 4: msg length 22, no BOM, ends with CR

Observations:

1. The first line carries a 3-byte UTF-8 BOM (U+FEFF, displays as zero-width space) which becomes a literal character in VRL strings.
2. EVERY line (not just the first) carries a trailing \r after the data.
3. If downstream parsing splits on whitespace or uses regex like [^\t]+$, the BOM-prefixed first column fails to match.

Verification:

# Create test file with BOM + CRLF:
printf '\xEF\xBB\xBFline1\tcol2\r\ndata1\tcol2\r\n' > /tmp/bom-test.tsv

# Run Vector with simple remap:
cat > /tmp/cfg.yaml << 'EOF'
sources:
  src:
    type: file
    include: [/tmp/bom-test.tsv]
    data_dir: /tmp/vdata
    read_from: beginning
    ignore_checkpoints: true
transforms:
  parse:
    type: remap
    inputs: [src]
    source: |
      .msg = string!(.message)
      .has_bom = starts_with(.msg, "\u{FEFF}")
sinks:
  out:
    type: file
    inputs: [parse]
    path: /tmp/out.jsonl
    encoding: { codec: json }
EOF
vector --config /tmp/cfg.yaml
# Expected output: .has_bom is true for line 1, false for line 2

Impact:

1. Every line of a CRLF file carries invisible CR at the end. Pipelines that use regex like [^\t]*$ fail to match the last column.
2. The first line carries 3 invisible BOM bytes. Downstream parsers (JSON Lines, CSV readers) see corrupted first column.
3. Metrics and labels that use VRL string operations contain \r and \u{FEFF} characters.
4. Common in mixed-OS deployments where upstream Linux files are sent through Windows tools (Excel, Notepad) before reaching Vector.

Expected behavior:

Default behavior should be configurable. Two reasonable options:

• Strip BOM from the first line, strip trailing CR from all lines (if line_delimiter is \n)
• Add explicit config flags strip_bom: true and strip_cr: true

Workaround:

In every remap transform, manually strip:

raw_line = string!(.message)
if starts_with(raw_line, "\u{FEFF}") {
  raw_line = slice!(raw_line, 3)  # 3-byte BOM
}
raw_line = replace(raw_line, "\r", "")

This workaround has to be repeated in every pipeline that ingests Windows-exported files. Failure to add it causes:

• First record to fail BOM-prefixed column parsing
• Every record's last column to retain trailing CR
• Idempotency: same input read twice may produce different output if first read consumes BOM

Confirmed in source: lib/file-source/src/file_watcher/mod.rs:247 (Vector 0.58.0)

Verified empirically: 5-row file with BOM + CRLF -> 5 events received, all 5 have trailing CR, first has \u{FEFF} prefix.

Start by reproducing the BOM and CRLF case from the issue, then inspect lib/file-source/src/file_watcher/mod.rs:247 and how raw_line is produced. Define the chosen default or configuration behavior for BOM and trailing CR handling, add regression coverage for both, and verify that all lines are emitted without the unwanted characters.