microsoft / microsoft/mssql-rs
Preserve unpaired UTF-16 code units in ODBC wide-character fetches
- Dominant language
- Rust
- Stars
- 53
- Forks
- 14
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 137
Description
### Describe the bug
The Rust ODBC driver replaces unpaired UTF-16 surrogates with U+FFFD when fetching bounded `nvarchar` as `SQL_C_WCHAR`. This changes the payload before the Python consumer can apply its decoding policy.
New regression coverage in **microsoft/mssql-python#793**, merged September 18, 2026 at commit [`6499f89a3eeae036c117604623f3a20ec31fb5fa`](https://github.com/microsoft/mssql-python/commit/6499f89a3eeae036c117604623f3a20ec31fb5fa), exposed this existing ODBC behavior. That PR fixes leading BOM-like characters and adds `tests/test_017_fetch_bounded_text.py`; the failures here are its **malformed-surrogate cases**, not the BOM cases.
### Steps to reproduce
1. Run the existing `mssql-python suite on mssql-odbc driver (cross-repo)` job against upstream commit `c7ed6f8e7a2dec212662a9233e372f2d537787dd`, as in failing build 176490. The job builds upstream pybind bindings, replaces its bundled ODBC binary with this repository's driver, and runs the upstream suite.
2. To narrow an already-prepared environment, run `python -m pytest tests/test_017_fetch_bounded_text.py -v` with the standard upstream connection fixture configured.
3. The relevant SQL payloads are `SELECT CAST(0x00D8 AS nvarchar(64))` and `SELECT CAST(0x00DC AS nvarchar(64))`: isolated high and low surrogate code units. Repeat with `, CAST(N'route' AS nvarchar(max))` to force the row-wise fetch path for the bounded column.
4. Exercise `fetchone()`, `fetchmany()`, and `fetchall()` through the existing parametrized tests.
### Expected behavior
Deliver the original UTF-16 code units to the caller's `SQL_C_WCHAR` buffer, with correct byte-length indicators and termination. Do not replace them, drop them, or manufacture a Python-specific result inside ODBC.
With those bytes preserved, the existing upstream consumer determines the outcome:
- Row-wise bounded fetches raise `UnicodeDecodeError` for an isolated surrogate (8 cases, including bounded columns beside MAX).
- Bounded batch decoding on Linux/macOS follows its existing fallback to `''` (4 cases). On Windows that upstream test expects the surrogate-preserving wide-character result instead.
- The connection/cursor remains usable for the subsequent `SELECT N'recovered'` assertions.
### Actual behavior
Build 176490 reports **12 failed, 42 passed** in `test_017_fetch_bounded_text.py`:
- `test_bounded_nvarchar_strict_decode_error`: 8 failures, `DID NOT RAISE UnicodeDecodeError`.
- `test_bounded_nvarchar_batch_malformed_fallback`: 4 failures, `assert [('�',)] == [('',)]`.
Whole-run summary: 64 files; 59 passed, 1 failed, 4 with no tests collected; **0 crashes, 0 timeouts, 0 harness errors**. Build 176489 independently shows the same 12 failures. The macOS cross-repo job in build 176490 passed; it is a different test path and is not evidence that this ODBC swap works on macOS.
### Version
- Failing mssql-rs build 176490: merge SHA `916723fec9c376be465e2b712158ec255ffabb83`, based on `b0ecac258eedaea1d43ca966b0528d90c79549f7`. That PR's diff changes pipeline/wheel-install files, not these fetch implementations.
- Upstream checkout recorded in its clone log: `c7ed6f8e7a2dec212662a9233e372f2d537787dd`.
- Upstream change introducing the failing test cases: `6499f89a3eeae036c117604623f3a20ec31fb5fa` (microsoft/mssql-python#793).
### Affected crate
mssql-odbc
### Environment
Existing ODBC-swap CI: Ubuntu 24.04 test container, Ubuntu 22.04 Rust build container, SQL Server `2025-latest`. The replacement driver reports `SQL_DRIVER_VER` as `18.6.2.1` in this run; the swap step, rather than that version string alone, identifies the loaded implementation.
### Additional context
**Root cause and implementation scope**
At mssql-rs `b0ecac258eedaea1d43ca966b0528d90c79549f7`:
- [`get_data.rs`](https://github.com/microsoft/mssql-rs/blob/b0ecac258eedaea1d43ca966b0528d90c79549f7/mssql-odbc/src/api/get_data.rs): `try_write_complete_buffered_string` (around line 613) and `try_write_direct_captured_string_chunk` (around line 1381) require `is_valid_utf16le` before delivering UTF-16 directly.
- [`fetch_scroll.rs`](https://github.com/microsoft/mssql-rs/blob/b0ecac258eedaea1d43ca966b0528d90c79549f7/mssql-odbc/src/api/fetch_scroll.rs): `deliver_encoded_string` (around line 2107) has the same requirement and falls back to `deliver_bound`.
- Those fallbacks go through `column_value_to_text` -> `sql_string_to_text` -> `SqlString::to_utf8_string` / `SqlString::decode`. [`sql_string.rs`](https://github.com/microsoft/mssql-rs/blob/b0ecac258eedaea1d43ca966b0528d90c79549f7/mssql-tds/src/datatypes/sql_string.rs#L132-L150) decodes UTF-16 with replacement through `encoding_rs`; re-encoding that Rust string cannot recover the original surrogate.
- The TDS `SqlString` already retains the original bytes. Wide-to-wide delivery should not require a Unicode-scalar round trip. Keep even-byte-length, destination-size, alignment, termination, and truncation checks; unpaired code units are distinct from structurally incomplete bytes.
**Required work / acceptance criteria**
- [ ] Preserve original UTF-16 units for `SQL_C_WCHAR` across both `SQLGetData` paths and bound/row-array fetches, including buffered fallbacks. Reuse a common raw-unit delivery primitive where practical rather than fixing only one fast path.
- [ ] Keep actual transcoding (`SQL_C_CHAR`, other encodings/types) separate; do not globally weaken `is_valid_utf16le` or change every `SqlString` consumer to satisfy an ODBC same-encoding copy.
- [ ] Replace the current test expectation that an unpaired surrogate is ineligible for direct wide delivery. Add assertions on actual output units and byte indicators, not only Python results.
- [ ] Cover high/low unpaired units, full and truncated/repeated `SQLGetData`, bound row arrays, buffered delivery, and bounded columns beside MAX. Verify continuation and end-of-data behavior over repeated calls.
- [ ] Preserve healthy behavior for valid surrogate pairs, leading U+FEFF/U+FFFE, embedded/trailing NUL, empty strings, NULL, and exact-size buffers. Keep odd-byte handling explicit and safe. Inspect streaming siblings; document any unchanged path and why.
- [ ] Pass all 54 upstream bounded-text tests and the full pinned ODBC-swap suite without skipping the new cases. Run native differential coverage against the repository's retail msodbcsql baseline and record the actual `SQL_DRIVER_VER`/build.
**CI evidence**
- [Failing build 176490, ODBC test log](https://dev.azure.com/sqlclientdrivers/904996cc-6198-4d39-8540-eca72bdf0b7b/_build/results?buildId=176490&view=logs&j=9618f721-860a-5718-2d72-a26226eed46e&t=9b6cc109-caa9-5135-b4e1-c3f9454d375b) (log 341); clone SHA is in log 332.
- [Independent failing build 176489](https://dev.azure.com/sqlclientdrivers/904996cc-6198-4d39-8540-eca72bdf0b7b/_build/results?buildId=176489) (log 153).
- [Earlier green build 176345](https://dev.azure.com/sqlclientdrivers/904996cc-6198-4d39-8540-eca72bdf0b7b/_build/results?buildId=176345) tested upstream `e279a4f6c12241603b978602acf01e64520a819f`: 57 passed files, 4 with no collected tests, no failing files. It predates the added regression file.
- [Upstream build 176426, Ubuntu SQL2025](https://dev.azure.com/sqlclientdrivers/904996cc-6198-4d39-8540-eca72bdf0b7b/_build/results?buildId=176426&view=logs&jobId=d19f7c64-15e2-54ad-ea0c-211da03c7db4) (log 240) passes all 12 exact cases against bundled Microsoft ODBC, reporting `SQL_DRIVER_VER = 18.06.0002`; whole leg: 5017 passed, 148 skipped, 42 deselected. This is measured upstream behavior, not a claim that this Linux build is identical to the repository's Windows retail baseline.
Investigation used existing CI logs and source tracing; no new local SQL Server reproduction or fix has been run.
Contributor guide
Research direction
Start in mssql-odbc/src/api/get_data.rs at try_write_complete_buffered_string and try_write_direct_captured_string_chunk, then inspect fetch_scroll.rs at deliver_encoded_string and mssql-tds/src/datatypes/sql_string.rs. Run tests/test_017_fetch_bounded_text.py with the upstream connection fixture and trace both SQLGetData and bound/row-array paths. Done means original UTF-16 units, indicators, truncation, termination, and continuation behavior pass the listed cases without changing unrelated transcoding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100