Proposal: optional native parser backend with pure-Python fallback

python-dotenv optional native parser backend

Problem

python-dotenv parsing is on the configuration path of real consumers such as
pydantic-settings. We measured complete consumer calls, not parser-only
throughput, on the same macOS arm64 / CPython 3.14 environment.

Proposal

• Keep the current pure-Python parser and pip install python-dotenv default.
• Add an optional native extra whose backend-only dependency is
  fast-dotenv-rs-backend.
• Keep the Rust package separate from the dotenv namespace and console
  script; the adapter only converts lossless parser records into the existing
  Binding and Original objects.
• If the extra/backend is absent, the existing pure-Python path remains in
  use. If an explicitly selected backend violates its versioned contract or
  raises, surface that failure rather than silently masking it.

Runtime evidence

• Upstream base: a00cb2eed0704cd6d2071b2004c37e95ccc86ee5.
• Backend implementation: fast-dotenv-rs commits
  7cbdca0 and 95ea5c1 in the Master integration clone; shared Rust core
  and backend-only wheel are ready for review.
• Upstream adapter patch: python-dotenv commits
  0c1d17b9878ae2b390616fbd8b64f605f9fd9d67 and
  f5856435e229485040e0e11530a191b0f51007df.
• Exact Binding differential: 1,088 records, including 500 generated valid
  and 500 generated malformed records; native OFF, native ON, and backend
  direct paths each retained 0 mismatches.
• Upstream suite: 264 passed, 1 skipped in both OFF and ON runs, using a
  portable fixture for the host-specific printenv --version test.
• Native call proof: fast_dotenv_rs_backend.parse_bindings was observed on
  the same parse_stream path; backend and official package coexist without
  a top-level dotenv collision.
• Real consumer benchmark: dotenv_values, load_dotenv,
  pydantic-settings service/worker settings, repeated loads, and cold
  startup across small/medium/large fixtures. Semantic hashes matched;
  30 p50 metrics classified as 21 win / 8 neutral / 1 loss under a 2%
  neutral band. The only loss was small cold CLI startup (-1.6675 ms);
  representative warm savings were 0.1266 ms for small dotenv_values,
  0.7477 ms for medium, and 4.9392 ms for large.

Maintainer questions

1. Is an optional backend extra acceptable, with backend wheels maintained
   outside the default python-dotenv release path?
2. Should activation use the native extra, another name, or a different
   boundary?
3. Which platform, PyPy/free-threaded, supply-chain, and rollback guarantees
   are required before merge?

This is a concrete proposal with a merge-ready local patch. The backend is not
yet published to PyPI, and the current wheel evidence is macOS arm64 only;
those are intentionally open release gates, not hidden claims.

Start with the existing parse_stream path and inspect how parser records become Binding and Original objects; then review the cited adapter commits and fast_dotenv_rs_backend.parse_bindings entry point. Run the upstream suite in native OFF and ON modes and compare the differential results. Done requires maintainer decisions on the extra, platform and interpreter support, supply-chain guarantees, and rollback before release.