huggingface / huggingface/swift-transformers

Heads-up: release-mode crash when co-linked with vapor/postgres-nio; upstream tracker swiftlang/swift#88794

Open
#350 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Swift
Stars
1.4k
Forks
209
Avg merge
2d 7h
Merged PRs (30d)
4

Description

# Heads-up: release-mode crash when co-linked with vapor/postgres-nio

Filing as a courtesy / FYI — this looks like a Swift release-mode codegen issue, not a defect in swift-transformers. Upstream tracker: **[swiftlang/swift#88794](https://github.com/swiftlang/swift/issues/88794)**.

## Symptom

A Swift release build that links **both** `huggingface/swift-transformers` (`Tokenizers` product, 1.3.0) and `vapor/postgres-nio` (1.33.0) into the same executable crashes with `freed pointer was not the last allocation` on the **second consecutive `PostgresClient.query(...)` call**. The first query succeeds; the second aborts the process.

Either dependency on its own works fine in release. A `swift build -c debug` build with both deps also works fine.

The trigger is `import Tokenizers` itself — no Tokenizers API has to be called for the crash to fire. That's what points the finger at `Tokenizers`-side codegen / module-init being part of the interaction (alongside NIO's allocator paths).

## Environment

- Swift 6.3.1 (`swiftlang-6.3.1.1.2`) / Xcode 26.4.1 / macOS 26.4.1, Apple M4
- swift-transformers 1.3.0
- postgres-nio 1.33.0
- swift-nio 2.99.0

## Minimal repro

The full minimal Package.swift + `main.swift` (≈30 lines) is in [swiftlang/swift#88794](https://github.com/swiftlang/swift/issues/88794). Two `SELECT N` queries against the bare `PostgresClient` are enough to reproduce, but only when `import Tokenizers` is also in the executable.

## Why I'm posting here

Pure FYI in case:

1. There's a known interaction between Tokenizers' module-init path (Hub/Tokenizers globals, lazy resource lookup, etc.) and Swift's allocator that the swift-transformers maintainers have context on, or
2. you want this on your radar in case other users hit it.

I'm not asking swift-transformers to fix this; the diagnosis points at the Swift toolchain. Feel free to close after triage. If a workaround on the swift-transformers side is feasible (e.g. avoiding whatever module-init path triggers it in release mode), that'd be a nice-to-have, but a Swift compiler patch is the right durable fix.

## Workarounds for users hitting this today

- Build the executable with `swift build -c debug` when both libraries are linked.
- Avoid linking `Tokenizers` and `PostgresNIO` into the same executable target (run tokenization in a separate process, or do tokenization Python-side and ship token IDs to the Swift side).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked upstream tracker swiftlang/swift#88794 and its minimal Package.swift plus main.swift reproduction. Run the release build with both dependencies and the two PostgresClient.query calls; this issue has no requested swift-transformers change, so completion depends on upstream diagnosis or a confirmed workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres, swift
Domain
compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.