huggingface / huggingface/swift-transformers

Offline-mode metadata lookup fails when downloadBase contains '..' path components

Open Beginner friendly
#366 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
1.4k
Forks
209
Avg merge
2d 7h
Merged PRs (30d)
4

Description

**swift-transformers version**: 0.1.24 (revision `f000aa7aec0e78acd0211685e4094e1fca84cd8b`)
**Platform**: macOS 26 (Apple Silicon)

## Summary

When `HubApi(downloadBase: someURL, useOfflineMode: true)` is constructed with a `downloadBase` whose path contains `..` components (e.g. `/Users/me/project/foo/../../resources/tokenizer`), the offline-mode snapshot loader fails with `EnvironmentError.offlineModeError("Metadata not available for ")` even when the cache layout is complete and the metadata markers exist on disk at the locations the loader would compute *after* canonicalizing.

Passing the same directory with a canonicalized absolute path (no `..`) succeeds.

## Root cause

In `Sources/Hub/HubApi.swift`, the `snapshot(...)` function (around line 538–565) walks files via `FileManager.enumerator(at: repoDestination, ...)`. The enumerator returns `URL`s whose `.path` is already canonicalized — `..` components are resolved away by the OS.

For each enumerated file, the metadata path is built via:

```swift
let metadataPath = URL(
fileURLWithPath: fileUrl.path.replacingOccurrences(
of: repoDestination.path,
with: repoMetadataDestination.path
) + ".metadata"
)
```

`fileUrl.path` is canonical (no `..`). `repoDestination.path` is whatever was passed in by the caller via `downloadBase`, which preserves `..` if present. The substring replacement therefore fails to match, the resulting `metadataPath` is wrong, `readDownloadMetadata` returns nil because the file doesn't exist at the wrong path, and the loader throws `Metadata not available`.

## Repro

Minimal Swift package using swift-transformers 0.1.24:

```swift
import Foundation
import Hub
import Tokenizers

// A well-formed offline cache at /tmp/cache-root/models/BAAI/bge-base-en-v1.5/
// produced by huggingface_hub.snapshot_download + writeDownloadMetadata equivalents.

// Path WITH '..' components — fails
let badBase = URL(fileURLWithPath: "/tmp/cache-root/sub/../")
let hub1 = HubApi(downloadBase: badBase, useOfflineMode: true)
_ = try await AutoTokenizer.from(pretrained: "BAAI/bge-base-en-v1.5", hubApi: hub1)
// → throws offlineModeError("Metadata not available for tokenizer_config.json")

// Same cache directory, canonical path — succeeds
let goodBase = URL(fileURLWithPath: "/tmp/cache-root")
let hub2 = HubApi(downloadBase: goodBase, useOfflineMode: true)
_ = try await AutoTokenizer.from(pretrained: "BAAI/bge-base-en-v1.5", hubApi: hub2)
// → loads fine
```

## Suggested fix

Canonicalize `repoDestination.path` (and/or `repoMetadataDestination.path`) before the substring replacement, or build the metadata path by URL appending rather than string substitution. Something like:

```swift
let repoDestinationCanonical = repoDestination.standardizedFileURL
let repoMetadataDestinationCanonical = repoMetadataDestination.standardizedFileURL
// ... use canonical paths in the substitution
```

Or, more robustly, compute the metadata path via URL operations rather than path-string surgery:

```swift
let relPath = fileUrl.path.replacingOccurrences(of: repoDestination.standardizedFileURL.path + "/", with: "")
let metadataPath = repoMetadataDestination.appendingPathComponent(relPath + ".metadata")
```

## Workaround

Callers can canonicalize the `downloadBase` URL before passing it to `HubApi`:

```swift
let hub = HubApi(downloadBase: userSuppliedURL.standardizedFileURL, useOfflineMode: true)
```

We've adopted this defensive workaround in our downstream project (verveguy/liminis#815) but the underlying behavior is surprising — silent metadata-lookup failure on a path with `..` is a meaningful footgun for callers building integrations.

## Related

Discovered during end-to-end validation of a Liminis embedding sidecar (BGE-base-en-v1.5 via CoreML). The cache layout was generated correctly by a `huggingface_hub.snapshot_download`-based script plus per-file `.metadata` markers written via the documented 3-line format. Standalone repro with the canonical path succeeds; sidecar (which passed a `\$PWD/../..` constructed env-var path) fails.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in Sources/Hub/HubApi.swift, around snapshot(...) lines 538–565, and reproduce the issue with the minimal Swift package example using a downloadBase containing '..'. The work is done when offline-mode metadata lookup succeeds for both the non-canonical and canonical cache paths, without the Metadata not available error.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.