kubeflow / kubeflow/sdk

HuggingFaceModelInitializer and S3ModelInitializer share a mutable default ignore_patterns list across instances

Open Beginner friendly
#612 3 comments 0 reactions 0 assignees View on GitHub
kind/bug needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What happened?

Description

While reviewing kubeflow/trainer/types/types.py, I noticed that both HuggingFaceModelInitializer and S3ModelInitializer define ignore_patterns as:

ignore_patterns: list[str] | None = field(
default_factory=lambda: constants.INITIALIZER_DEFAULT_IGNORE_PATTERNS
)

At first glance this looks correct because default_factory is used. However, the factory returns the module-level mutable list directly instead of creating a new list for each instance.

INITIALIZER_DEFAULT_IGNORE_PATTERNS is defined as:

INITIALIZER_DEFAULT_IGNORE_PATTERNS = [
"*.msgpack",
"*.h5",
"*.bin",
"*.pt",
"*.pth",
]

Since the same list object is returned every time, all initializer instances share the same underlying ignore_patterns list.

Verified locally
from kubeflow.trainer.types.types import HuggingFaceModelInitializer

a = HuggingFaceModelInitializer(storage_uri="hf://user/model1")
b = HuggingFaceModelInitializer(storage_uri="hf://user/model2")

print(a.ignore_patterns is b.ignore_patterns)

Output:

True

Both instances reference the exact same list object.

To verify the impact:

print("Before:")
print("A:", a.ignore_patterns)
print("B:", b.ignore_patterns)

a.ignore_patterns.append("*.custom")

print("After:")
print("A:", a.ignore_patterns)
print("B:", b.ignore_patterns)

Output:

Before:
A: ['*.msgpack', '*.h5', '*.bin', '*.pt', '*.pth']
B: ['*.msgpack', '*.h5', '*.bin', '*.pt', '*.pth']

After:
A: ['*.msgpack', '*.h5', '*.bin', '*.pt', '*.pth', '*.custom']
B: ['*.msgpack', '*.h5', '*.bin', '*.pt', '*.pth', '*.custom']

Even though only a.ignore_patterns was modified, the change is immediately visible in b.ignore_patterns because both instances share the same mutable list.

Existing pattern in the codebase

The same file already uses the correct pattern for CustomTrainer.pip_index_urls:

pip_index_urls: list[str] = field(
default_factory=lambda: list(constants.DEFAULT_PIP_INDEX_URLS)
)

Creating a new list for every instance avoids sharing mutable state.

Expected behavior

Each HuggingFaceModelInitializer and S3ModelInitializer instance should receive its own independent copy of the default ignore_patterns list.

Suggested fix
ignore_patterns: list[str] | None = field(
default_factory=lambda: list(constants.INITIALIZER_DEFAULT_IGNORE_PATTERNS)
)

This matches the pattern already used elsewhere in the same file.

Environment

Repository: kubeflow/sdk
File: kubeflow/trainer/types/types.py
Affected classes:
HuggingFaceModelInitializer
S3ModelInitializer

I'd be happy to submit a PR if this behavior is considered a bug.

### What did you expect to happen?

Each `HuggingFaceModelInitializer` and `S3ModelInitializer` instance should receive its own independent copy of the default `ignore_patterns` list.

Mutating the `ignore_patterns` of one instance should not affect any other initializer instance.

For example, after creating two independent initializer objects, appending a new ignore pattern to one instance should leave the other instance unchanged.

### Environment

Kubernetes version:
```bash
$ kubectl version

```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"

```
Kubeflow Python SDK version:
```bash
$ pip show kubeflow

```

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Research direction

Start in kubeflow/trainer/types/types.py at HuggingFaceModelInitializer and S3ModelInitializer, then compare their ignore_patterns definitions with CustomTrainer.pip_index_urls. Verify that separately created initializers no longer share mutable default state, and confirm that changing one instance leaves the other unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.