LAION-AI / LAION-AI/CLIP_benchmark
Eval custom open_clip architecture – Add model config
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 814
- Forks
- 103
- PR merge metrics
- No merged PRs in 30d
Description
What should be the appropriate way of adding a new model architecture without having to clone the repo? With open_clip we can write the model config json and add it using use add_model_config from open_clip.factory. However, doing this before runing the clip_benchmark cli has no effect, since this opens a new process that doesn't see the new architecture added.
Simlarly, what is the correct way of passing my architecture and my pretrained weights to the cli?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading open_clip.factory.add_model_config and the clip_benchmark CLI entry point, focusing on how the separate process discovers model configurations. Determine how a custom architecture and pretrained weights should be supplied to the CLI without cloning the repository. Done means the issue documents or enables a supported workflow for evaluating the custom model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100