LAION-AI / LAION-AI/CLIP_benchmark
Organize CLI arguments into groups
Open
Nobody has claimed this yet.
UX
- Dominant language
- Python
- Stars
- 814
- Forks
- 103
- PR merge metrics
- No merged PRs in 30d
Description
Use parser.add_argument_group() to organize the CLI arguments logically; currently, the help text is just a wall of text, and it's likely to only get longer over time. (Not a high priority though)
options:
-h, --help show this help message and exit
--dataset DATASET Dataset to use for the benchmark
--split SPLIT Dataset split to use
--model MODEL Model architecture to use from OpenCLIP
--pretrained PRETRAINED
Model checkpoint name to use from OpenCLIP
--task {zeroshot_classification,zeroshot_retrieval,linear_probe}
--amp whether to use mixed precision
--num_workers NUM_WORKERS
--recall_k RECALL_K [RECALL_K ...]
for retrieval, select the k for Recall@K metric.
--fewshot_k FEWSHOT_K
for linear probe, how many shots. -1 = whole dataset.
--fewshot_epochs FEWSHOT_EPOCHS
for linear probe, how many epochs.
--fewshot_lr FEWSHOT_LR
for linear probe, what is the learning rate.
--skip_load for linear probes, when everything is cached, no need
to load model.
--seed SEED random seed.
--batch_size BATCH_SIZE
--model_cache_dir MODEL_CACHE_DIR
directory to where downloaded models are cached
--dataset_root DATASET_ROOT
dataset root folder where the datasets are downloaded.
--feature_root FEATURE_ROOT
feature root folder where the features are stored.
--annotation_file ANNOTATION_FILE
text annotation file for retrieval datasets. Only
needed for when `--task` is `zeroshot_retrieval`.
--language LANGUAGE language of classname and prompts to use for zeroshot
classification.
--output OUTPUT output file where to dump the metrics
--verbose verbose mode
--cupl Use natural language prompt from CuPL paper
--save_clf SAVE_CLF optionally save the classification layer output by the
text tower
--load_clfs LOAD_CLFS [LOAD_CLFS ...]
optionally load and average mutliple layers output by
text towers.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the CLI argument parser that defines the options shown in the issue and review how the arguments are currently added. Organize the existing options into logical groups with argparse argument groups, then run the CLI help command and confirm the output is no longer a single wall of text.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100