LAION-AI / LAION-AI/CLIP_benchmark

Organize CLI arguments into groups

Open
#66 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

UX
Dominant language
Python
Stars
814
Forks
103
PR merge metrics
No merged PRs in 30d

Description

Use parser.add_argument_group() to organize the CLI arguments logically; currently, the help text is just a wall of text, and it's likely to only get longer over time. (Not a high priority though)

options:
  -h, --help            show this help message and exit
  --dataset DATASET     Dataset to use for the benchmark
  --split SPLIT         Dataset split to use
  --model MODEL         Model architecture to use from OpenCLIP
  --pretrained PRETRAINED
                        Model checkpoint name to use from OpenCLIP
  --task {zeroshot_classification,zeroshot_retrieval,linear_probe}
  --amp                 whether to use mixed precision
  --num_workers NUM_WORKERS
  --recall_k RECALL_K [RECALL_K ...]
                        for retrieval, select the k for Recall@K metric.
  --fewshot_k FEWSHOT_K
                        for linear probe, how many shots. -1 = whole dataset.
  --fewshot_epochs FEWSHOT_EPOCHS
                        for linear probe, how many epochs.
  --fewshot_lr FEWSHOT_LR
                        for linear probe, what is the learning rate.
  --skip_load           for linear probes, when everything is cached, no need
                        to load model.
  --seed SEED           random seed.
  --batch_size BATCH_SIZE
  --model_cache_dir MODEL_CACHE_DIR
                        directory to where downloaded models are cached
  --dataset_root DATASET_ROOT
                        dataset root folder where the datasets are downloaded.
  --feature_root FEATURE_ROOT
                        feature root folder where the features are stored.
  --annotation_file ANNOTATION_FILE
                        text annotation file for retrieval datasets. Only
                        needed for when `--task` is `zeroshot_retrieval`.
  --language LANGUAGE   language of classname and prompts to use for zeroshot
                        classification.
  --output OUTPUT       output file where to dump the metrics
  --verbose             verbose mode
  --cupl                Use natural language prompt from CuPL paper
  --save_clf SAVE_CLF   optionally save the classification layer output by the
                        text tower
  --load_clfs LOAD_CLFS [LOAD_CLFS ...]
                        optionally load and average mutliple layers output by
                        text towers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the CLI argument parser that defines the options shown in the issue and review how the arguments are currently added. Organize the existing options into logical groups with argparse argument groups, then run the CLI help command and confirm the output is no longer a single wall of text.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.