aws-samples / aws-samples/foundation-model-benchmarking-tool
Experiment config file's model name doesn't match with pricing_fallback.yml file's model name
- Dominant language
- Jupyter Notebook
- Stars
- 256
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
When running the experiment using this [config file](https://github.com/aws-samples/foundation-model-benchmarking-tool/blob/main/src/fmbench/configs/bedrock/config-bedrock-llama3-1.yml), the model id is [here](https://github.com/aws-samples/foundation-model-benchmarking-tool/blob/main/src/fmbench/configs/bedrock/config-bedrock-llama3-1.yml#L156).
This model id is not in the [pricing_fallback.yml](https://github.com/aws-samples/foundation-model-benchmarking-tool/blob/main/src/fmbench/configs/pricing_fallback.yml) file.
Contributor guide
Research direction
Compare the model id at line 156 of src/fmbench/configs/bedrock/config-bedrock-llama3-1.yml with the entries in src/fmbench/configs/pricing_fallback.yml. Determine the matching model name and align the configuration data so the experiment's model is represented in the pricing fallback file. Confirm that the two files use the same model identifier.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, yaml
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100