NVIDIA-NeMo/Skills

View on GitHub

A project to improve skills of large language models

Stars: 1k
Forks: 204
Open beginner issues: 0
Indexed issues: 76
Avg merge: 2d 19h
Merged PRs (30d): 5

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 11, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

76 open issues indexed

Verify evals on Papers with Code Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA-NeMo/Skills#1516 · Opened Aug 16, 2026
[Bug] GPQA MCQ answer extraction prefers boxed equation over final "Answer: D" Open

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

NVIDIA-NeMo/Skills#1511 · Opened Jul 26, 2026
BFCL v3 preprocessing differs from actual BFCL Repository, causing incorrect evaluation for Java/JavaScript splits Open

Difficulty 3/5 1-2 days Newbie friendliness 65/100

NVIDIA-NeMo/Skills#1510 · Opened Jul 23, 2026
fix: yaml.safe_load returns none for empty yaml files, propagating as none to callers expecting dict Open

Difficulty 2/5 1-3 hours Newbie friendliness 64/100

NVIDIA-NeMo/Skills#1497 · Opened Jun 27, 2026
Add IHEval (Instruction Hierarchy Evaluation) benchmark Open

Difficulty 5/5 Over a week Newbie friendliness 42/100

NVIDIA-NeMo/Skills#1463 · 3 comments · Opened May 27, 2026
Add MT-Bench and Komt-Bench (Korean MT-Bench) benchmarks Open

benchmarks

NVIDIA-NeMo/Skills#1377 · 1 assignee · Opened Apr 17, 2026
Explore options for tool call execution concurrency Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA-NeMo/Skills#1369 · Opened Apr 16, 2026
Fix mismatch between `SLURM_PROCID=0` and `SLURM_JOB_NODELIST` for multi-node deployments dependent on `SLURM_MASTER_NODE` Open

bug

NVIDIA-NeMo/Skills#1330 · 1 assignee · Opened Mar 30, 2026
Merge DSBenchEvaluator into MathEvaluator Open

NVIDIA-NeMo/Skills#1268 · 1 assignee · Opened Feb 22, 2026
Add docs for CritPt benchmark Open

NVIDIA-NeMo/Skills#1266 · 1 comment · 1 reaction · 1 assignee · Opened Feb 21, 2026
Refactor registering external repositories Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

NVIDIA-NeMo/Skills#1260 · Opened Feb 19, 2026
BaseMetrics:_compute_majority_at_k and MathMetrics:_compute_reward_at_k handling of duplicate answer strings could potentially be incorrect for non-deterministic judges Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA-NeMo/Skills#1257 · 2 comments · Opened Feb 19, 2026
Add a proper ci check for broken links and other issues in docs Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

NVIDIA-NeMo/Skills#1207 · Opened Feb 2, 2026
Megatron server Open

NVIDIA-NeMo/Skills#1176 · 1 assignee · Opened Jan 20, 2026
`is_mounted_filepath` incorrect logic does not allow different paths inside and outside of container Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

NVIDIA-NeMo/Skills#1173 · 5 comments · Opened Jan 19, 2026
compute_eval Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

NVIDIA-NeMo/Skills#1171 · 9 comments · Opened Jan 16, 2026
Remove patch for litellm logging Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

NVIDIA-NeMo/Skills#1168 · Opened Jan 14, 2026
Unify answer bench and grading bench Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

NVIDIA-NeMo/Skills#1163 · Opened Jan 12, 2026
Trend Anomalies in Ruler V2 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA-NeMo/Skills#1143 · 5 comments · Opened Dec 25, 2025
Redistributing input data across chunks upon job rerun or resume Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA-NeMo/Skills#1121 · 1 reaction · Opened Dec 17, 2025
Pipeline for nemotron-math dataset release Open

NVIDIA-NeMo/Skills#1110 · 1 assignee · Opened Dec 15, 2025
Default regex/boxed order changed in math grader Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 45/100

NVIDIA-NeMo/Skills#1084 · Opened Dec 9, 2025
Concurrency improvements for prover generation Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 38/100

NVIDIA-NeMo/Skills#1083 · Opened Dec 9, 2025
[nemo-evaluator] Implement sandboxing Open

enhancement

NVIDIA-NeMo/Skills#1068 · 1 assignee · Opened Dec 3, 2025
Improve VLLM Multi-Node Command Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA-NeMo/Skills#1059 · Opened Nov 30, 2025
[nemo-evaluator] Ensure the dependent-jobs scenario works Open

enhancement

NVIDIA-NeMo/Skills#1040 · 1 assignee · Opened Nov 10, 2025
[nemo-evaluator] Extend deployments from vllm Open

enhancement

NVIDIA-NeMo/Skills#1039 · 1 assignee · Opened Nov 10, 2025
[nemo-evaluator] Document the configs Open

NVIDIA-NeMo/Skills#1038 · 1 assignee · Opened Nov 10, 2025
Make sure data_dir works without --cluster parameter in prepare_data Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 35/100

NVIDIA-NeMo/Skills#1036 · Opened Nov 7, 2025
Need to find a way to customize judge setup only for part of benchmarks for groups Open

bug

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA-NeMo/Skills#1035 · Opened Nov 7, 2025
keep_mounts_for_sandbox is unused in generate pipeline Open

bug

NVIDIA-NeMo/Skills#1028 · 1 reaction · 1 assignee · Opened Nov 4, 2025
Move scicode test data download to runtime Open

enhancement

NVIDIA-NeMo/Skills#1027 · 1 assignee · Opened Nov 4, 2025
Wait for sandbox needs improvements Open

bug

NVIDIA-NeMo/Skills#1015 · 1 assignee · Opened Oct 31, 2025
Duplicate all commands in the docs in both cli and python interface Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA-NeMo/Skills#984 · Opened Oct 23, 2025
Allow passing custom containers as an argument to all pipelines Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA-NeMo/Skills#971 · Opened Oct 17, 2025
qwen3coder_30b_swebench slurm test always fails in case NEMO_SKILLS_DATA_DIR env var is set Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 38/100

NVIDIA-NeMo/Skills#967 · Opened Oct 17, 2025
Slurm test submission frequently failed because prepare_data is called from other tests when some tests are being submitted and packaging fails Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 20/100

NVIDIA-NeMo/Skills#964 · Opened Oct 17, 2025
Add slurm tests for coding benchmarks (lcb c++, lcb with pypy3, etc.) Open

enhancement

NVIDIA-NeMo/Skills#962 · 1 assignee · Opened Oct 17, 2025
Add slurm test for IOI benchmark Open

enhancement

NVIDIA-NeMo/Skills#940 · 1 assignee · Opened Oct 13, 2025
Add a function automatically search optimal config for a new model Open

enhancement

NVIDIA-NeMo/Skills#915 · 1 assignee · Opened Oct 8, 2025
Add support for other translation metrics, such as COMET and MetricX Open

NVIDIA-NeMo/Skills#902 · 1 assignee · Opened Oct 7, 2025
Refactor BLEU metrics computation so it reuses parent method functions from pass@k Open

NVIDIA-NeMo/Skills#901 · 1 assignee · Opened Oct 7, 2025
Add support for TerminalBench Open

benchmarks

Difficulty 5/5 Over a week Newbie friendliness 15/100

NVIDIA-NeMo/Skills#898 · Opened Oct 7, 2025
Eval jobs fail if HF model isn't downloaded and num_jobs is set Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

NVIDIA-NeMo/Skills#867 · 1 comment · Opened Sep 30, 2025
Slurm Testing Documentation/Ergonomics Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA-NeMo/Skills#864 · 2 comments · Opened Sep 29, 2025
Address remaining issues for prompt sensitivity pipeline Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

NVIDIA-NeMo/Skills#861 · Opened Sep 29, 2025
Add a proxy server that will catch openai requests and redirect through our clients Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA-NeMo/Skills#858 · Opened Sep 29, 2025
Figure out training container setup on arm64 Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

NVIDIA-NeMo/Skills#857 · Opened Sep 29, 2025
Overriding max_new_tokens in generation_config.json Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

NVIDIA-NeMo/Skills#845 · Opened Sep 24, 2025
Switch all outputs to be in "messages" format Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA-NeMo/Skills#835 · 1 reaction · Opened Sep 22, 2025

NVIDIA-NeMo/Skills

Get new issues in your inbox