AnswerDotAI / AnswerDotAI/llmdojo

Feature Request:External katas

Open
#13 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
6
Forks
1
Avg merge
1m
Merged PRs (30d)
3

Description

### Problem
Currently, to run custom katas, you have to fork llmdojo or hardcode the imports into the core runner. There is no native way to automatically discover and load katas from outside the repository.

### Proposed Solution
Use Python's native importlib.metadata.entry_points to allow llmdojo to discover and load external katas dynamically.
Downstream packages would simply register their katas in their pyproject.toml:
[project.entry-points."llmdojo.katas"]
custom_kata = "custom_dep_module.katas.custom_kata"

During initialization, llmdojo would scan the llmdojo.katas group and append any discovered modules to its existing execution list.

### Benefits
* Zero dependencies: Relies entirely on the standard library.
* Backward compatible: Behaves exactly the same if no external entry points are found.
* Extensible: Enables a modular ecosystem where users can distribute custom kata suites via PyPI without bloating the core project.
* Mental model shift: Responsibility of how to use and evaluate a skill lies in the module that reflects the code, docs, prompts and the pyskill itself. The So you would config what katas to load in the harness's project.toml, the modules implement the actual katas and llmdojo executes them.

This would help to evaluate skill usage fast and make iterations on the docs /prompts faster.

I would be very interested in your thoughts on this and if it aligns to your plans of the project.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the initialization code that builds the existing kata execution list and review Python's importlib.metadata.entry_points API. Check the downstream pyproject.toml entry-point format described in the issue, then verify that modules registered under llmdojo.katas are discovered and loaded while the existing behavior remains unchanged when none are registered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.