Adaptor docs: Create adaptor knowledge tests
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 5
- Forks
- 10
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 17
Description
We need a test set that focuses on adaptor-specific code generation. It should test for specific knowledge contained in different parts of the adaptor documentation across several adaptors.
Example of a user feedback case that needs to be targeted:
"AI will write a sendMessages call to Gmail and include as an attribute « text » or « message » which are undefined (and cause it to crash), instead of using « body » which is clearly defined in the adaptor doc."
This case currently fails because we currently only add adaptor signatures to prompts, while this requires knowledge from the "Interfaces" section of the Gmail docs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the existing adaptor-signature prompt path and review how adaptor knowledge is currently tested. Use the Gmail documentation's Interfaces section as the first case, then compare several adaptors for similarly specific knowledge. Done means the test set catches the incorrect text/message attributes and expects the documented body attribute.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook
- Domain
- ai, documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100