Dyalog / Dyalog/ullu

Use GenAI to make test scenarios

Open
#99 4 comments 0 reactions 0 assignees View on GitHub
enhancement feature
Dominant language
APL
Stars
5
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Extremely chaotic but worth considering with how fast the new AI models run.

My idea is to use the OpenAI Package on Tatin(or any other LLM) and make a call to OpenAI(or similar) and ask it to give us some data that it thinks the APL primitive might break over. We should log everything this does.

I think it will give a lot of false positives but non-deterministic testing is a new way of doing things. Its on the same line as chaos testing.

Pros:
- Lots of new data
- Lots of uncommon scenarios that might give aplcores (or errors that are incorrect) because its gonna write something that does not work a lot of times
- Its a new thing ` ¯\(⍣)/¯ `
- (I think) We need more chaos testing

Cons:
- Expensive because it will need a subscription to an AI model (we can use an open source one like LLAMA3, its pretty good)
- Time taking- the time taken for each test will be increased (can be dealt with these only running with the slow tests)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.