Brainstorming Testing Development

Below is a result of the brain storming session for testing development. The goal is to flesh out a CI/CD system as described in issue #204 . Use this to start dividing up issues as tests.

* Server Tests
** Does the server complete it's setup, to respond to HTTP(S) requests?
*** Test
run the flask app holding the endpoint
*** Passing?
The app offers an http(s) endpoint for requests

** Test individual Endpoints
*** Test
Test the completion endpoint to ensure a generative text (model Specific)
*** Passing?
Text is generated from the completion endpoint, with a 200 status code

* Data Synthesis Tests
** Data Quality Tests
*** How do we measure quality?
- Check with a larger model
- Cosine similarity
- Perplexity scores
- Correlation scores
- tokenization metrics
- A/B testing
- Data quality is defined in the classical way
** You provide examples that you'd like to scale
** We send to openAI "give me 50 examples like this one"
** Structure JSON object for finetuning
** at data-threshold we finetune the model
*** Dataset that came from gpt
**** 70% for training
**** 30% for testing
**** How useful is the model that you just created, to you?
**** How 'well' did our finetuning process work?
***** Did we give enough examples?
***** Were the examples of high enough "quality"?
** save model / load into completion
*** Reporting Metric
Natural Language Processing to assess the quality of the data generated