Create a chunk summary and discarting irrevelant information.

How would it be possible to have a summary made out of a chunk? With a prompt named: `summary_gen.yaml`

**_EDIT: While I had trouble getting my head around the code in `./original`, I started from scratch with `BOILERPLATE_TO_MAKE_YOUR_OWN_PIPELINE`. So I kinda figured how to this on my own._**

Will close this in a bit when I get it working, for now, I am sharing some of my work in abstracting the pipeline, abusing the prompts and optimizing the quality of the output while minimizing token use.

----

I have had some passable results by telling it to identify the main theme of the chunk and what domain of knowledge it's about and that any information not in relation to it should be discarded, like publicity or irrelevant mixed text. That would help for generating good content for continuous training. Different levels of shrinking it down could be tried: long summary, summary, short summary.

```Summarize the following text by keeping only what is consistent with the main idea, theme, or key points. Remove anything that is not relevant or seems off-topic." ```

Also, if this works well, it could be possible to preprocess the chunk for later extraction of QA data, validating it against the original chunk. Small 1.5b and 3b models could be used to pump quick **and cheap** Q-A that could be classified, verified, grouped by `something common` and reworded in a complex Q-A that convey more information.

Would have to get it done and compare *normal pipeline* with a *summary based pipeline* to see if there is any difference in dataset quality or speed to generate it.

Thanks