Setup check. Script to get keywords for comparing against SimpleMaths, TextRank and Philology results

So currently using the GPT and some read manuals. Did I correctly setup the code and transformer model? Or are there any suggestions which I could use? I will also try with ngrams up to 3.

Maybe some preprocessing suggestions or how to achieve to results with KeyBERT POS ner tags in the process.

I am comparing currently KeyBERT vs SketchEngine (SimpleMaths) and TextRank and also Philologically found keywords. And for my masters I thought KeyBERT would be best. I would like to checkout the efficiency with Estonian language models (tartuNLP/EstBERT) but also try with mBART(facebook/mbart-large-50) and mT5(google/mt5-base) and could add some value to the research. I tried also reaching out via linkedIn

All the best.

```
import csv
import os
from flair.embeddings import TransformerDocumentEmbeddings
from keybert import KeyBERT
import re

def load_and_preprocess_stopwords(file_path):
with open(file_path, 'r', encoding='UTF-8') as file:
# Normalize each stop word by lowering case and removing extra characters
stopwords = [re.sub(r'\W+', '', line.strip().lower()) for line in file]
return stopwords

def extract_keywords_and_write(text_path, csv_path, output_path, stopwords):
# Read the text file
with open(text_path, 'r', encoding="UTF-8") as file:
text_content = file.read()

# Count rows in the corresponding CSV file
with open(csv_path, 'r', newline='', encoding='UTF-8') as csvfile:
reader = csv.reader(csvfile)
next(reader, None) # Skip the header row
row_count = sum(1 for row in reader) # Count rows excluding the header

# Load the model and extract keywords
estBERT = TransformerDocumentEmbeddings('tartuNLP/EstBERT')
kw_model = KeyBERT(model=estBERT)
keywords = kw_model.extract_keywords(text_content, keyphrase_ngram_range=(1, 1), stop_words=stopwords, nr_candidates=row_count, top_n=row_count)

# Write keywords to a new CSV file in the output directory
with open(output_path, 'w', newline='', encoding="UTF-8") as csvfile:
writer = csv.writer(csvfile)
writer.writerow(("keywords", "score")) # Write header
writer.writerows(keywords) # Write keywords and scores

# Load and preprocess Estonian stopwords
estonian_stopwords = load_and_preprocess_stopwords('estonian-stopwords.txt')

# Define directories
txt_dir = 'raw_text'
csv_dir = 'pre_processed_text_data'
output_dir = 'keybert'
os.makedirs(output_dir, exist_ok=True)

# Process each text file in the txt_dir
for txt_filename in os.listdir(txt_dir):
if txt_filename.endswith('.txt'):
base_filename = os.path.splitext(txt_filename)[0]
csv_filename = f"{base_filename}.csv"
txt_file_path = os.path.join(txt_dir, txt_filename)
csv_file_path = os.path.join(csv_dir, csv_filename)
output_file_path = os.path.join(output_dir, csv_filename)

if os.path.exists(csv_file_path):
extract_keywords_and_write(txt_file_path, csv_file_path, output_file_path, estonian_stopwords)

```

Start with the Python script in the issue and trace its inputs from estonian-stopwords.txt, raw_text, and pre_processed_text_data into the keybert output directory. Reproduce the current KeyBERT and EstBERT setup, then document whether it works and what comparisons or preprocessing guidance are still needed for the listed models and methods.

Written by the indexing model from the issue text.