codeforboston / codeforboston/maple
Grouping templated testimony
- Dominant language
- TypeScript
- Stars
- 56
- Forks
- 175
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 13
Description
First draft of project requirements:
1. Deliver a python function that accepts a list of strings, representing testimony documents, and returns a list of integers representing what document groups the documents belong to (or if they are unique)
2. Expectation is that we will implement this using LLM embeddings, with an automated workflow that 1) embeds each document, 2) does 2D dimensionality reduction, 3) clusters 2D embeddings using agglomerative clustering (e.g. hdbscan)
3. Do some analysis to decide how to extract appropriate clusters from the hdbscan dendogram, e.g. setting appropriate tree 'height' parameter
4. Develop a testining dataset to test and optimise step # 3, e.g. a public comment dataset (e.g. FTC net neutrality comments) where there are clear document clusters we can use for evaluatiopn
Contributor guide
Assessment
This issue has not been assessed yet.