codeforboston / codeforboston/maple

Grouping templated testimony

Open
#1,094 0 comments 0 reactions 0 assignees View on GitHub
Epic
Dominant language
TypeScript
Stars
56
Forks
175
Avg merge
2d 5h
Merged PRs (30d)
13

Description

First draft of project requirements:

1. Deliver a python function that accepts a list of strings, representing testimony documents, and returns a list of integers representing what document groups the documents belong to (or if they are unique)
2. Expectation is that we will implement this using LLM embeddings, with an automated workflow that 1) embeds each document, 2) does 2D dimensionality reduction, 3) clusters 2D embeddings using agglomerative clustering (e.g. hdbscan)
3. Do some analysis to decide how to extract appropriate clusters from the hdbscan dendogram, e.g. setting appropriate tree 'height' parameter
4. Develop a testining dataset to test and optimise step # 3, e.g. a public comment dataset (e.g. FTC net neutrality comments) where there are clear document clusters we can use for evaluatiopn

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.