codeforboston / codeforboston/maple
Upgrade LLM used in summarization pipeline
- Dominant language
- TypeScript
- Stars
- 56
- Forks
- 175
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 13
Description
**Summary**
We are currently using an ancient GPT4 model in the automated summarization and tagging pipeline. it should be updated to a more recent model, or abstracted to automatically update to a later model.
**Work Detail**
In a perfect world, this issue would include,
* Migration to latest low cost model and confirmation there is no performance regression.
* Evaluating and/or actually migrating from OpenAI models (as originally used) to Google models (now used primarily in new feature dev)
* Implementation of automated evals to help us detect when migrating to a new LLM introduces substantive performance issues across summarization accuracy, length, reading level, or bias.
* Only apply to new summaries going forward; should not regenerate summaries already in the database.
Contributor guide
Research direction
Start by locating the automated summarization and tagging pipeline and the current GPT4 integration. Compare the existing OpenAI usage with the Google models used in newer feature development, then define evaluation coverage for accuracy, length, reading level, and bias. Done means a newer or automatically updated model is used for new summaries without regenerating existing database records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100