Use tiktoken (or some other means) to limit documents instead of `byteSize`
Open
enhancement
good first issue
- Dominant language
- TypeScript
- Stars
- 40
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Currently, Dryad uses a byte count (file size) to avoid passing too-large source files to ChatGPT for summarization. This is both inefficient and inaccurate. It would be best to know **exactly** how many tokens a given source file would be so we can more accurately limit prompt payloads based on the capability of whatever chat model we're using.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.