get-convex / get-convex/dryad

Use tiktoken (or some other means) to limit documents instead of `byteSize`

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
TypeScript
Stars
40
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Currently, Dryad uses a byte count (file size) to avoid passing too-large source files to ChatGPT for summarization. This is both inefficient and inaccurate. It would be best to know **exactly** how many tokens a given source file would be so we can more accurately limit prompt payloads based on the capability of whatever chat model we're using.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.