a2ui-project / a2ui-project/a2ui

[FEATURE]: Make the express format perform well with Gemma 2B models

Aperta
#2,569 1 commento 0 reazioni 1 assegnatario Rivendicata da @jacobsimionato Vedi su GitHub
P2
Lingua principale
TypeScript
Stelle
16.4k
Fork
1.3k
Merge medio
2g 13h
PR unite (30g)
134

Descrizione

https://github.com/a2ui-project/a2ui/pull/2551 proposes a new "Vertical" inference format which is faster and more accurate, especially on Gemma models.

Vertical performs way better than Express. But, potentially we could update the Express system prompt to be more descriptive, or make the compiler more permissive, or change the format in some way to avoid the types of errors that we see.

Steps
- Use the new eval cases added in https://github.com/a2ui-project/a2ui/pull/2551 and evaluate the same Gemma 2B model with Express format
- Brainstorm different changes we could make to the express prompt or parser, and see if they fix the issues. We can just do small runs, e.g. 5-10 data points to verify this at first. Let's prefer simple tweaks e.g. to the prompt first, before making more radical changes the compiler or format itself.
- Prepare a report on the different approaches that are possible, and how effective they are. They could be implemented as "options" on the express format perhaps. At this point, consider doing larger eval runs with 2-5 runs per data point to build confidence that the fixes work well.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.