MoonshotAI / MoonshotAI/FlashKDA
gate mode diff
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 1.3k
- Forks
- 122
- PR merge metrics
- No merged PRs in 30d
Description
Would you consider supporting softplus as an alternative activation mode (e.g., via a flag like gate_mode: "lower_bound" | "softplus")?
And, I tested fla's chunk_kda vs FlashKDA, in end to end result, lower_bound gate mode will cause repeatition in model's output.Both fla's chunk_kda with lower_bound gate mode and FlashKDA have the same problem, but when use fla's chunk_kda with softplus gate mode, it works fine.
the model used is kimi linear (https://modelscope.cn/models/moonshotai/Kimi-Linear-48B-A3B-Instruct).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the chunk_kda and FlashKDA entry points and comparing how their gate modes are handled. Reproduce the reported repetition with Kimi Linear, then establish how softplus mode should be selected and verify that the end-to-end output no longer repeats.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100