allenai / allenai/longformer

Compatibility with latest Huggingface implementation of Longformers?

未关闭
#140 2 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
2.2k
派生
286
PR 合并指标
30 天内没有已合并 PR

描述

Hi! I've been running the `convert_model_to_long.ipynb` and works great with 🤗 transformers 3.0.2.
When upgrading to the latest version of Huggingface (3.5.1), the notebook doesn’t t work and they’ve seem to changed their [implementation](https://github.com/huggingface/transformers/issues/6465#issuecomment-673948643) of the Longformer and LongformerSelfAttention

Is there an easy way change the `RobertaLongSelfAttention` to the current Huggingface implementation or plans for updataing the notebook?
Following [hitskyer](https://github.com/allenai/longformer/issues/129), the dimensions becomes correct, but I don't know how to proceed and change the old forward pass into [This](https://github.com/huggingface/transformers/blob/c89bdfbe720bc8f41c7dc6db5473a2cb0955f224/src/transformers/models/longformer/modeling_longformer.py#L415).

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。