huggingface / huggingface/transformers

Siamese Multi-depth Transformer-based Hierarchical Encoder

Open
#9,526 3 comments 25 reactions 0 assignees View on GitHub
Feature request New model
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 8h
Merged PRs (30d)
276

Description

# 🌟 New model addition

## Model description
Recently Google is published paper titled ["Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching"](https://arxiv.org/abs/2004.12297). And according to paper for long-form document matching SMITH model outperforms the previous state-of-the-art models including hierarchical attention, multi-depth attention-based hierarchical recurrent neural network, and BERT.

I feel it is will add value to already awesome transformers models collection :slightly_smiling_face:

## Open source status

* [X] the model implementation is available: https://github.com/google-research/google-research/tree/master/smith
* [X] the model weights are available: [SMITH-WP+SP model checkpoint](http://storage.googleapis.com/gresearch/smith_gwikimatch/smith_wsp_pretrain_ckpt_opensource.zip) and [GWikiMatch data](http://storage.googleapis.com/gresearch/smith_gwikimatch/gwikimatch_open_source.zip)
* [X] who are the authors: https://github.com/yangliuy, https://github.com/eladeban

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.