Fine-tuning GPT-J-6B in colab: 8-bit weights with low-rank adaptors

# 🌟 New model addition

## Model description

This is a version of EleutherAI's GPT-J with 6 billion parameters that is modified so you can generate and fine-tune the model in colab or equivalent desktop GPU (e.g. single 1080Ti).

The original GPT-J takes 22+ GB memory for float32 parameters. Even if you cast everything to 16-bit, it will still not fit onto most single-GPU setups short of A6000 and A100. You can inference it on TPU or CPUs, but fine-tuning is way more expensive.

## Implementation
Proof-of-concept notebook is available here: [](https://colab.research.google.com/drive/1ft6wQU0BhqG5PRlwgaZJv2VukKKjU4Es?usp=sharing#scrollTo=P8Y75B6WDIN-)

[Model card](https://huggingface.co/hivemind/gpt-j-6B-8bit) has more detailed explanations and auxiliary notebooks (e.g. model conversion and perplexity check).

The current implementation is somewhat hacky, but it can be integrated easily with modelling_gptj.py if you like the idea.

## Open source status

* the model implementation is available [here](https://colab.research.google.com/drive/1ft6wQU0BhqG5PRlwgaZJv2VukKKjU4Es)
* the model weights are available [here](https://huggingface.co/hivemind/gpt-j-6B-8bit)
* who are the authors:
- the [original GPT-J-6B](https://huggingface.co/EleutherAI/gpt-j-6B) was trained by [Eleuther AI](https://www.eleuther.ai/) (citation: Ben Wang and Aran Komatsuzaki)
- fast quantization from [bitsandbytes](https://github.com/facebookresearch/bitsandbytes) by [Tim Dettmers](https://github.com/TimDettmers)
- low-rank adapters were proposed for GPT-like models by [Hu et al (2021)](https://arxiv.org/abs/2106.09685)
- this notebook was created by me ( @deniskamazur ) with some help from Yozh ( @justheuristic)