[BUG] Int8 Inference Does Not Work For GPTJ

Describe the bug
Trying to use DeepSpeed Inference with int8 does not work for GPTJ. I get created an issue that has more details on the DeepSpeed MII repo, but due to the nature of the issue, I feel it likely belongs here.

https://github.com/microsoft/DeepSpeed-MII/issues/155

To Reproduce
Steps to reproduce the behavior:

1. Load GPTJ with float16

model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-j-6B",torch_dtype=torch.float16).cuda()

2. Load model with DeepSpeed

world_size = 1
dtype = torch.int8
engine = deepspeed.init_inference(model,
                                   mp_size=world_size,
                                    dtype=dtype,
                                    replace_method='auto',
                                    max_tokens=2048,
                replace_with_kernel_inject=True)

3. Try to generate tokens
4. Alternatively, use DeepSpeed MII for the same issue

Expected behavior
I expect a memory reduction, and speed improvement, for little or no degradation in performance
ds_report output

DeepSpeed C++/CUDA extension op report

NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.

JIT compiled ops requires ninja
ninja .................. [OKAY]

op name ................ installed .. compatible

async_io ............... [YES] ...... [OKAY]
cpu_adagrad ............ [YES] ...... [OKAY]
cpu_adam ............... [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [YES] ...... [OKAY]
quantizer .............. [YES] ...... [OKAY]
random_ltd ............. [YES] ...... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
spatial_inference ...... [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
utils .................. [YES] ...... [OKAY]

DeepSpeed general environment info:
torch install path ............... ['/usr/local/lib/python3.8/dist-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/usr/local/lib/python3.8/dist-packages/deepspeed']
deepspeed info ................... 0.8.0+bf6b9802, bf6b9802, HEAD
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7

System info (please complete the following information):

• OS: Ubuntu 20.04
• Two RTX 3090s
• See the linked issue for DeepSpeed MII info

Docker context is very similar but not identical to this

pip list:
Package Version

accelerate 0.16.0
aiohttp 3.8.4
aiosignal 1.3.1
anyio 3.6.2
async-timeout 4.0.2
asyncio 3.4.3
attr 0.3.2
attrs 22.2.0
bitsandbytes-cuda117 0.30.1
certifi 2022.12.7
charset-normalizer 3.0.1
click 8.1.3
coloredlogs 15.0.1
datasets 2.10.1
deepspeed 0.8.0+bf6b9802
deepspeed-mii 0.0.5+bb801d3
dill 0.3.6
evaluate 0.4.0
fastapi 0.89.1
filelock 3.9.0
flatbuffers 23.3.3
frozenlist 1.3.3
fsspec 2023.3.0
grpcio 1.51.3
grpcio-tools 1.51.3
h11 0.14.0
hjson 3.1.0
huggingface-hub 0.12.0
humanfriendly 10.0
idna 3.4
markdown-it-py 2.1.0
mdurl 0.1.2
mpmath 1.2.1
multidict 6.0.4
multiprocess 0.70.14
nest-asyncio 1.5.6
ninja 1.11.1
numpy 1.24.2
nvidia-cublas-cu11 11.10.3.66
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11 8.5.0.96
nvidia-pyindex 1.0.9
onnx 1.13.1
onnxruntime 1.14.1
onnxruntime-gpu 1.14.1
optimum 1.7.1
packaging 23.0
pandas 1.5.3
Pillow 9.4.0
pip 20.0.2
polygraphy 0.44.2
protobuf 3.20.2
psutil 5.9.4
py-cpuinfo 9.0.0
pyarrow 11.0.0
pydantic 1.10.4
Pygments 2.14.0
python-dateutil 2.8.2
pytz 2022.7.1
PyYAML 6.0
redis 4.5.0
regex 2022.10.31
requests 2.28.2
responses 0.18.0
rich 13.3.1
sentencepiece 0.1.97
setuptools 45.2.0
six 1.16.0
sniffio 1.3.0
starlette 0.22.0
sympy 1.11.1
timm 0.6.12
tokenizers 0.13.2
torch 1.13.1
torchaudio 0.13.1
torchvision 0.14.1
tqdm 4.64.1
transformers 4.26.0
triton 1.0.0
typing-extensions 4.4.0
Unidecode 1.3.6
urllib3 1.26.14
uvicorn 0.20.0
wheel 0.34.2
xxhash 3.2.0
yarl 1.8.2

Additional context
I may be mistaken and this was never supposed to work outside of the box. Perhaps MoQ is required?