Inference issues on Volta-based swarm

Hello folks,

Trying to run a private swarm on a 7x Volta-generation GPUs. As suggested by docs, i've set torch_dtype to float16 and NUM_BLOCKS to 10 (these are 32GB GPUs) and removed load-8-bit argument. All 7 are running on the same linux host.

Swarm starts ok and loads all the model blocks, but many of included tests fail.

Trying even the most basic generation seems to always generate the same token (0 == UNK)

Are older GPUs even supported? There are some notes in documentation on what to set for pre-Turing, but the arxiv paper says the server needs to have Turing or later gen GPU.

If older GPUs are supported, do i also need to specify the torch_dtype to be float16 on instantiating model?
(i get RuntimeError: "LayerNormKernelImpl" not implemented for 'Half' when running .generate() in this case)

It is torch 1.12.1+cu113 on cuda 11.3

This is what i get as tests:

tests/test_aux_functions.py::test_throughput_basic FAILED <-- this system is behind proxy, i think this is expected
tests/test_block_exact_match.py::test_remote_block_exact_match FAILED
tests/test_chained_calls.py::test_forward_backward_exact_match FAILED
tests/test_chained_calls.py::test_chained_inference_exact_match FAILED
tests/test_full_model.py::test_full_model_exact_match[True] FAILED
tests/test_full_model.py::test_full_model_exact_match[False] FAILED
tests/test_full_model.py::test_greedy_generation PASSED
tests/test_full_model.py::test_sampling[sampling_options0] SKIPPED (Sampling is currently not consistent with outputs from Transformers)
tests/test_full_model.py::test_sampling[sampling_options1] SKIPPED (Sampling is currently not consistent with outputs from Transformers)
tests/test_full_model.py::test_sampling[sampling_options2] SKIPPED (Sampling is currently not consistent with outputs from Transformers)
tests/test_full_model.py::test_sampling[sampling_options3] SKIPPED (Sampling is currently not consistent with outputs from Transformers)
tests/test_full_model.py::test_beam_search_generation FAILED
tests/test_linear8bitlt.py::test_layout_exact_match SKIPPED (this test requires a turing-generation or newer GPU, see bitsandbytes docs)
tests/test_linear8bitlt.py::test_linear_exact_match SKIPPED (this test requires a turing-generation or newer GPU, see bitsandbytes docs)
tests/test_linear8bitlt.py::test_linear_no_igemmlt PASSED
tests/test_priority_pool.py::test_priority_pools PASSED
tests/test_remote_sequential.py::test_remote_sequential FAILED
tests/test_remote_sequential.py::test_remote_sequential_prompts FAILED

The greedy search test seems to pass, but i'm suspicious... could it be an issue with a test?

Here is what i see from .generate():

model = DistributedBloomForCausalLM.from_pretrained("bigscience/bloom-petals", initial_peers=INITIAL_PEERS)
inputs = tokenizer("Cat sat on", return_tensors="pt")["input_ids"]
outputs = model.generate(inputs, max_new_tokens=5)
print(tokenizer.decode(outputs[0]))
Cat sat on
type(outputs), outputs.shape, outputs
(torch.Tensor,
torch.Size([1, 8]),
tensor([[40171, 13770, 664, 0, 0, 0, 0, 0]]))

If needed, can provide the log from tests.