Experiment - Running on AWS EC2 Graviton c6g.xlarge

Hi folks! First of all, congrats on this fantastic research result!

I managed to run Llama3-8B-1.58-100B-tokens on a relatively small Amazon EC2 instance.
Instance Family: c6g (ARM)
Instance Size: xlarge
Architecture: ARM 64
Specs: 4v CPUs, 8GiB RAM
OS: Ubuntu
Cost: 0.1360 USD per hours in us-east-1

https://github.com/user-attachments/assets/db244081-4a83-4923-8cba-62768723c975

Build

I built the model on a larger instance (r7g.4xlarge), then made an AMI and ran it on a smaller machine. It took 21 minutes to build the model.

Run

python run_inference.py -m models/Llama3-8B-1.58-100B-tokens/ggml-model-i2_s.gguf -p "Daniel went back to the the the garden. Mary travelled to the kitchen. Sandra journeyed to the kitchen. Sandra went to the hallway. John went to the bedroom. Mary went back to the garden. Where is Mary?\nAnswer:" -n 6 -temp 0 -t 4

Notes

It took about 10 minutes to load the model for the first time, but after that I can run inference at will. Is this expected?

I had to use an experimental fork of this repo (#79), as I was experiencing the issue described in #74

All the best!

Start with run_inference.py and the reported c6g.xlarge setup, then compare the first-load behavior with the larger r7g.4xlarge build environment. Review issue #74 and the experimental fork referenced as #79. Done means establishing whether the ten-minute initial load is expected and recording the relevant ARM/AWS findings.