Local LLM setup

Agentic AI can be very useful for specific tasks. It has also become considerably more expensive, with prices rising fast. On the one hand we find organisations tokenmaxxing, using token consumption as a measure of success (see Goodhart's law). On the other hand we find organisations with budget restrictions and/or strict information governance that restricts the use of sensitive data with external services such as AI providers.

What about local LLMs? Can we run them on our local hardware, and if so how do they fare? What's the easiest way to deploy them locally? What kind of hardware is required?

References

• Running local models is good now
• Local Qwen isn't a worse Opus, it's a different tool
• TBC