How Much Does a Local LLM Actually Cost to Run? I Measured Every Watt on Apple Silicon

Editor's take

Running open-source large language models locally on Apple Silicon, specifically models like Llama 2 and Mistral, consumes a tangible amount of electricity, with sustained inference showing surprisingly high energy draw comparable to, and exceeding, dedicated high-end NVIDIA GPUs like the RTX 3090 when factoring in real-world electricity costs.

This analysis is significant because it quantifies the often-overlooked operational expense of deploying LLMs outside of cloud environments, impacting individuals and organizations considering on-premise AI solutions. The findings challenge the perception that local inference is inherently cheap, especially as model sizes grow, and highlight the importance of energy efficiency in hardware design for AI workloads.

Future investigations should focus on the power efficiency of emerging Apple Silicon chips and the impact of quantization techniques on both performance and energy consumption for these models. Additionally, understanding how different inference optimization libraries (e.g., llama.cpp, MLC LLM) affect energy usage will be crucial for optimizing local LLM deployments.