How Much Does It Actually Cost to Run a Local LLM? (Euros per Million Tokens, Measured)

Editor's take

This analysis quantifies the real-world electricity costs of running eight distinct large language models locally on an NVIDIA RTX 3090 GPU, measured in Euros per million tokens. The findings reveal a surprising disconnect between model size and operational expense, with smaller models not always being the most economical and larger ones not always the most prohibitive.

This research is critical for democratizing AI inference, offering tangible data for individuals and smaller organizations considering on-premises LLM deployment. It directly impacts the economic viability of running models like Llama 2, Mistral 7B, or potentially even larger parameter count models within budget constraints, moving beyond theoretical benchmarks to practical, measurable expenditure.

Future investigation should explore how these costs scale across different hardware configurations, particularly with newer generations of GPUs like the RTX 40 series, and how power efficiency improvements in model architectures might shift these cost dynamics. Understanding the impact of inference optimization techniques, such as quantization or specialized hardware accelerators, on these measured costs will also be crucial.