GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature.

Editor's take

The reported 1.8 trillion parameter count for GPT-4 is not a measure of its active model size, but rather the total capacity of its parameter dictionary, with only a sparse subset utilized per token inference. This distinction is crucial, suggesting a Mixture-of-Experts (MoE) architecture, a technique previously explored by Google in models like Switch Transformer, allowing for greater efficiency and scalability.

This approach matters significantly for the economics and deployment of large language models. By activating only a fraction of its parameters, GPT-4 could achieve performance comparable to denser models while drastically reducing computational costs during inference. This is especially relevant as companies like OpenAI and Google continue to push the boundaries of LLM capabilities, facing increasing pressure to optimize for real-world application and energy consumption.

Future developments will hinge on the actual performance gains and cost reductions this MoE design affords compared to traditional dense architectures. Further understanding of the gating mechanisms and the specific "experts" within GPT-4 will be key. Moreover, observing how competitors like Meta's Llama series adapt their own scaling strategies in response will provide crucial insights into the prevailing architectural trends in LLM development.