Running artificial intelligence models on personal computers has become more accessible, but video memory (VRAM) still imposes considerable limits on users. Strategies such as quantization, FlashAttention, KV cache optimization, and CPU offloading offer ways to maximize the use of an existing GPU, although there are no universal rules. A 13-billion parameter model does not necessarily require the same amount of memory every time, and reducing layers does not guarantee maintaining model quality.
With the popularization of open-source models and tools like llama.cpp, Ollama, and vLLM, the possibility of running language models without depending on external APIs has expanded. However, many users still face challenges when trying to operate models that exceed the GPU's memory capacity before even generating the first token.
Quantization has proven to be one of the most effective techniques for running large models on systems with limited resources, reducing memory consumption by storing weights using fewer bits. Alternatives like GPTQ and AWQ offer different degrees of quantization, but reducing precision can lead to quality loss depending on the task. Additionally, the context length a model maintains in its KV cache also significantly impacts VRAM consumption, and optimizations like FlashAttention can help reduce this consumption.
With 8 GB GPUs, many small and medium models can be run efficiently, while configurations of 12 or 16 GB allow greater flexibility. True optimization goes beyond the number of parameters or available VRAM, involving the model architecture, operating mode, and overall system efficiency. In corporate environments, inference infrastructure must consider VRAM, CPU, data flow, and energy consumption to ensure optimal performance.
