Unweight: how we compressed an LLM 22% without sacrificing quality

Unweight is a lossless compression system reducing LLM weights by 15-22% while preserving exact outputs, addressing GPU memory bandwidth bottlenecks in inference. - Top 16 exponent values in BF16 weights cover 99% of weights in typical layers, enabling effective Huffman coding compression. - Weights are decompressed in fast on-chip shared memory and fed directly to tensor cores, avoiding slow main memory round-trips. - System provides four execution pipelines with different tradeoffs between decompression and computation, automatically selected per batch size. - On Llama-3.1-8B: ~30% MLP weight compression, 3GB VRAM savings, enabling more models per GPU.

This summary was automatically generated by AI based on the original article and may not be fully accurate.