How to pack ternary numbers in 8-bit bytes

The contribution, posted on Hacker News, describes a compact representation that stores five trits (the base‑3 analogue of bits) in eight bits, yielding 1.6 bits per trit—just 0.4 % above the information‑theoretic optimum of log₂3 ≈ 1.585 bits. The author implements the scheme in the ggml‑org fork of llama.cpp (pull request #8151) and provides SIMD kernels for both AVX2 and ARM NEON. Packing is performed with a single ceiling division, while unpacking avoids costly integer modulo operations by multiplying the byte by three and extracting the top two bits of the 10‑bit product, a pattern that maps cleanly onto vector units. A C test confirms lossless round‑tripping across all 243 possible 5‑trit values, proving the method works for the entire weight space of BitNet b1.58, a ternary‑weight LLM that has attracted attention for its extreme compression.

This development arrives amid a broader push to shrink large language model (LLM) footprints through low‑precision quantization. While 4‑bit and 8‑bit quantizers dominate current deployments, ternary quantization promises even lower memory bandwidth and storage, at the cost of more complex encoding. By delivering a near‑optimal packing ratio and a SIMD‑compatible unpack path, the technique narrows the performance gap that has kept ternary models largely experimental. It also aligns with the open‑source momentum around llama.cpp, which serves as a de‑facto reference implementation for running LLMs on commodity CPUs and mobile devices. The inclusion of both x86 (AVX2) and ARM (NEON) kernels suggests the authors anticipate widespread adoption across server‑grade and edge hardware, potentially giving ternary models a practical edge over binary or low‑bit floating‑point alternatives.

Looking ahead, the primary risk lies in the limited hardware support for native ternary arithmetic; the scheme still relies on integer multiplication and bit‑shifts, which may introduce latency on older CPUs. Moreover, the packing step’s division, although infrequent, could become a bottleneck in scenarios where weights are frequently re‑quantized, such as during on‑device fine‑tuning. Future work will likely focus on integrating the method into broader quantization toolchains and measuring real‑world throughput gains in inference pipelines. Watch for benchmark releases comparing BitNet b1.58 with 4‑bit quantized models on the same hardware, as those results will determine whether the theoretical density advantage translates into measurable speed or cost savings.