UC Berkeley's LEANN Vector Index Shrinks a 60M-Chunk RAG Datastore From 201 GB to 6 GB

Researchers at UC Berkeley’s Sky Computing Lab have open-sourced LEANN, a storage-compressed vector index accepted at MLSys 2026 that reduces the disk and memory footprint of a 60-million-chunk retrieval-augmented generation datastore from 201 GB down to under 6 GB. Built on top of FAISS and released under an MIT license, the engine achieves a 30x to 50x storage reduction by discarding raw dense float embeddings entirely and recomputing exact vectors on the fly for only the small subset of nodes visited during graph traversal.

In standard production RAG architectures, the vector index regularly consumes two to three times more storage than the underlying text corpus itself. Indexing a 76 GB English Wikipedia snapshot of 60 million passages using standard 768-dimensional float32 embeddings requires 173 GB just to store the embedding vectors, plus an additional 15 GB for the Hierarchical Navigable Small World (HNSW) graph pointers—bringing the total footprint to between 188 GB and 201 GB. That memory expansion forces enterprises either to pay for expensive high-RAM cloud instances or to offload sensitive corporate data to third-party hosted vector databases.

LEANN eliminates that storage multiplier by observing that a single HNSW search query touches only a tiny fraction of the total graph nodes. Instead of persisting every dense vector in memory or on NVMe drives, LEANN stores a pruned graph topology alongside a lightweight Product Quantization (PQ) lookup table. When a query arrives, the engine navigates the graph using cheap quantized distance estimates and dynamically batches only the highest-scoring candidate passages for live GPU re-embedding.

Because modern GPU tensor cores are frequently idle during the retrieval phase of a local RAG pipeline—waiting for the CPU or disk to return context chunks before LLM prefill begins—LEANN converts unused GPU compute cycles into massive storage savings. On an NVIDIA RTX 4090 workstation with 32 GB of system RAM, LEANN matches the exact 25.5% Natural Questions accuracy of an uncompressed 201 GB HNSW index at 90% Recall@3 while adding just 2.4 seconds to end-to-end generation time.

Fast Facts: UC Berkeley LEANN Vector Index (MLSys 2026)

• Origin & Open-Source Release: Developed by UC Berkeley Sky Computing Lab on top of FAISS, presented at MLSys 2026, and detailed in MarkTechPost’s MLSys 2026 technical review; installable via uv pip install leann under an MIT license.
• Wikipedia 60M-Chunk Compression: Shrinks a 60-million-chunk (76 GB raw text) RAG datastore from 188–201 GB (173 GB embeddings + 15 GB HNSW graph) down to 4–6 GB total (<5% of raw text size).
• Personal & Edge Datastore Footprint: Compresses 400,000 chat messages from 1.8 GB down to 64 MB, and 38,000 browser history records from 130 MB down to 6.4 MB.
• Accuracy Parity: On the Natural Questions (NQ) benchmark using a Qwen3-4B generator and Contriever (110M parameter) retriever, LEANN matches full HNSW at 25.5% downstream answer accuracy at 90% Recall@3 (compared to 18.3% for BM25 and 17.9% for standalone Product Quantization).
• Latency & Speedup Metrics: Two-level dynamic batching across hops (batch size = 64) delivers a 1.8x speedup on RTX 4090 and up to 200x faster search than the EdgeRAG IVF baseline; retrieval takes 2.48s vs. 0.05s for in-memory HNSW, yielding 23.34s vs. 20.95s end-to-end RAG latency.
• Graph Pruning Efficiency: High-degree-preserving pruning retains dense connectivity only on the top 2% hub nodes while reducing average graph out-degree from M=30–64 down to M~12.

Benchmark Comparison: LEANN vs. HNSW, PQ, and BM25 on 60M Chunks

The table below compares index storage requirements, retrieval recall, downstream QA accuracy, and latency across retrieval architectures evaluated on the 60-million-chunk Wikipedia Natural Questions benchmark using an RTX 4090 (32 GB host RAM) with Contriever (110M) and Qwen3-4B.

Architectural Mechanics: Two-Level Search and Hub-Preserving Graph Pruning

LEANN solves two distinct bottlenecks that previously prevented on-demand embedding recomputation from working at scale: graph metadata bloat and GPU batching starvation.

First, even if an engineer deletes the 173 GB of raw float32 embeddings from a 60-million-node HNSW index, the graph structure itself still occupies roughly 15 GB of disk space because standard HNSW maintains 30 to 64 outgoing edges (M=30..64) per node. At the same time, every neighbor check during standard greedy traversal would require running the embedding model on all 30+ neighbors at each hop, saturating the GPU with thousands of redundant forward passes.

To eliminate the graph bloat, the UC Berkeley team introduced high-degree-preserving graph pruning. In navigable small-world graphs, a small minority of “hub” nodes act as long-range highways across the vector space, while the vast majority of nodes serve only local neighborhoods. LEANN preserves dense outgoing edges solely on the top 2% highest-degree hub nodes and aggressively prunes edge counts across the remaining 98% of vertices, bringing the average degree down to approximately 12 (M~12) without breaking multi-hop navigability.

Second, to prevent the embedding model (such as the 110-million-parameter Contriever encoder) from running inefficient single-item forward passes at every graph hop, LEANN implements a two-level hybrid search with cross-hop dynamic batching:

1. Coarse PQ Filtering: Every node retains a tiny Product Quantization code in memory. When visiting a node, LEANN computes approximate PQ distances to all immediate neighbors on the CPU and discards low-scoring branches immediately.
2. Cross-Hop GPU Batching: Only the top fraction of promising candidate nodes are placed into an evaluation queue. Rather than invoking the GPU immediately for two or three candidates at a single hop, the traversal engine continues expanding the frontier across multiple hops until it accumulates a hardware-optimal batch of 64 text chunks. It then executes a single parallelized GPU forward pass to compute exact float32 inner products, yielding a 1.8x throughput speedup on an RTX 4090.

While standalone PQ degrades Natural Questions accuracy from 25.5% down to 17.9% due to quantization distortion, LEANN uses PQ solely as a routing heuristic and relies on exact recomputed embeddings for final ranking—recovering the full 25.5% HNSW accuracy while running up to 200x faster than prior recomputation baselines like EdgeRAG IVF.

Deploying Zero-Embedding-Storage RAG on Edge and Enterprise Nodes

For enterprise AI architects, LEANN changes the unit economics of private retrieval-augmented generation, local developer assistants, and air-gapped compliance search.

The 4-Step Enterprise Implementation Playbook

1. Migrate Cold and Multi-Tenant RAG Indexes to LEANN: In multi-tenant SaaS environments where thousands of customer workspaces maintain isolated vector indexes that are queried sporadically, keeping terabytes of uncompressed HNSW vectors in memory is prohibitively expensive. Convert low-QPS tenant indexes to LEANN via uv pip install leann to cut cloud block-storage and RAM costs by over 95%.
2. Size Retriever Checkpoints to Match Available Tensor Core Headroom: Because LEANN’s 2.48-second retrieval latency depends on fast on-demand encoding of 64-chunk batches, pair the index with compact, high-throughput bi-encoders in the 100M–300M parameter class (such as Contriever-110M, BGE-base, or Arctic-Embed-M) compiled with FP16/BF16 TensorRT or FlashAttention rather than multi-billion-parameter embedding models.
3. Co-Locate Text Chunks on Fast NVMe Page Cache: During a LEANN query, the engine must fetch the raw text strings of the 64 batched candidate nodes from disk so the GPU can re-embed them. Store the raw chunk table in a memory-mapped (mmap) columnar store or LMDB file on local NVMe SSDs so random reads complete in microseconds without blocking the GPU batch queue.
4. Ship Full-Corpus Local Knowledge Bases on Employee Laptops: With a 400,000-message chat archive shrinking from 1.8 GB to 64 MB and a 60M-chunk encyclopedia fitting in 6 GB, security-sensitive organizations can distribute complete internal engineering documentation, codebases, and ticket archives directly onto developer MacBooks and workstation GPUs without relying on centralized cloud vector APIs.