NeoMME Encoders Deliver 51 Pages per Second Visual Document Retrieval
Hcompany introduced NeoMME, an Apache 2.0 family of 260M and 800M unified multimodal encoders that eliminate separate vision towers and causal decoders. NeoMME-Retriever processes 51 document pages per second on an NVIDIA L40S and shrinks late-interaction index storage by 255×.
Impact: Medium
Why it matters
Replace bloated visual language models in your retrieval pipelines to index high-resolution PDFs directly with 255× smaller storage footprints.
TL;DR
- 01Processes text tokens and 32×32 image patches inside a single bidirectional Transformer without separate vision towers or decoders.
- 02NeoMME-Retriever-260M delivers 51 pages/sec on an L40S GPU and matches ColQwen2.5 on ViDoRe v3 with 14× fewer parameters.
- 03Hierarchical token pooling and asymmetric quantization compress late-interaction vector storage by 255× to 6 kB per page.
Key facts
- Throughput
- 51 pages/sec (260M model at 2048x2048 on NVIDIA L40S, self-reported)
- ViDoRe v3 nDCG@10
- 0.523 (260M model) vs 0.556 (800M model), self-reported
- Index Compression
- 1.5 MB down to 6 kB per page (255x reduction, self-reported)
- Context Length
- 16,384 tokens (up to two 3840x2160 4K UHD images)
- License
- Apache 2.0
Unified Multimodal Architecture
Traditional visual document retrieval adapters bind a separate vision backbone such as SigLIP to an autoregressive language decoder. NeoMME discards both separate stages. Text tokens and raw 32×32 image patches feed directly into one bidirectional Transformer encoder. Because retrieval, ranking, and token classification do not require autoregressive text generation, cutting out the causal decoder removes substantial computational and memory overhead. The encoder stack uses grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings (RoPE), squared-ReLU MLPs, and an alternating attention structure where symmetric sliding-window attention is punctuated by global attention every sixth layer and at the final layer.
Masked-Diffusion Pretraining
NeoMME is pretrained from scratch as a discrete masked-diffusion text denoiser over 524 billion packed input tokens (including 290 billion text-only tokens) using the NorMuon optimizer. While text-only inputs undergo masking rates between 0 and 1, multimodal samples receive corruption rates between 0.3 and 1 while visual patches remain visible. Heavy text corruption forces the encoder to extract semantic context directly from visual evidence rather than relying on language shortcuts.
Dual-Head Retrieval and 255× Index Reduction
NeoMME-Retriever attaches two jointly trained heads to the backbone: a dense head that mean-pools hidden states for approximate nearest neighbor (ANN) search, and a late-interaction head that projects tokens to 128-dimensional normalized vectors. On an NVIDIA L40S GPU, the 260M model encodes 51 pages per second at 2048×2048 resolution—roughly double the throughput of ColModernVBERT. For large production indices, hierarchical token pooling and asymmetric quantization compress multi-vector index footprints from ~1.5 MB to 6 kB per page (255× smaller) while preserving more than 95% of baseline nDCG@10.
Try it in 2 minutes
from transformers import AutoModel, AutoProcessor
# Load NeoMME-Retriever from Hugging Face
model_id = "Hcompany/neomme-retriever-260m"
model = AutoModel.from_pretrained(model_id, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)python
✓ When to use
- Visual document retrieval over complex PDFs featuring tables, diagrams, layouts, and arbitrary fonts without OCR preprocessing.
- High-scale document search where late-interaction index memory overhead must be minimized down to kilobytes per page.
- Hybrid vector search pipelines requiring simultaneous dense retrieval and late-interaction reranking passes.
✕ When NOT to use
- Generative text workflows that require conversational responses, code synthesis, or autoregressive completions.
- Plain text search architectures where standard sub-100M bi-encoders already meet accuracy and latency budgets.
What to do today
- Evaluate NeoMME-Retriever-260M in Hugging Face Transformers to eliminate OCR pre-processing steps on visual PDFs.
- Test dual-stage search: retrieve top document candidates via dense embeddings, then rerank with late-interaction heads.
- Apply hierarchical pooling and asymmetric quantization to scale visual document indexes without ballooning storage.
Sources