Photoroom PRX Data Playbook: JPEG 92 Storage and Lance-MDS Sharding
Photoroom shared its data pipeline details for training the 7B PRX multimodal model, demonstrating that JPEG 92 storage is practically indistinguishable from PNG while being 3-10x smaller, and utilizing Lance for exploration paired with Mosaic MDS for streaming.
Why it matters
Shows how to optimize storage for massive image datasets without sacrificing model quality, while balancing interactive data engineering (Lance) with efficient training throughput (MDS).
TL;DR
- 01JPEG 92 is a storage-efficient, high-quality alternative to PNG for pre-training.
- 02Lance is superior for interactive dataset exploration, while MDS is ideal for high-throughput streaming.
- 03Computing text latents on-the-fly provides flexibility at a minor throughput cost (3-4%).
Decoupling Exploration and Streaming: Lance + MDS
Photoroom uses a hybrid data setup:
- Lance: A columnar data format optimized for interactive feature engineering, vector search, and querying billions of rows.
- Mosaic MDS: Used for streaming dataset shards directly from storage to SLURM nodes. Since MDS is rigid, Lance handles the curation and preparation.
Lossless PNG vs. JPEG 92
Photoroom evaluated JPEG 92 against PNG:
- Imperceptible Degeneration: JPEG 92 converges quickly, and repeated cycles remain in the imperceptible range. PNG files are 3x to 10x larger with no perceptual training gains.
- Quantization Test: Generations from models trained on PNG vs. JPEG 92 were indistinguishable, with only about 1 in 10 generations from either model showing detectable quantization structure.
On-the-Fly Text Latents with Qwen3-VL
Computing latents on-the-fly with Qwen3-VL added only a 3-4% throughput cost (approx. 1 extra day on a 30-day run) but allows for changing the text encoder without rewriting terabytes of latents and enables storage on local SSDs.