Github

Converts PDF, PNG, and JPEG documents into clean markdown and structured text using a distributed worker architecture backed by S3 or local file-based work queues. The pipeline orchestrates page-level processing through a queue system that coordinates multiple worker processes, each invoking a fine-tuned 7B vision-language model (olmOCR-2-7B based on Qwen2.5-VL) via vLLM server instances. Workers pull tasks from the queue, process pages with rotation correction and layout analysis, and write results back to persistent storage, enabling horizontal scaling across machines.

Automatically detects and corrects page rotation by invoking the vision-language model on each page image to determine correct orientation before full OCR processing. The system analyzes visual cues (text direction, layout coherence) through the VLM to identify if a page is rotated 0°, 90°, 180°, or 270°, then applies geometric transformations to normalize orientation before downstream text extraction. This pre-processing step improves downstream OCR accuracy by ensuring consistent text direction.

Applies data augmentation techniques (rotation, scaling, noise injection, color jittering) to training images and filters low-quality training examples based on heuristics (image blur, text clarity, layout complexity). The augmentation pipeline increases training data diversity, improving model robustness to document variations. Filtering removes corrupted or low-quality examples that would degrade training, focusing compute on high-quality data.

Provides runners and evaluation harnesses for comparing olmOCR against competing OCR systems (Tesseract, NanoNets, Google Vision, etc.) on standardized benchmarks. The framework converts outputs from different OCR systems to a common format, applies the same evaluation metrics, and generates comparison reports. This enables fair comparison across systems with different output formats and capabilities.

Generates OCR output in Dolma format (structured JSON with document metadata, page-level information, and extracted text), enabling integration with downstream document processing pipelines and training data generation. The format preserves metadata including page numbers, source document paths, processing timestamps, and quality scores. This structured output enables filtering, sorting, and analysis of OCR results at scale.

Analyzes document page layouts to identify multi-column regions and reconstructs natural reading order by processing spatial coordinates of text blocks extracted by the VLM. The system groups text elements by column position, sorts them top-to-bottom within columns, then merges columns left-to-right to produce markdown output that follows the intended document flow. This capability handles complex layouts including figures, insets, and mixed single/multi-column pages.

Extracts mathematical equations and tables from document pages and formats them as LaTeX (for equations) or HTML/Markdown (for tables) within the output markdown. The VLM recognizes equation regions and table structures, then generates appropriate markup that preserves mathematical notation and tabular relationships. Equations are rendered as inline or block LaTeX, while tables are converted to HTML or Markdown table syntax, maintaining semantic structure for downstream processing.

Automatically detects and removes headers and footers from document pages by classifying text regions as header/footer/body content using spatial position heuristics and VLM-based content analysis. The system identifies text appearing consistently at the top or bottom of pages (page numbers, running titles, repeated metadata) and excludes it from the final markdown output. This improves readability by eliminating repetitive non-content text.

Converts PDF pages to high-quality PNG or JPEG images at configurable DPI (typically 150-300 DPI) using PyPDF2 or similar libraries, with optional filtering to skip blank or low-quality pages. The system renders each page as a raster image suitable for VLM inference, applying quality checks to detect and optionally skip pages that are blank, corrupted, or contain only images without text. This preprocessing ensures only processable pages are sent to the VLM, reducing wasted inference compute.

Provides a benchmarking framework (olmOCR-Bench) that evaluates OCR quality across 7,000+ test cases covering 1,400 documents, with automated synthetic test case generation for equations (via KaTeX rendering), tables (via HTML rendering), and handwriting. The system compares olmOCR output against ground truth using metrics like character error rate (CER), equation accuracy, and table structure preservation. Test cases are mined from real documents and augmented with synthetic variations to ensure comprehensive coverage of edge cases.

Implements supervised fine-tuning (SFT) of the base Qwen2.5-VL model on document OCR tasks using training data generated from the benchmarking system and augmented with synthetic variations. The training pipeline loads document images and ground truth markdown outputs, applies data augmentation (rotation, scaling, noise), and optimizes the model using standard cross-entropy loss on token prediction. Fine-tuning is performed on Beaker distributed training infrastructure, enabling multi-GPU training across multiple machines.

Implements Group Relative Policy Optimization (GRPO) reinforcement learning to optimize the fine-tuned model for OCR quality metrics (character error rate, equation accuracy, table F1 score) beyond supervised fine-tuning. The system uses the benchmarking framework to generate reward signals based on OCR output quality, then applies GRPO to adjust model weights to maximize these rewards. This enables the model to learn from its own errors and improve on metrics that matter for downstream applications.

Orchestrates distributed model training across Beaker clusters, managing multi-GPU training jobs, data distribution, and checkpoint synchronization. The system submits training jobs to Beaker with specified resource requirements (GPU count, memory), distributes training data across workers, and coordinates gradient synchronization using PyTorch's DistributedDataParallel. This enables efficient scaling of training from single-GPU to multi-GPU setups without code changes.