Reexpress

Implements a trained SDM estimator that compares LLM responses against a database of 120,159+ verified examples from the OpenVerification dataset to produce statistically calibrated confidence scores. The estimator extracts similarity, distance, and magnitude features from response pairs and maps them to high-reliability regions (≥90%, ≤89%, <60%, or Out-of-Distribution) using offline calibration at α=0.9, enabling principled confidence estimation without ground-truth labels.

Automatically routes each LLM response to three independent verification models (GPT-5.2 via Azure/OpenAI, Gemini-3-Pro via Google, and local Granite-3.3-8B) in parallel or sequential mode, aggregates their outputs, and feeds the ensemble results to the SDM estimator. This architecture isolates verification from the primary LLM, reducing bias and enabling cross-model consistency checks.

Implements a unified API abstraction for calling three LLM providers (OpenAI/Azure GPT-5.2, Google Gemini-3-Pro, local Granite-3.3-8B) with consistent request/response handling, error recovery, and rate limiting. The layer handles provider-specific authentication, request formatting, and response parsing, allowing the SDM estimator to treat all three models as interchangeable verification backends.

Implements a centralized configuration system that manages SDM estimator hyperparameters, file access control rules, LLM provider credentials, and calibration thresholds. Configuration is loaded from environment variables, YAML files, or Python constants, enabling deployment-specific customization without code changes. Includes validation and default values for all configuration options.

Implements storage and retrieval of trained SDM models, calibration curves, training datasets, and feedback buffers using a file-based or database backend. Includes versioning of model artifacts, checkpointing during training, and recovery from incomplete training runs. Supports both local file storage and cloud storage backends (S3, GCS).

Enables LLM clients to use SDM verification as a reasoning tool within multi-step task decomposition workflows. The LLM can call reexpress_verify to check intermediate results, adjust reasoning based on confidence levels, and request re-verification if confidence is low. This creates a feedback loop where verification guides task decomposition and error recovery.

Provides three MCP tools that allow users to incrementally update the SDM estimator with feedback without full retraining: reexpress_add_true marks a response as correct, reexpress_add_false marks it as incorrect, and reexpress_add_ood flags it as out-of-distribution. These tools update an in-memory feedback buffer that can be periodically flushed to the training dataset, enabling the estimator to adapt to domain-specific patterns over time.

Implements offline calibration of the SDM estimator using empirical calibration curves at α=0.9, mapping SDM feature vectors to discrete confidence regions: ≥90% (high confidence), ≤89% (medium confidence), <60% (low confidence), or Out-of-Distribution. Calibration is performed once during training and stored as lookup tables or decision boundaries, enabling fast inference without per-query calibration overhead.

Implements a Model Context Protocol (MCP) server that exposes SDM verification and feedback tools as callable functions for LLM clients. The server includes a file access control system that restricts which files or directories the LLM can access during verification, a dynamic tool registry for managing available tools, and request/response serialization compatible with Claude Opus/Sonnet and other MCP-compliant clients.

Implements a multi-stage training pipeline that prepares data from the OpenVerification1 dataset, applies iterative shuffling to reduce overfitting, trains the SDM estimator on similarity/distance/magnitude features, and evaluates performance using calibration metrics. The pipeline includes data validation, feature engineering, and hyperparameter tuning stages, with checkpointing to enable resumable training.

Implements offline calibration by fitting empirical curves to the relationship between SDM features and response correctness, then mapping feature space to discrete high-reliability regions (≥90%, ≤89%, <60%, OOD). Calibration uses the training dataset to compute confidence intervals and decision boundaries, stored as lookup tables or parametric models for fast inference.

Extracts three classes of features from response pairs: similarity features (semantic overlap between responses), distance features (divergence in key attributes), and magnitude features (response length, complexity, token count). Features are computed using embedding-based similarity (cosine distance in embedding space), string-based distance metrics (edit distance, token overlap), and response metadata, then normalized and fed to the SDM estimator.

Implements comprehensive evaluation of the SDM estimator using calibration-specific metrics: expected calibration error (ECE), Brier score, AUC-ROC, and reliability diagrams. Evaluation is performed on held-out test sets to assess generalization, and includes per-confidence-region metrics to validate that each region (≥90%, ≤89%, <60%, OOD) achieves its target reliability.

Provides interactive visualizations of SDM features and confidence regions, including scatter plots of similarity/distance/magnitude features colored by confidence level, reliability diagrams showing confidence vs. accuracy, and out-of-distribution detection visualizations. Visualizations enable exploration of why specific responses are classified as high/medium/low confidence or out-of-distribution.