imgsys

Implements a competitive ranking system that evaluates multiple generative image models (e.g., DALL-E, Midjourney, Stable Diffusion, etc.) against identical prompts through crowdsourced or automated preference voting. The arena architecture collects user votes on side-by-side image outputs, aggregates preference signals, and maintains a dynamic leaderboard that ranks models by win-rate and Elo-style scoring. This enables real-time performance tracking across model versions and providers without requiring direct model access or inference infrastructure.

Provides a unified interface to submit text prompts and receive generated images from multiple underlying generative models (DALL-E, Midjourney, Stable Diffusion, etc.) through fal.ai's inference orchestration layer. The system routes requests to appropriate model endpoints, handles authentication/API key management for each provider, and returns standardized image outputs. This abstracts away provider-specific API differences and enables easy model switching without client-side code changes.

Continuously ingests user preference votes on image pairs, applies Elo-style ranking algorithms to update model scores, and publishes live leaderboard updates to the web interface with minimal latency. The system maintains vote history, handles tie-breaking logic, and recomputes rankings incrementally as new votes arrive rather than batch-processing, enabling real-time score visibility. Vote data is persisted and queryable for historical analysis and trend detection.

Maintains a curated set of standardized prompts across diverse categories (e.g., portraits, landscapes, abstract art, text rendering, specific objects) that are used consistently across all model evaluations in the arena. These prompts are designed to probe different model capabilities and reduce variance from prompt engineering. The system may include prompt templates, difficulty ratings, and category tags to enable stratified analysis of model performance across capability dimensions.

Collects and aggregates inference latency, API response times, and cost-per-image metrics across different generative image models and providers. The system tracks these metrics alongside quality rankings, enabling users to make cost-benefit tradeoffs when selecting models. Latency data is collected from actual inference requests, and cost data is sourced from provider pricing APIs or manual configuration. Results are displayed as a multi-dimensional leaderboard that can be sorted by quality, speed, or cost.