GPT Prompt Engineer

Generates multiple diverse candidate prompts by invoking a designated LLM (CANDIDATE_MODEL) with a task description and test cases as input. The system synthesizes variations automatically rather than requiring manual prompt engineering, using the LLM's generative capacity to explore the prompt space. Each candidate is seeded with different instructions to encourage diversity in approach, tone, and structure.

Tests each candidate prompt against user-provided test cases by executing the prompt with a GENERATION_MODEL and capturing outputs. The system then performs pairwise comparisons between prompt outputs using a RANKING_MODEL to determine which prompt produces better results. This tournament-style evaluation avoids absolute scoring (which is subjective) in favor of relative comparisons, which are more reliable for LLM outputs.

Generates candidate prompts with intentional diversity by seeding the generation model with different instruction styles, tones, and structural approaches. Rather than generating candidates independently, the system explicitly instructs the generation model to create variations that differ in approach (e.g., 'generate a step-by-step prompt', 'generate a direct prompt', 'generate a Socratic prompt'). This ensures the candidate pool explores different solution strategies rather than producing near-duplicates.

Implements a chess-style ELO rating system where each prompt starts at 1200 rating points and gains/loses points based on pairwise comparison outcomes. For each prompt pair and test case, the system updates ratings using a K-factor of 32, meaning each comparison can shift ratings by up to ~32 points depending on expected vs. actual outcome. Final rankings are determined by cumulative ELO scores across all comparisons, providing a mathematically principled ranking that accounts for strength of competition.

Provides multiple pre-configured system variants (Standard GPT, Classification, Claude 3, GPT Planner) that swap out the underlying models and evaluation strategies while maintaining the same core pipeline. Each variant is optimized for different task types: Standard GPT for general tasks, Classification for categorical outputs, Claude 3 for reasoning-heavy tasks, and GPT Planner for multi-step planning. The system abstracts model selection, allowing users to choose a variant matching their task characteristics.

Converts optimized prompts from expensive, high-capability models (e.g., Claude 3 Opus, GPT-4) to cheaper alternatives (e.g., Claude 3 Haiku, GPT-4o-mini) while attempting to preserve task performance. The system uses a dedicated conversion model to rewrite prompts for the target model's capabilities and cost profile. Includes specialized converters for Opus→Haiku, Claude→GPT-4o-mini, Llama 405B→8B, and generic XL→XS conversions.

Orchestrates the entire prompt optimization workflow through a single entry point (generate_optimal_prompt function) that accepts task description, test cases, and configuration parameters. The pipeline is fully configurable: users can specify which models to use for generation, ranking, and candidate synthesis; set the number of candidates and comparison rounds; and define evaluation criteria. The system chains together candidate generation → testing → pairwise evaluation → ELO ranking in a deterministic, reproducible manner.

Integrates with Weights & Biases (W&B) to log and visualize prompt optimization runs, including candidate prompts, test case results, pairwise comparisons, and ELO rankings. The system logs each step of the pipeline to W&B, enabling users to track optimization progress, compare runs across different configurations, and analyze which prompts performed best. This provides observability into the optimization process without requiring custom logging code.

Integrates with Portkey to route LLM API calls across multiple providers (OpenAI, Anthropic, etc.) with automatic failover and load balancing. If one provider is unavailable or rate-limited, Portkey automatically routes requests to an alternative provider without interrupting the optimization pipeline. This enables the system to continue running even if a primary provider experiences outages, improving reliability for long-running optimization jobs.

Converts prompts optimized for instruction-tuned models (e.g., GPT-4, Claude 3) to work with base models (e.g., davinci-003, raw Claude) that lack instruction-following capabilities. The system rewrites prompts to use few-shot examples and in-context learning instead of relying on model instruction-following, enabling deployment on cheaper base models. This is a specialized variant of the cost-aware downconversion capability targeting the instruct→base transition.

Provides a specialized optimization variant for classification tasks where outputs are categorical (e.g., sentiment, intent, entity type). The system evaluates prompts based on classification accuracy, precision, recall, or F1 score rather than generic pairwise comparisons. This variant includes custom evaluation logic that parses model outputs as categories and compares against ground truth labels, enabling more precise evaluation for classification tasks.