Few Shot Example Synthesis And Selection

via “few-shot example sampling with stratification and caching”

EleutherAI's evaluation framework — 200+ benchmarks, powers Open LLM Leaderboard.

Unique: Integrates few-shot sampling directly into the request generation pipeline with built-in caching and stratification support. The system computes sampling once per task, caches results, and reuses them across all evaluation instances. Stratified sampling uses class labels to ensure balanced representation, which is critical for imbalanced datasets where random sampling might miss minority classes.

vs others: Provides stratified sampling (not just random) and automatic caching that alternatives like simple prompt engineering lack; integrates sampling into the evaluation pipeline rather than requiring manual example selection

via “few-shot example synthesis and selection”

Stanford framework that replaces manual prompting with automatically optimized LLM programs.

Unique: Automatically selects examples from training data based on metric-driven feedback, rather than relying on manual curation or random sampling. Advanced optimizers like GEPA can synthesize new examples using reflective reasoning, generating demonstrations that target specific failure modes.

vs others: More sophisticated than random example selection and more scalable than manual curation, DSPy's example synthesis integrates with the optimization loop to learn examples that maximize task-specific metrics.

via “few-shot example management and dynamic selection”

LMQL is a query language for large language models.

Unique: Integrates example selection and formatting into the LMQL query language, allowing examples to be selected dynamically based on input and constrained by token budgets within the same query execution

vs others: More integrated than manually managing examples in application code; more flexible than static few-shot prompts because example selection is dynamic and can adapt to input characteristics