The request names deepeval, which none of the candidate sources describe or link to, so no product here can be confirmed as deepeval itself or its official doors. The closest alternatives for LLM evaluation are Pydantic Evals, which runs offline Python evaluations 3, Langfuse, an open-source platform for evaluating and improving AI agents 4, and MLflow, which offers systematic evaluation of LLM applications and agents 2.
- 1
Pydantic Evalspydantic.devverified and signed
Runs offline Python evaluations (pydantic_evals) and reviews them in Logfire, per the logfire-evals action 3.
+25 community tools built for it
for agents: logfire-evals · building-pydantic-ai-agents · logfire-instrumentation
- 2
Langfuselangfuse.comverified and signed
Open-source platform to trace, evaluate and improve AI agents, with an LLM-as-a-judge evaluation skill 4.
+30 community tools built for it
for agents: langfuse · llm-as-a-judge · llm
- 3
MLflowmlflow.orgverified and signed
Open-source AI engineering platform that runs systematic evaluations of LLM applications and agents 2.
+6 community tools built for it
for agents: agent-evaluation · instrumenting-with-mlflow-tracing · mlflow
Sources
- 2MLflow - Open Source AI Platform for Agents, LLMs & Modelsmlflow.org · read 2026-10-29
- 3Pydantic | The end-to-end AI engineering stackpydantic.dev · read 2026-10-29
- 4Langfuse: Open Source Agent Evals & Observabilitylangfuse.com · read 2026-10-29
For agents: the 10 actions behind this answer
arize-evaluator · arize-evaluator verified and signed
Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallucination, faithfulne
agent-evaluation · agent-evaluation verified and signed
Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong/incomplete responses. Evaluates agents systematically using MLflow evaluation w
logfire-evals · logfire-evals verified and signed
Run offline Python (`pydantic_evals`) or Node.js (`logfire/evals`) evaluations and review them in Logfire. Also redirect existing Braintrust `Eval()` suites. Use for eval setup, test datasets, AI scoring, agent checks, LLM judges, Braintrust migration, or Logfire Datasets & Experiments. Not for live
langfuse · langfuse verified and signed
Skills for working with Langfuse, the open-source LLM engineering platform for tracing, prompt management, and evaluation.
add · mcp-evals verified and signed
prompt_test_suite · ia-qa mcp llm and RAG testing - Dev/QA toolbox verified and signed
Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.
andreasjansson/flash-eval · andreasjansson/flash-eval verified and signed
Suite of models to evaluate the image quality of text-to-image models with respect to their input prompts.
evaluate_with_llm_judge · iris-eval/mcp-server found in code, not yet checked
· CompletionKit website read, tools not yet checked
Prompt evaluation for AI apps. Run prompts against real inputs, score outputs 1-5 with an LLM judge, ship only what improved. Free on Cloud, or self-host.
shell · shin-watcher-oss found in code, not yet checked
Run a shell command in the cloned BerriAI/litellm working tree. Use this for git, uv, curl, ls, cat, pytest, etc.
Labels are the instruments' verdict and are never changed by ranking. Agents and robots get this same answer as JSON at /api/v1/find?q=…&answer=1 and through the MCP server.