The request names Arize, so Arize comes first: its agent observability and evaluation platform 1 offers arize-annotation, which creates and manages annotation configs and annotation queues for human review 11. Among the other candidates, Langfuse 5 provides annotation queues for human review and scoring of traces 15, and Arize's own action set also includes arize-link for deep links to labeling queues 11.
- 1
Arize AXarize.comverified and signed
Arize's agent observability and evaluation platform; its arize-annotation action creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues for human review workflows 111.
+3 community tools built for it
for agents: arize-annotation · arize-link · arize-experiment
- 2
Langfuselangfuse.comfound in code, not yet checked
Open source platform for tracing, evaluating and improving AI agents, with annotation queues that collect trace or observation items for human review and scoring 515.
+30 community tools built for it
for agents: createAnnotationQueue · getAnnotationQueue · createAnnotationQueueItem
Sources
- 1Agent Observability, Evaluation & Improvement Platform | Arize AIarize.com · read 2026-10-30
- 5Langfuse: Open Source Agent Evals & Observabilitylangfuse.com · read 2026-10-29
- 11arize-annotation: declares "arize-annotation" (the record we read and signed)unfragile.ai · read 2026-10-24
- 15langfuse: declares "createAnnotationQueue" (the record we read)github.com · read 2026-10-24
For agents: the 10 actions behind this answer
arize-annotation · arize-annotation verified and signed
Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedbac
phoenix-cli · phoenix-cli verified and signed
Debug LLM applications using the Phoenix CLI (`px`, from @arizeai/phoenix-cli). Fetch traces, spans, and sessions, annotate them, analyze errors, measure how often a failure or pattern occurs across traces, inspect datasets, review experiments, query annotation configs, and use the GraphQL API. Use
· Arize Openinference website read, tools not yet checked
Continuously improve AI agents with agent observability, evaluation, tracing, and experimentation.
get_query_examples · Text-to-GraphQL MCP Server found in code, not yet checked
Get example queries to help users understand what they can ask for
· Arize Tutorials website read, tools not yet checked
Continuously improve AI agents with agent observability, evaluation, tracing, and experimentation.
get_arize_tracing_docs · arize-tracing-assistant found in code, not yet checked
Get docs and examples to instrument an app and send traces/spans to Arize. If the framework is not in the list use manual instrumentation with open telemetry.
arize-annotation · arize-annotation verified and signed
Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedbac
radar-get-annotation-by-id · Cloudflare API verified and signed
Get annotation by ID
getAnnotation · Bitbucket API verified and signed
Get an annotation
createAnnotationQueue · langfuse found in code, not yet checked
Create an annotation queue, a worklist that collects trace or observation items for human review and scoring.
Labels are the instruments' verdict and are never changed by ranking. Agents and robots get this same answer as JSON at /api/v1/find?q=…&answer=1 and through the MCP server.